New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Imports are slower than with CPython #2854
Comments
Hmm, interesting. My first thought was Py2-style imports, but you're using Importing has never really been a priority for optimisation, more of a "just make it work across all Python versions" kind of task. But I can't see a reason why Cython should be slower than CPython here, let alone 4x as slow. Definitely worth investigating. |
I investigated this. CPython imports the module by dotted name and then looks up attributes on the result until it found the final imported sub-sub-sub-…module. Cython (and probably Pyrex already) took the easy path of passing a non-empty name list into the import function (actually the list |
…ally under Py3.7. Cython used to be 3x slower than CPython here. Also enable absolute imports automatically in top-level (non-package) modules.
Ok, all not that easy, due to CPython's long-grown import C-API, which is entirely backwards. If I want to import a module, then please give me the module and don't leave me standing in the rain with the top-level module that I have to look up attributes on. :( While many things can be worked around, it becomes tricky for circular and concurrent imports. If one module imports itself, then it might be in
|
Actually, there is already some code in CPython that you can copy to have a fast path while still guarding against uninitialized modules: see https://bugs.python.org/issue35943#msg335097 Great to see this advancing towards resolution :-) |
Of course, if you want to limit the reliance on private APIs, you could just defer to regular importing if the |
Thanks, I had already found that code in I had already implemented the flag check, but then noticed that I would then also have to duplicate the whole "import importlib, call locking function" part, which is a lot of overhead. (Also, importing importlib in import code feels a bit … meta.) cython/Cython/Utility/ImportExport.c Lines 110 to 131 in 4993ba6
|
Well, the cost of calling into importlib will almost never be paid, so I'm not sure you should worry about that. Still, correctness is important IMHO. As I wrote above, you needn't call importlib yourself, you can just fall back on the normal non-optimized path if the |
Well, "almost never" does not seem to take circular imports into account, which do not seem all that uncommon to me. |
But you would only pay the price on the first import, not once the module is entirely loaded... The point of the optimization here is to make imports of already-loaded modules fast. Making fresh imports faster is not really achievable within Cython (unless you decide to break import semantics, that is ;-)). |
As you can see from the code above, I've actually tried it. The problem then was to find a way how to make CPython use the right fallback. The import C-API is really horrible. Important functionality is scattered across different functions and entry points, some things are implemented in Python, and some are even in I agree about the correctness and will give it another try. |
Ah... well, I suppose you could do whatever you did before this optimization? :-) |
So, the problem I had was really with circular imports. While resolving them, the module is already in |
@scoder Falling back to checking sys.modules is pretty close to what CPython does to handle circular imports - it's a last-gasp attempt in the import opcodes when looking up the module attribute throws I agree on this being spectacularly convoluted though (and will comment further on that in Antoine's capi-sig thread). |
Since cython@4993ba6, we returned the top-level package module instead of the module that was actually imported with its dotted name ("collections" instead of "collections.abc"). Closes cython#5308 See cython#2854
Since cython@4993ba6, we returned the top-level package module instead of the module that was actually imported with its dotted name ("collections" instead of "collections.abc"). Closes cython#5308 See cython#2854
If I compile the following functions using Cython:
And then measure their performance:
The import overhead (~ 900 ns. per import statement) appears much slower than what CPython achieves:
This is with Python 3, Cython 0.29.4 and
language_level = 3
.The text was updated successfully, but these errors were encountered: