stat cache for import bootstrap #63415
Comments
|
The import library uses excessive stat() calls. I've implemented a simple cache for the bootstrap module that reduces the amount of stat() calls by almost 1/3 (236 -> 159 on Linux). |
|
See also bpo-14604. |
|
Benchmarks? |
|
A cursory look at the patch suggests that the cache use is permanent and so any dynamic changes to a file or directory after an initial caching will not be picked up. Did you run the test suite with this patch as it should have failed. |
|
Is the content of the bootstrap module used after the interpreter is boot strapped? I see ... that's a problem. It's a proof of concept anyway and the speed up is minimal. On my computer with a SSD the speedup barely measurable. I'd like to see if it makes a difference on a Raspbarry Pi or a NFS shares I have another idea, too. Could we add an optional 'stat' argument to __init__() of FileLoader and ExtensionFileLoader so we can pass the stat object around and reuse it for loading? |
|
importlib/_bootstrap.py is importlib, period, so there is no separation of what is used to start Python and what is used after interpreter startup is completed. As for adding a 'stat' argument to the loaders, it's possible but as always it comes down to whether it will break someone or not. Since loaders do not necessarily execute immediately you are running the risk of a very stale cached stat object. Plus Eric Snow has his PEP where the API in terms of loader __init__ signature so you would want to look into that. |
|
With ModuleSpec (PEP-451), the finder creates the spec object (where it stores the loader). At that point the finder is free to store any stat object you like in spec.loader_state. The spec is made available to the loader during exec (if the loader supports it, which the importlib loaders will). So there is no need to add anything to any loader __init__. The only catch is the slim possibility that the stat object will be stale by the time it gets used. I seem to remember a case where something like this happened (related to distros building their system Python or something). |
|
For interpreter startup, stats are not involved for builtin and frozen modules[1]. They are tied to imports that involve traversing sys.path (a.k.a. PathFinder). Most stats happen in FileFinder.find_loader. The remainder are for source (.py) files (a.k.a. SourceFileLoader). Here's a rough sketch of what typically happens currently during the import of a path-based module[2], as related to stats (and other FS access): (lines with FS access start with *) def load_module(fullname):
suffixes = ['.cpython-34m.so', '.abi3.so', '.so', '.py', '.pyc']
tailname = fullname.rpartition('.')[2]
for entry in sys.path:
* mtime = os.stat(entry).st_mtime
if mtime != cached_mtime:
* cached_listdir = os.listdir(entry)
if tailname in cached_listdir:
basename = entry/tailname
* if os.stat(basename).st_mode implies directory: # superfluous?
# package?
for suffix in suffixes:
full_path = basename + suffix
* if os.stat(full_path).st_mode implies file:
if is_extension:
* <dlopen>(full_path)
elif is_sourceless:
* open(full_path).read()
else:
load_from_source(full_path)
return
# ...non-package module?
for suffix in suffixes:
full_path = entry/tailname + suffix
if tailname + suffix in cached_listdir:
* if os.stat(full_path).st_mode implies file: # superfluous?
if is_extension:
* <dlopen>(full_path)
elif is_sourceless:
* open(full_path).read()
else:
load_from_source(full_path)
def load_from_source(sourcepath):
* st = os.stat(sourcepath)
if st:
* open(bytecodepath).read()
else:
* open(sourcepath).read()
* os.stat(sourcepath).st_mode
for parent in ancestor_dirs(sourcepath):
* os.stat(parent).st_mode -> missing_parents
for parent in missing_parents:
* os.mkdir(parent)
* open(tempname).write()
* os.replace(tempname, bytecodepath)Obviously there are some unix-isms in there. Windows ends up not that different though. stat/FS count load_module (per path entry): load_from_source: Highlights:
Possible improvements:
[1] Maybe we should freeze the stdlib. <0.5 wink> |
|
So the 2 stat calls in the general case are superfluous, it's just a question of whether they make any performance difference. Turns out that at least on my Macbook their is no performance difference and thus not worth the cost of breaking semantics over it: http://bugs.python.org/issue18810 . As for completely turning off stat calls during interpreter startup, that would definitely buy us something, but the question is how much and how do we make it work reliably? |
|
I realized those two stats are not superfluous in the case that a directory name has a .py suffix or a file doesn't have any suffix. However, I expect that's pretty uncommon. Worst case, these cases cost 2 stats per path entry. In practice they cost nothing due to the dir caching we already do. |
|
I forgot to mention that optimizing the default composition of sys.path (from site) could help speed things up, though it might already be optimized in that regard. I also forgot to mention the idea of zipping up the stdlib. Sorry for the sidetrack. Now, back to the stat discussion... |
|
The real problem here is that the definition of "bootstrap" or "startup" is fuzzy. How do you decide when you stop caching? |
|
Would it be feasible to have an explicit (but private?) flag in importlib indicating stat checking (or even all FS checking) should be disabled, defaulting to True? runpy could set it to False after initializing importlib and then back to True when startup is done. If that was useful for more than just startup, we could also add a contextmanager for it in importlib.util. |
I don't really understand the algorithm you're proposing. Also, have you |
In importlib._bootstrap: We have some global like "_CHECK_STAT=True". FileFinder would use it to decide on using stat checks or not. At the end of import_init(), we set importlib._bootstrap _CHECK_STAT to False. Then at the end of _Py_InitializeEx_Private() we set it back to True. (As an alternative, we could always not do stat checking for just the standard library)
About the fuzziness of when startup is finished? As implied above, I'd say at the end of Py_Initialize(). |
You only have imported a handful of modules by then. Real-world $ python -v -c pass 2>&1 | grep "^import" | wc -l
33
$ python -v `which hg` 2>&1 | grep "^import" | wc -l
117Note that Mercurial has a lazy importer in order to improve startup |
|
The benefit of avoiding stat() calls seems to not be obvious to everybody. Moreover, importlib now implements a "path cache". I close the issue. The most efficient solution is to pack all your modules and the Python stdlib into a ZIP file: everything is done in memory, no more filesystem access. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: