Deep Lazy Imports - Interpreter-level deferred module loading

[Kronuz]

BPO	46963
Nosy	@carljm, @DinoV, @Kronuz, @zsol, @itamaro

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2022-03-08.19:32:14.652>
labels = ['interpreter-core', 'type-feature', '3.11']
title = 'Deep Lazy Imports - Interpreter-level deferred module loading'
updated_at = <Date 2022-03-15.21:34:28.541>
user = 'https://github.com/Kronuz'

bugs.python.org fields:

activity = <Date 2022-03-15.21:34:28.541>
actor = 'zsol'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2022-03-08.19:32:14.652>
creator = 'Kronuz'
dependencies = []
files = []
hgrepos = []
issue_num = 46963
keywords = []
message_count = 1.0
messages = ['414770']
nosy_count = 5.0
nosy_names = ['carljm', 'dino.viehland', 'Kronuz', 'zsol', 'itamaro']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46963'
versions = ['Python 3.11']

[Kronuz]

As the size of a Python project increases, the number of modules and the complexity of its dependencies increases too, producing two problems in large codebases: increased risk of import cycles and slow start times due to the number of modules that tend to need getting loaded.

We propose implementing a robust and transparent lazy loader in CPython, based on an existing implementation we have in Cinder that proved to be extremely valuable to thousands of developers across Meta, and battle-tested in Instagram Server in production (https://docs.google.com/document/d/1l8I-FDE1xrIShm9eSNJqsGmY_VanMDX5-aK_gujhYBI/edit#heading=h.pu7ja6wu0ib).

Our internal implementation is based on introducing deferred objects and converting top-level imported names to these deferred objects that are evaluated on first use. Enabling this across multiple types of workloads in Meta consistently shown improvements in startup times (up to 70%) and memory footprint (up to 40%), while virtually eliminating occurrences of import cycles.

Converting imported names to deferred objects is a semantic change, and the benefits are usually meaningful only at very large scale, so we propose not changing the default behavior, and making it possible for users to opt-in to this lazy loading mechanism via -x flags or environment variables.

This change would require related documentation changes that would cover the gotchas and edge cases, such as: some packages rely and import-time side-effects for correct operation; nested modules can’t be accessed unless imported explicitly; deferred loading may also defer exceptions from import time and first-access time which could be confusing.

The import-time side effects gotcha can be mitigated using eager-import lists and directives, though it could be desirable to use this as a forcing function to discourage package maintainers from relying on these side-effects.

[warsaw]

From the linked document:

Our implementation supports enabling lazy imports globally (using an environment variable or a -x flag), as well as opting in or out at a per module level (using future imports). The primary gotchas, when enabled, include unexpected behaviors when using modules that rely on import-time side effects, and postponed error messages that can occur out of context. We don’t have good solutions for these issues, and would like to see the community at large reducing reliance on import-time side effects as much as possible.

If I'm reading this correctly, once enabled all imports are lazy. Have you done any analysis to see which modules in a CLI actually need to be lazily imported? IOW, not all modules actually affect start up time, but certainly entry points would win from lazy imports. Have you done any research to see the effect of decorators (I'm thinking like click overhead). How does it affect the way CLI's are written? Do you have any code examples you can share? Have you written a PR against upstream Python for porting this to CPython?

[Kronuz]

@warsaw, when you enable via -X lazyimportsall, yes all imports at the module level become lazy (even imports inside the standard library); however, this is only one of the options. You can also enable in a per-module basis (currently importing from future: from __future__ import lazy_imports, and all imports in the module become lazy.

Any code that gets loaded is always slower than no code being loaded, so in my experience there are quite a lot of things that do benefit from globally enabled Lazy Imports. Decorators work the same way, almost everything works transparently, as long as it doesn't rely in import side effects. It was surprising to find that there really weren't that many issues and not too many modules relying in side effects when enabling Lazy Imports globally in our codebases, so we don't currently use the per-module option anymore. The reliance in side effects is mainly used for global registration of something or patching of things. We found out it's very easy to just enable it everywhere and fix the few modules that rely in import side effects for them to register or patch things explicit by calling functions designated for that purpose.

There's a lot left to do, but the results we've seen and the easiness of use is overwhelmingly promising to say the least. We'll work in a PR and a PEP for porting this to CPython. it should be really straightforward since it doesn't really use any Cinder specific features or specifics.

[brandtbucher]

How does this feature interact with the semi-common "try to import, otherwise define" pattern? What happens here:

from __future__ import lazy_imports

try:
    from _c_accelerator import foo
except ImportError:
    def foo(): ...

Also, how does it work with starred imports?

[Kronuz]

@brandtbucher, in the cases where imports are inside try/except/finally, imports are always eager (precisely for that reason) and star imports are also eager. These two cases are not too common, so making the imports eager doesn't hurt the benefits too much.

[brandtbucher]

Any other special cases? I can't think of any other reasonably-common ones, besides maybe if/else guards:

if condition:
    from foo import bar
else:
    def bar(): ...

[Kronuz]

if/else doesn't make imports eager; still, however, the above snippet should work as expected and the imported bar would be lazy.

[brandtbucher]

Got it, thanks!