diff --git a/peps/pep-0810.rst b/peps/pep-0810.rst index 496e1dfd78b..dd6969a8b77 100644 --- a/peps/pep-0810.rst +++ b/peps/pep-0810.rst @@ -246,7 +246,7 @@ Syntax restrictions ~~~~~~~~~~~~~~~~~~~ The soft keyword is only allowed at the global (module) level, **not** inside -functions, class bodies, with ``try``/``with`` blocks, or ``import *``. Import +functions, class bodies, ``try`` blocks, or ``import *``. Import statements that use the soft keyword are *potentially lazy*. Imports that can't be lazy are unaffected by the global lazy imports flag, and instead are always eager. Additionally, ``from __future__ import`` statements cannot be @@ -270,10 +270,6 @@ Examples of syntax errors: except ImportError: pass - # SyntaxError: lazy import not allowed inside with blocks - with suppress(ImportError): - lazy import json - # SyntaxError: lazy from ... import * is not allowed lazy from json import * @@ -465,54 +461,37 @@ immediately resolve all lazy objects (e.g. ``lazy from`` statements) that referenced the module. It **only** resolves the lazy object being accessed. Accessing a lazy object (from a global variable or a module attribute) reifies -the object. Accessing a module's ``__dict__`` reifies **all** lazy objects in -that module. Calling ``dir()`` at the global scope will not reify the globals -and calling ``dir(mod)`` will be special cased in ``mod.__dir__`` avoid -reification as well. - -Example using ``__dict__`` from external code: - -.. code-block:: python - - # my_module.py - import sys - lazy import json +the object. - print('json' in sys.modules) # False - still lazy - - # main.py - import sys - import my_module - - # Accessing __dict__ from external code DOES reify all lazy imports - d = my_module.__dict__ - - print('json' in sys.modules) # True - reified by __dict__ access - print(type(d['json'])) # +However, calling ``globals()`` or accessing a module's ``__dict__`` does +**not** trigger reification -- they return the module's dictionary, and +accessing lazy objects through that dictionary still returns lazy proxy +objects that need to be manually reified upon use. A lazy object can be +resolved explicitly by calling the ``resolve`` method. Calling ``dir()`` at +the global scope will not reify the globals, nor will calling ``dir(mod)`` +(through special-casing in ``mod.__dir__``.) Other, more indirect ways of +accessing arbitrary globals (e.g. inspecting ``frame.f_globals``) also do +**not** reify all the objects. -However, calling ``globals()`` does **not** trigger reification -- it returns -the module's dictionary, and accessing lazy objects through that dictionary -still returns lazy proxy objects that need to be manually reified upon use. A -lazy object can be resolved explicitly by calling the ``resolve`` method. -Other, more indirect ways of accessing arbitrary globals (e.g. inspecting -``frame.f_globals``) also do **not** reify all the objects. - -Example using ``globals()``: +Example using ``globals()`` and ``__dict__``: .. code-block:: python + # my_module.py import sys lazy import json # Calling globals() does NOT trigger reification g = globals() - print('json' in sys.modules) # False - still lazy print(type(g['json'])) # + # Accessing __dict__ also does NOT trigger reification + d = __dict__ + print(type(d['json'])) # + # Explicitly reify using the resolve() method resolved = g['json'].resolve() - print(type(resolved)) # print('json' in sys.modules) # True - now loaded @@ -707,15 +686,15 @@ Where ```` can be: * ``"normal"`` (or unset): Only explicitly marked lazy imports are lazy -* ``"all"``: All module-level imports (except in ``try`` or ``with`` +* ``"all"``: All module-level imports (except in ``try`` blocks and ``import *``) become *potentially lazy* * ``"none"``: No imports are lazy, even those explicitly marked with ``lazy`` keyword When the global flag is set to ``"all"``, all imports at the global level -of all modules are *potentially lazy* **except** for those inside a ``try`` or -``with`` block or any wild card (``from ... import *``) import. +of all modules are *potentially lazy* **except** for those inside a ``try`` +block or any wild card (``from ... import *``) import. If the global lazy imports flag is set to ``"none"``, no *potentially lazy* import is ever imported lazily, the import filter is never called, and @@ -1251,36 +1230,9 @@ either the lazy proxy or the final resolved object. Can I force reification of a lazy import without using it? ---------------------------------------------------------- -Yes, accessing a module's ``__dict__`` will reify all lazy objects in that -module. Individual lazy objects can be resolved by calling their ``resolve()`` +Yes, individual lazy objects can be resolved by calling their ``resolve()`` method. -What's the difference between ``globals()`` and ``mod.__dict__`` for lazy imports? ----------------------------------------------------------------------------------- - -Calling ``globals()`` returns the module's dictionary without reifying lazy -imports -- you'll see lazy proxy objects when accessing them through the -returned dictionary. However, accessing ``mod.__dict__`` from external code -reifies all lazy imports in that module first. This design ensures: - -.. code-block:: python - - # In your module: - lazy import json - - g = globals() - print(type(g['json'])) # - your problem - - # From external code: - import sys - mod = sys.modules['your_module'] - d = mod.__dict__ - print(type(d['json'])) # - reified for external access - -This distinction means adding lazy imports and calling ``globals()`` is your -responsibility to manage, while external code accessing ``mod.__dict__`` -always sees fully loaded modules. - Why not use ``importlib.util.LazyLoader`` instead? -------------------------------------------------- @@ -1664,6 +1616,52 @@ From the discussion on :pep:`690` it is clear that this is a fairly contentious idea, although perhaps once we have wide-spread use of lazy imports this can be reconsidered. +Disallowing lazy imports inside ``with`` blocks +------------------------------------------------ + +An earlier version of this PEP proposed disallowing ``lazy import`` statements +inside ``with`` blocks, similar to the restriction on ``try`` blocks. The +concern was that certain context managers (like ``contextlib.suppress(ImportError)``) +could suppress import errors in confusing ways when combined with lazy imports. + +However, this restriction was rejected because ``with`` statements have much +broader semantics than ``try/except`` blocks. While ``try/except`` is explicitly +about catching exceptions, ``with`` blocks are commonly used for resource +management, temporary state changes, or scoping -- contexts where lazy imports +work perfectly fine. The ``lazy import`` syntax is explicit enough that +developers who write it inside a ``with`` block are making an intentional choice, +aligning with Python's "consenting adults" philosophy. For genuinely problematic +cases like ``with suppress(ImportError): lazy import foo``, static analysis +tools and linters are better suited to catch these patterns than hard language +restrictions. + +Forcing eager imports in ``with`` blocks under the global flag +--------------------------------------------------------------- + +Another rejected idea was to make imports inside ``with`` blocks remain eager +even when the global lazy imports flag is set to ``"all"``. The rationale was +to be conservative: since ``with`` statements can affect how imports behave +(e.g., by modifying ``sys.path`` or suppressing exceptions), forcing imports to +remain eager could prevent subtle bugs. However, this would create inconsistent +behavior where ``lazy import`` is allowed explicitly in ``with`` blocks, but +normal imports remain eager when the global flag is enabled. This inconsistency +between explicit and implicit laziness is confusing and hard to explain. + +The simpler, more consistent rule is that the global flag affects imports +everywhere that explicit ``lazy import`` syntax is allowed. This avoids having +three different sets of rules (explicit syntax, global flag behavior, and filter +mechanism) and instead provides two: explicit syntax rules match what the global +flag affects, and the filter mechanism provides escape hatches for edge cases. +For users who need fine-grained control, the filter mechanism +(``sys.set_lazy_imports_filter()``) already provides a way to exclude specific +imports or patterns. Additionally, there's no inverse operation: if the global +flag forces imports eager in ``with`` blocks but a user wants them lazy, there's +no way to override it, creating an asymmetry. + +In summary: imports in ``with`` blocks behave consistently whether marked +explicitly with ``lazy import`` or implicitly via the global flag, creating a +simple rule that's easy to explain and reason about. + Modification of the dict object ------------------------------- @@ -1868,55 +1866,38 @@ from a real dict in almost all cases, which is extremely difficult to achieve correctly. Any deviation from true dict behavior would be a source of subtle bugs. -Reifying lazy imports when ``globals()`` is called ---------------------------------------------------- +Automatically reifying on ``__dict__`` or ``globals()`` access +-------------------------------------------------------------- -Calling ``globals()`` returns the module's namespace dictionary without -triggering reification of lazy imports. Accessing lazy objects through the -returned dictionary yields the lazy proxy objects themselves. This is an -intentional design decision for several reasons: - -**The key distinction**: Adding a lazy import and calling ``globals()`` is the -module author's concern and under their control. However, accessing -``mod.__dict__`` from external code is a different scenario -- it crosses -module boundaries and affects someone else's code. Therefore, ``mod.__dict__`` -access reifies all lazy imports to ensure external code sees fully realized -modules, while ``globals()`` preserves lazy objects for the module's own -introspection needs. - -**Technical challenges**: It is impossible to safely reify on-demand when -``globals()`` is called because we cannot return a proxy dictionary -- this -would break common usages like passing the result to ``exec()`` or other -built-ins that expect a real dictionary. The only alternative would be to -eagerly reify all lazy imports whenever ``globals()`` is called, but this -behavior would be surprising and potentially expensive. - -**Performance concerns**: It is impractical to cache whether a reification -scan has been performed with just the globals dictionary reference, whereas -module attribute access (the primary use case) can efficiently cache -reification state in the module object itself. - -**Use case rationale**: The chosen design makes sense precisely because of -this distinction: adding a lazy import and calling ``globals()`` is your -problem to manage, while having lazy imports visible in ``mod.__dict__`` -becomes someone else's problem. By reifying on ``__dict__`` access but not on -``globals()``, we ensure external code always sees fully loaded modules while -giving module authors control over their own introspection. - -Note that three options were considered: +Three options were considered for how ``globals()`` and ``mod.__dict__`` should +behave with lazy imports: 1. Calling ``globals()`` or ``mod.__dict__`` traverses and resolves all lazy objects before returning. 2. Calling ``globals()`` or ``mod.__dict__`` returns the dictionary with lazy - objects present. + objects present (chosen). 3. Calling ``globals()`` returns the dictionary with lazy objects, but ``mod.__dict__`` reifies everything. -We chose the third option because it properly delineates responsibility: if -you add lazy imports to your module and call ``globals()``, you're responsible -for handling the lazy objects. But external code accessing your module's -``__dict__`` shouldn't need to know about your lazy imports -- it gets fully -resolved modules. +We chose option 2: both ``globals()`` and ``__dict__`` return the raw +namespace dictionary without triggering reification. This provides a clean, +predictable model where low-level introspection APIs don't trigger side +effects. + +Having ``globals()`` and ``__dict__`` behave identically creates symmetry and +a simple mental model: both expose the raw namespace view. Low-level +introspection APIs should not automatically trigger imports, which would be +surprising and potentially expensive. Real-world experience implementing lazy +imports in the standard library (such as the traceback module) showed that +automatic reification on ``__dict__`` access was cumbersome and forced +introspection code to load modules it was only examining. + +Option 1 (always reifying) was rejected because it would make ``globals()`` +and ``__dict__`` access surprisingly expensive and prevent introspecting the +lazy state of a module. Option 3 was initially considered to "protect" external +code from seeing lazy objects, but real-world usage showed this created more +problems than it solved, particularly for stdlib code that needs to introspect +modules without triggering side effects. Acknowledgements ================