From e028ff335ffad8385b39303cc9a047932a9c3f33 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Mon, 13 Oct 2025 19:03:33 +0100 Subject: [PATCH 1/6] PEP 810: Update decisions on with blocks and __dict__ reification This commit updates PEP 810 based on team discussion and feedback: 1. Allow lazy imports inside with blocks - Removed with blocks from syntax restrictions - Added rejected ideas section explaining why restriction was considered 2. __dict__ does not automatically reify lazy imports - Both globals() and __dict__ return raw dictionary without reification - Updated reification section, FAQ, and rejected ideas --- peps/pep-0810.rst | 206 +++++++++++++++++++++++++--------------------- 1 file changed, 113 insertions(+), 93 deletions(-) diff --git a/peps/pep-0810.rst b/peps/pep-0810.rst index 496e1dfd78b..6203ac88e0c 100644 --- a/peps/pep-0810.rst +++ b/peps/pep-0810.rst @@ -246,7 +246,7 @@ Syntax restrictions ~~~~~~~~~~~~~~~~~~~ The soft keyword is only allowed at the global (module) level, **not** inside -functions, class bodies, with ``try``/``with`` blocks, or ``import *``. Import +functions, class bodies, ``try`` blocks, or ``import *``. Import statements that use the soft keyword are *potentially lazy*. Imports that can't be lazy are unaffected by the global lazy imports flag, and instead are always eager. Additionally, ``from __future__ import`` statements cannot be @@ -270,10 +270,6 @@ Examples of syntax errors: except ImportError: pass - # SyntaxError: lazy import not allowed inside with blocks - with suppress(ImportError): - lazy import json - # SyntaxError: lazy from ... import * is not allowed lazy from json import * @@ -465,54 +461,37 @@ immediately resolve all lazy objects (e.g. ``lazy from`` statements) that referenced the module. It **only** resolves the lazy object being accessed. Accessing a lazy object (from a global variable or a module attribute) reifies -the object. Accessing a module's ``__dict__`` reifies **all** lazy objects in -that module. Calling ``dir()`` at the global scope will not reify the globals -and calling ``dir(mod)`` will be special cased in ``mod.__dir__`` avoid +the object. Calling ``dir()`` at the global scope will not reify the globals +and calling ``dir(mod)`` will be special cased in ``mod.__dir__`` to avoid reification as well. -Example using ``__dict__`` from external code: - -.. code-block:: python - - # my_module.py - import sys - lazy import json - - print('json' in sys.modules) # False - still lazy - - # main.py - import sys - import my_module - - # Accessing __dict__ from external code DOES reify all lazy imports - d = my_module.__dict__ - - print('json' in sys.modules) # True - reified by __dict__ access - print(type(d['json'])) # +However, calling ``globals()`` or accessing a module's ``__dict__`` does **not** +trigger reification -- they return the module's dictionary, and accessing lazy +objects through that dictionary still returns lazy proxy objects that need to +be manually reified upon use. A lazy object can be resolved explicitly by +calling the ``resolve`` method. Other, more indirect ways of accessing +arbitrary globals (e.g. inspecting ``frame.f_globals``) also do **not** reify +all the objects. -However, calling ``globals()`` does **not** trigger reification -- it returns -the module's dictionary, and accessing lazy objects through that dictionary -still returns lazy proxy objects that need to be manually reified upon use. A -lazy object can be resolved explicitly by calling the ``resolve`` method. -Other, more indirect ways of accessing arbitrary globals (e.g. inspecting -``frame.f_globals``) also do **not** reify all the objects. - -Example using ``globals()``: +Example using ``globals()`` and ``__dict__``: .. code-block:: python + # my_module.py import sys lazy import json # Calling globals() does NOT trigger reification g = globals() - print('json' in sys.modules) # False - still lazy print(type(g['json'])) # + # Accessing __dict__ also does NOT trigger reification + d = __dict__ + print(type(d['json'])) # + # Explicitly reify using the resolve() method resolved = g['json'].resolve() - print(type(resolved)) # print('json' in sys.modules) # True - now loaded @@ -707,15 +686,15 @@ Where ```` can be: * ``"normal"`` (or unset): Only explicitly marked lazy imports are lazy -* ``"all"``: All module-level imports (except in ``try`` or ``with`` +* ``"all"``: All module-level imports (except in ``try`` blocks and ``import *``) become *potentially lazy* * ``"none"``: No imports are lazy, even those explicitly marked with ``lazy`` keyword When the global flag is set to ``"all"``, all imports at the global level -of all modules are *potentially lazy* **except** for those inside a ``try`` or -``with`` block or any wild card (``from ... import *``) import. +of all modules are *potentially lazy* **except** for those inside a ``try`` +block or any wild card (``from ... import *``) import. If the global lazy imports flag is set to ``"none"``, no *potentially lazy* import is ever imported lazily, the import filter is never called, and @@ -1251,17 +1230,17 @@ either the lazy proxy or the final resolved object. Can I force reification of a lazy import without using it? ---------------------------------------------------------- -Yes, accessing a module's ``__dict__`` will reify all lazy objects in that -module. Individual lazy objects can be resolved by calling their ``resolve()`` -method. +Yes, individual lazy objects can be resolved by calling their ``resolve()`` +method. Note that accessing a module's ``__dict__`` or calling ``globals()`` +does **not** automatically reify lazy imports -- you'll see the lazy proxy +objects themselves, which you can then manually resolve if needed. What's the difference between ``globals()`` and ``mod.__dict__`` for lazy imports? ---------------------------------------------------------------------------------- -Calling ``globals()`` returns the module's dictionary without reifying lazy -imports -- you'll see lazy proxy objects when accessing them through the -returned dictionary. However, accessing ``mod.__dict__`` from external code -reifies all lazy imports in that module first. This design ensures: +Both ``globals()`` and ``mod.__dict__`` return the module's dictionary without +reifying lazy imports. Accessing lazy objects through either will yield lazy +proxy objects. This provides a consistent low-level API for introspection: .. code-block:: python @@ -1269,17 +1248,22 @@ reifies all lazy imports in that module first. This design ensures: lazy import json g = globals() - print(type(g['json'])) # - your problem + print(type(g['json'])) # + + d = __dict__ + print(type(d['json'])) # # From external code: import sys mod = sys.modules['your_module'] d = mod.__dict__ - print(type(d['json'])) # - reified for external access + print(type(d['json'])) # -This distinction means adding lazy imports and calling ``globals()`` is your -responsibility to manage, while external code accessing ``mod.__dict__`` -always sees fully loaded modules. +Both ``globals()`` and ``__dict__`` expose the raw namespace view without +implicit side effects. This symmetry makes the behavior predictable: accessing +the namespace dictionary never triggers imports. If you need to ensure an +import is resolved, call the ``resolve()`` method explicitly or access the +attribute directly (e.g., ``json.dumps``). Why not use ``importlib.util.LazyLoader`` instead? -------------------------------------------------- @@ -1664,6 +1648,59 @@ From the discussion on :pep:`690` it is clear that this is a fairly contentious idea, although perhaps once we have wide-spread use of lazy imports this can be reconsidered. +Disallowing lazy imports inside ``with`` blocks +------------------------------------------------ + +An earlier version of this PEP proposed disallowing ``lazy import`` statements +inside ``with`` blocks, similar to the restriction on ``try`` blocks. The +concern was that certain context managers (like ``contextlib.suppress(ImportError)``) +could suppress import errors in confusing ways when combined with lazy imports. + +However, this restriction was rejected because ``with`` statements have much +broader semantics than ``try/except`` blocks. While ``try/except`` is explicitly +about catching exceptions, ``with`` blocks are commonly used for resource +management, temporary state changes, or scoping -- contexts where lazy imports +work perfectly fine. The ``lazy import`` syntax is explicit enough that +developers who write it inside a ``with`` block are making an intentional choice, +aligning with Python's "consenting adults" philosophy. For genuinely problematic +cases like ``with suppress(ImportError): lazy import foo``, static analysis +tools and linters are better suited to catch these patterns than hard language +restrictions. + +Additionally, forbidding explicit ``lazy import`` in ``with`` blocks would +create complex rules for how the global lazy imports flag should behave, +leading to confusing inconsistencies between explicit and implicit laziness. By +allowing ``lazy import`` in ``with`` blocks, the rule is simple: the global +flag affects all module-level imports except those in ``try`` blocks and wild +card imports, matching exactly what's allowed with explicit syntax. + +Forcing eager imports in ``with`` blocks under the global flag +--------------------------------------------------------------- + +Another rejected idea was to make imports inside ``with`` blocks remain eager +even when the global lazy imports flag is set to ``"all"``. The rationale was +to be conservative: since ``with`` statements can affect how imports behave +(e.g., by modifying ``sys.path`` or suppressing exceptions), forcing imports to +remain eager could prevent subtle bugs. However, this would create inconsistent +behavior where ``lazy import`` is allowed explicitly in ``with`` blocks, but +normal imports remain eager when the global flag is enabled. This inconsistency +between explicit and implicit laziness is confusing and hard to explain. + +The simpler, more consistent rule is that the global flag affects imports +everywhere that explicit ``lazy import`` syntax is allowed. This avoids having +three different sets of rules (explicit syntax, global flag behavior, and filter +mechanism) and instead provides two: explicit syntax rules match what the global +flag affects, and the filter mechanism provides escape hatches for edge cases. +For users who need fine-grained control, the filter mechanism +(``sys.set_lazy_imports_filter()``) already provides a way to exclude specific +imports or patterns. Additionally, there's no inverse operation: if the global +flag forces imports eager in ``with`` blocks but a user wants them lazy, there's +no way to override it, creating an asymmetry. + +In summary: imports in ``with`` blocks behave consistently whether marked +explicitly with ``lazy import`` or implicitly via the global flag, creating a +simple rule that's easy to explain and reason about. + Modification of the dict object ------------------------------- @@ -1868,55 +1905,38 @@ from a real dict in almost all cases, which is extremely difficult to achieve correctly. Any deviation from true dict behavior would be a source of subtle bugs. -Reifying lazy imports when ``globals()`` is called ---------------------------------------------------- +Automatically reifying on ``__dict__`` or ``globals()`` access +-------------------------------------------------------------- -Calling ``globals()`` returns the module's namespace dictionary without -triggering reification of lazy imports. Accessing lazy objects through the -returned dictionary yields the lazy proxy objects themselves. This is an -intentional design decision for several reasons: - -**The key distinction**: Adding a lazy import and calling ``globals()`` is the -module author's concern and under their control. However, accessing -``mod.__dict__`` from external code is a different scenario -- it crosses -module boundaries and affects someone else's code. Therefore, ``mod.__dict__`` -access reifies all lazy imports to ensure external code sees fully realized -modules, while ``globals()`` preserves lazy objects for the module's own -introspection needs. - -**Technical challenges**: It is impossible to safely reify on-demand when -``globals()`` is called because we cannot return a proxy dictionary -- this -would break common usages like passing the result to ``exec()`` or other -built-ins that expect a real dictionary. The only alternative would be to -eagerly reify all lazy imports whenever ``globals()`` is called, but this -behavior would be surprising and potentially expensive. - -**Performance concerns**: It is impractical to cache whether a reification -scan has been performed with just the globals dictionary reference, whereas -module attribute access (the primary use case) can efficiently cache -reification state in the module object itself. - -**Use case rationale**: The chosen design makes sense precisely because of -this distinction: adding a lazy import and calling ``globals()`` is your -problem to manage, while having lazy imports visible in ``mod.__dict__`` -becomes someone else's problem. By reifying on ``__dict__`` access but not on -``globals()``, we ensure external code always sees fully loaded modules while -giving module authors control over their own introspection. - -Note that three options were considered: +Three options were considered for how ``globals()`` and ``mod.__dict__`` should +behave with lazy imports: 1. Calling ``globals()`` or ``mod.__dict__`` traverses and resolves all lazy objects before returning. 2. Calling ``globals()`` or ``mod.__dict__`` returns the dictionary with lazy - objects present. + objects present (chosen). 3. Calling ``globals()`` returns the dictionary with lazy objects, but ``mod.__dict__`` reifies everything. -We chose the third option because it properly delineates responsibility: if -you add lazy imports to your module and call ``globals()``, you're responsible -for handling the lazy objects. But external code accessing your module's -``__dict__`` shouldn't need to know about your lazy imports -- it gets fully -resolved modules. +We chose option 2: both ``globals()`` and ``__dict__`` return the raw +namespace dictionary without triggering reification. This provides a clean, +predictable model where low-level introspection APIs don't trigger side +effects. + +Having ``globals()`` and ``__dict__`` behave identically creates symmetry and +a simple mental model: both expose the raw namespace view. Low-level +introspection APIs should not automatically trigger imports, which would be +surprising and potentially expensive. Real-world experience implementing lazy +imports in the standard library (such as the traceback module) showed that +automatic reification on ``__dict__`` access was cumbersome and forced +introspection code to load modules it was only examining. + +Option 1 (always reifying) was rejected because it would make ``globals()`` +and ``__dict__`` access surprisingly expensive and prevent introspecting the +lazy state of a module. Option 3 was initially considered to "protect" external +code from seeing lazy objects, but real-world usage showed this created more +problems than it solved, particularly for stdlib code that needs to introspect +modules without triggering side effects. Acknowledgements ================ From b18af22667cf744f3eca8c60292a06f51645a406 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Mon, 13 Oct 2025 20:30:59 +0200 Subject: [PATCH 2/6] Update pep-0810.rst Co-authored-by: T. Wouters --- peps/pep-0810.rst | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/peps/pep-0810.rst b/peps/pep-0810.rst index 6203ac88e0c..8bfbae54adc 100644 --- a/peps/pep-0810.rst +++ b/peps/pep-0810.rst @@ -465,13 +465,15 @@ the object. Calling ``dir()`` at the global scope will not reify the globals and calling ``dir(mod)`` will be special cased in ``mod.__dir__`` to avoid reification as well. -However, calling ``globals()`` or accessing a module's ``__dict__`` does **not** -trigger reification -- they return the module's dictionary, and accessing lazy -objects through that dictionary still returns lazy proxy objects that need to -be manually reified upon use. A lazy object can be resolved explicitly by -calling the ``resolve`` method. Other, more indirect ways of accessing -arbitrary globals (e.g. inspecting ``frame.f_globals``) also do **not** reify -all the objects. +However, calling ``globals()`` or accessing a module's ``__dict__`` does +**not** trigger reification -- they return the module's dictionary, and +accessing lazy objects through that dictionary still returns lazy proxy +objects that need to be manually reified upon use. A lazy object can be +resolved explicitly by calling the ``resolve`` method. Calling ``dir()`` at +the global scope will not reify the globals, nor will calling ``dir(mod)`` +(through special-casing in ``mod.__dir__``.) Other, more indirect ways of +accessing arbitrary globals (e.g. inspecting ``frame.f_globals``) also do +**not** reify all the objects. Example using ``globals()`` and ``__dict__``: From f5ec622943c84ea0838305357128030d3416d8b7 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Mon, 13 Oct 2025 20:31:13 +0200 Subject: [PATCH 3/6] Update pep-0810.rst Co-authored-by: T. Wouters --- peps/pep-0810.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/peps/pep-0810.rst b/peps/pep-0810.rst index 8bfbae54adc..888ea690825 100644 --- a/peps/pep-0810.rst +++ b/peps/pep-0810.rst @@ -461,9 +461,7 @@ immediately resolve all lazy objects (e.g. ``lazy from`` statements) that referenced the module. It **only** resolves the lazy object being accessed. Accessing a lazy object (from a global variable or a module attribute) reifies -the object. Calling ``dir()`` at the global scope will not reify the globals -and calling ``dir(mod)`` will be special cased in ``mod.__dir__`` to avoid -reification as well. +the object. However, calling ``globals()`` or accessing a module's ``__dict__`` does **not** trigger reification -- they return the module's dictionary, and From e30f83f5b8d0fb2079711e651e3e0649824a3b66 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Mon, 13 Oct 2025 20:31:29 +0200 Subject: [PATCH 4/6] Update pep-0810.rst Co-authored-by: T. Wouters --- peps/pep-0810.rst | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/peps/pep-0810.rst b/peps/pep-0810.rst index 888ea690825..b05615d17f0 100644 --- a/peps/pep-0810.rst +++ b/peps/pep-0810.rst @@ -1231,9 +1231,7 @@ Can I force reification of a lazy import without using it? ---------------------------------------------------------- Yes, individual lazy objects can be resolved by calling their ``resolve()`` -method. Note that accessing a module's ``__dict__`` or calling ``globals()`` -does **not** automatically reify lazy imports -- you'll see the lazy proxy -objects themselves, which you can then manually resolve if needed. +method. What's the difference between ``globals()`` and ``mod.__dict__`` for lazy imports? ---------------------------------------------------------------------------------- From 29332ffd0667d56f14edf9f33780ce750c8e2ae2 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Tue, 14 Oct 2025 13:18:00 +0200 Subject: [PATCH 5/6] Update pep-0810.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0810.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0810.rst b/peps/pep-0810.rst index b05615d17f0..14842576d19 100644 --- a/peps/pep-0810.rst +++ b/peps/pep-0810.rst @@ -461,7 +461,7 @@ immediately resolve all lazy objects (e.g. ``lazy from`` statements) that referenced the module. It **only** resolves the lazy object being accessed. Accessing a lazy object (from a global variable or a module attribute) reifies -the object. +the object. However, calling ``globals()`` or accessing a module's ``__dict__`` does **not** trigger reification -- they return the module's dictionary, and From 5c461161ea322ea2daa4d13020779c841c3a590b Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Tue, 14 Oct 2025 16:13:02 +0100 Subject: [PATCH 6/6] fixup! Update pep-0810.rst --- peps/pep-0810.rst | 37 ------------------------------------- 1 file changed, 37 deletions(-) diff --git a/peps/pep-0810.rst b/peps/pep-0810.rst index 14842576d19..dd6969a8b77 100644 --- a/peps/pep-0810.rst +++ b/peps/pep-0810.rst @@ -1233,36 +1233,6 @@ Can I force reification of a lazy import without using it? Yes, individual lazy objects can be resolved by calling their ``resolve()`` method. -What's the difference between ``globals()`` and ``mod.__dict__`` for lazy imports? ----------------------------------------------------------------------------------- - -Both ``globals()`` and ``mod.__dict__`` return the module's dictionary without -reifying lazy imports. Accessing lazy objects through either will yield lazy -proxy objects. This provides a consistent low-level API for introspection: - -.. code-block:: python - - # In your module: - lazy import json - - g = globals() - print(type(g['json'])) # - - d = __dict__ - print(type(d['json'])) # - - # From external code: - import sys - mod = sys.modules['your_module'] - d = mod.__dict__ - print(type(d['json'])) # - -Both ``globals()`` and ``__dict__`` expose the raw namespace view without -implicit side effects. This symmetry makes the behavior predictable: accessing -the namespace dictionary never triggers imports. If you need to ensure an -import is resolved, call the ``resolve()`` method explicitly or access the -attribute directly (e.g., ``json.dumps``). - Why not use ``importlib.util.LazyLoader`` instead? -------------------------------------------------- @@ -1665,13 +1635,6 @@ cases like ``with suppress(ImportError): lazy import foo``, static analysis tools and linters are better suited to catch these patterns than hard language restrictions. -Additionally, forbidding explicit ``lazy import`` in ``with`` blocks would -create complex rules for how the global lazy imports flag should behave, -leading to confusing inconsistencies between explicit and implicit laziness. By -allowing ``lazy import`` in ``with`` blocks, the rule is simple: the global -flag affects all module-level imports except those in ``try`` blocks and wild -card imports, matching exactly what's allowed with explicit syntax. - Forcing eager imports in ``with`` blocks under the global flag ---------------------------------------------------------------