diff --git a/peps/pep-0810.rst b/peps/pep-0810.rst index 4345a94a1b1..c8401f5ecd1 100644 --- a/peps/pep-0810.rst +++ b/peps/pep-0810.rst @@ -246,7 +246,8 @@ The soft keyword is only allowed at the global (module) level, **not** inside functions, class bodies, with ``try``/``with`` blocks, or ``import *``. Import statements that use the soft keyword are *potentially lazy*. Imports that can't be lazy are unaffected by the global lazy imports flag, and instead are -always eager. +always eager. Additionally, ``from __future__ import`` statements cannot be +lazy. Examples of syntax errors: @@ -273,6 +274,9 @@ Examples of syntax errors: # SyntaxError: lazy from ... import * is not allowed lazy from json import * + # SyntaxError: lazy from __future__ import is not allowed + lazy from __future__ import annotations + Semantics --------- @@ -305,7 +309,7 @@ sequence of fully qualified module names (strings) to make *potentially lazy* that module are also lazy, but not necessarily imports of sub-modules. The normal (non-lazy) import statement will check the global lazy imports -flag. If it is "enabled", all imports are *potentially lazy* (except for +flag. If it is "all", all imports are *potentially lazy* (except for imports that can't be lazy, as mentioned above.) Example: @@ -318,7 +322,7 @@ Example: result = json.dumps({"hello": "world"}) print('json' in sys.modules) # True -If the global lazy imports flag is set to "disabled", no *potentially lazy* +If the global lazy imports flag is set to "none", no *potentially lazy* import is ever imported lazily, and the behavior is equivalent to a regular import statement: the import is *eager* (as if the lazy keyword was not used). @@ -565,20 +569,27 @@ After several calls, ``LOAD_GLOBAL`` specializes to ``LOAD_GLOBAL_MODULE``: Lazy imports filter ------------------- -This PEP adds two new functions to the ``sys`` module to manage the lazy -imports filter: +This PEP adds the following new functions to manage the lazy imports filter: + +* ``importlib.set_lazy_imports_filter(func)`` - Sets the filter function. If + ``func=None`` then the import filter is removed. The ``func`` parameter must + have the signature: ``func(importer: str, name: str, fromlist: tuple[str, ...] | None) -> bool`` -* ``sys.set_lazy_imports_filter(func)`` - Sets the filter function. The - ``func`` parameter must have the signature: ``func(importer: str, name: str, - fromlist: tuple[str, ...] | None) -> bool`` +* ``importlib.get_lazy_imports_filter()`` - Returns the currently installed + filter function, or ``None`` if no filter is set. -* ``sys.get_lazy_imports_filter()`` - Returns the currently installed filter - function, or ``None`` if no filter is set. +* ``importlib.set_lazy_imports(enabled=None, /)`` - Programmatic API for + controlling lazy imports at runtime. The ``enabled`` parameter can be + ``None`` (respect ``lazy`` keyword only), ``True`` (force all imports to be + potentially lazy), or ``False`` (force all imports to be eager). The filter function is called for every potentially lazy import, and must return ``True`` if the import should be lazy. This allows for fine-grained control over which imports should be lazy, useful for excluding modules with -known side-effect dependencies or registration patterns. +known side-effect dependencies or registration patterns. The filter function +is called at the point of execution of the lazy import or lazy from import +statement, not at the point of reification. The filter function may be +called concurrently. The filter mechanism serves as a foundation that tools, debuggers, linters, and other ecosystem utilities can leverage to provide better lazy import @@ -618,7 +629,7 @@ Example: return True # Allow lazy import # Install the filter - sys.set_lazy_imports_filter(exclude_side_effect_modules) + importlib.set_lazy_imports_filter(exclude_side_effect_modules) # These imports are checked by the filter lazy import data_processor # Filter returns True -> stays lazy @@ -638,31 +649,33 @@ The global lazy imports flag can be controlled through: * The ``-X lazy_imports=`` command-line option * The ``PYTHON_LAZY_IMPORTS=`` environment variable -* The ``sys.set_lazy_imports(mode)`` function (primarily for testing) +* The ``importlib.set_lazy_imports(mode)`` function (primarily for testing) Where ```` can be: -* ``"default"`` (or unset): Only explicitly marked lazy imports are lazy +* ``"normal"`` (or unset): Only explicitly marked lazy imports are lazy -* ``"enabled"``: All module-level imports (except in ``try`` or ``with`` +* ``"all"``: All module-level imports (except in ``try`` or ``with`` blocks and ``import *``) become *potentially lazy* -* ``"disabled"``: No imports are lazy, even those explicitly marked with +* ``"none"``: No imports are lazy, even those explicitly marked with ``lazy`` keyword -When the global flag is set to ``"enabled"``, all imports at the global level +When the global flag is set to ``"all"``, all imports at the global level of all modules are *potentially lazy* **except** for those inside a ``try`` or ``with`` block or any wild card (``from ... import *``) import. -If the global lazy imports flag is set to ``"disabled"``, no *potentially +If the global lazy imports flag is set to ``"none"``, no *potentially lazy* import is ever imported lazily, the import filter is never called, and the behavior is equivalent to a regular ``import`` statement: the import is *eager* (as if the lazy keyword was not used). -Python code can run the :func:`!sys.set_lazy_imports` function to override +Python code can run the :func:`!importlib.set_lazy_imports` function to override the state of the global lazy imports flag inherited from the environment or CLI. This is especially useful if an application needs to ensure that all imports -are evaluated eagerly, via ``sys.set_lazy_imports('disabled')``. +are evaluated eagerly, via ``importlib.set_lazy_imports('none')``. +Alternatively, :func:`!importlib.set_lazy_imports` can be used with boolean +values for programmatic control. Backwards Compatibility @@ -759,7 +772,7 @@ The `pyperformance suite`_ confirms the implementation is performance-neutral. Filter function performance ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The filter function (set via ``sys.set_lazy_imports_filter()``) is called for +The filter function (set via ``importlib.set_lazy_imports_filter()``) is called for every *potentially lazy* import to determine whether it should actually be lazy. When no filter is set, this is simply a NULL check (testing whether a filter function has been registered), which is a highly predictable branch that @@ -856,9 +869,18 @@ see: - `Inside HRT's Python Fork `__ (Hudson River Trading) +- `Create an On-Demand Initializer for PySide + `__ + (Qt for Python/PySide) - Christian Tismer's implementation of lazy + initialization for PySide6 based on ideas from :pep:`690`, showing 10-20% + startup time improvement for PySide applications. This demonstrates the + particular value of lazy imports for frameworks with extensive + initialization at import time. The benefits scale with codebase complexity: the larger and more -interconnected the codebase, the more dramatic the improvements. +interconnected the codebase, the more dramatic the improvements. The +PySide implementation particularly highlights how frameworks with heavy +initialization overhead can benefit significantly from opt-in lazy loading. Typing and tools ---------------- @@ -873,6 +895,10 @@ Security Implications ===================== There are no known security vulnerabilities introduced by lazy imports. +Security-sensitive tools that need to ensure all imports are evaluated eagerly +can use :func:`!importlib.set_lazy_imports` with ``enabled=False`` to force +eager evaluation, or use :func:`!importlib.set_lazy_imports_filter` for fine-grained +control. How to Teach This ================= @@ -888,6 +914,14 @@ profiling to understand the import time overhead in their codebase and mark the necessary imports as ``lazy``. In addition, developers can mark imports that will only be used for type annotations as ``lazy``. +Additional documentation will be added to the Python documentation, including +guidance, a dedicated how-to guide, and updates to the import system +documentation covering: identifying slow-loading modules with profiling tools +(such as ``-X importtime``), migration strategies for existing codebases, best +practices for avoiding common pitfalls with import-time side effects, and +patterns for using lazy imports effectively with type annotations and circular +imports. + Below is guidance on how to best take advantage of lazy imports and how to avoid incompatibilities: @@ -1214,14 +1248,14 @@ exclude specific modules that are known to have problematic side effects: return False # Import eagerly return True # Allow lazy import - sys.set_lazy_imports_filter(my_filter) + importlib.set_lazy_imports_filter(my_filter) The filter function receives the importer module name, the module being imported, and the fromlist (if using ``from ... import``). Returning ``False`` forces an eager import. -Alternatively, set the global mode to ``"disabled"`` via ``-X -lazy_imports=disabled`` to turn off all lazy imports for debugging. +Alternatively, set the global mode to ``"none"`` via ``-X +lazy_imports=none`` to turn off all lazy imports for debugging. Can I use lazy imports inside functions? ---------------------------------------- @@ -1415,11 +1449,105 @@ No, future imports can't be lazy because they're parser/compiler directives. It's technically possible for the runtime behavior to be lazy but there's no real value in it. -Why you chose ``lazy`` as the keyword name? -------------------------------------------- +Why did you choose ``lazy`` as the keyword name? +------------------------------------------------ Not "why"... memorize! :) +Deferred Ideas +============== + +The following ideas have been considered but are deliberately deferred to focus +on delivering a stable, usable core feature first. These may be considered for +future enhancements once we have real-world experience with lazy imports. + +Alternative syntax and ergonomic improvements +---------------------------------------------- + +Several alternative syntax forms have been suggested to improve ergonomics: + +* **Type-only imports**: A specialized syntax for imports used exclusively in + type annotations (similar to the ``type`` keyword in other contexts) could be + added, such as ``type from collections.abc import Sequence``. This would make + the intent clearer than using ``lazy`` for type-only imports and would signal + to readers that the import is never used at runtime. However, since ``lazy`` + imports already solve the runtime cost problem for type annotations, we prefer + to start with the simpler, more general mechanism and evaluate whether + specialized syntax adds sufficient value after gathering usage data. + +* **Block-based syntax**: Grouping multiple lazy imports in a block, such as: + + .. code-block:: python + + as lazy: + import foo + from bar import baz + + This could reduce repetition when marking many imports as lazy. However, it + would require introducing an entirely new statement form (``as lazy:`` blocks) + that doesn't fit into Python's existing grammar patterns. It's unclear how + this would interact with other language features or what the precedent would + be for similar block-level modifiers. This approach also makes it less clear + when scanning code whether a particular import is lazy, since you must look at + the surrounding context rather than the import line itself. + +While these alternatives could provide different ergonomics in certain contexts, +they share similar drawbacks: they would require introducing new statement +forms or overloading existing syntax in non-obvious ways, and they open the +door to many other potential uses of similar syntax patterns that would +significantly expand the language. We prefer to start with the explicit +``lazy import`` syntax and gather real-world feedback before considering +additional syntax variations. Any future ergonomic improvements should be +evaluated based on actual usage patterns rather than speculative benefits. + +Automatic lazy imports for ``if TYPE_CHECKING`` blocks +------------------------------------------------------- + +A future enhancement could automatically treat all imports inside +``if TYPE_CHECKING:`` blocks as lazy: + +.. code-block:: python + + from typing import TYPE_CHECKING + + if TYPE_CHECKING: + from foo import Bar # Could be automatically lazy + +However, this would require significant changes to make this work at compile +time, since ``TYPE_CHECKING`` is currently just a runtime variable. The +compiler would need special knowledge of this pattern, similar to how +``from __future__ import`` statements are handled. Additionally, making +``TYPE_CHECKING`` a built-in would be required for this to work reliably. +Since ``lazy`` imports already solve the runtime cost problem for type-only +imports, we prefer to start with the explicit syntax and evaluate whether +this optimization adds sufficient value. + +Module-level lazy import mode +------------------------------ + +A module-level declaration to make all imports in that module lazy by default: + +.. code-block:: python + + from __future__ import lazy_imports + import foo # Automatically lazy + +This was discussed but deferred because it raises several questions. Using +``from __future__ import`` implies this would become the default behavior in a +future Python version, which is unclear and not currently planned. It also +raises questions about how such a mode would interact with the global flag and +what the transition path would look like. The current explicit syntax and +``__lazy_modules__`` provide sufficient control for initial adoption. + +Package metadata for lazy-safe declarations +-------------------------------------------- + +Future enhancements could allow packages to declare in their metadata whether +they are safe for lazy importing (e.g., no import-time side effects). This +could be used by the filter mechanism or by static analysis tools. The current +filter API is designed to accommodate such future additions without requiring +changes to the core language specification. + Alternate Implementation Ideas ============================== @@ -1559,6 +1687,30 @@ to add more specific re-enabling mechanisms later, when we have a clearer picture of real-world use and patterns, than it is to remove a hastily added mechanism that isn't quite right. +Using a decorator syntax for lazy imports +------------------------------------------ + +A decorator-based syntax could mark imports as lazy: + +.. code-block:: python + + @lazy + import json + + @lazy + from foo import bar + +This approach was rejected because it introduces too many open questions and +complications. Decorators in Python are designed to wrap and transform callable +objects (functions, classes, methods), not statements. Allowing decorators on +import statements would open the door to many other potential statement +decorators (``@cached``, ``@traced``, ``@deprecated``, etc.), significantly +expanding the language's syntax in ways we don't want to explore. Furthermore, +this raises the question of where such decorators would come from: they would +need to be either imported or built-in, creating a bootstrapping problem for +import-related decorators. This is far more speculative and generic than the +focused ``lazy import`` syntax. + Using a context manager instead of a new soft keyword -----------------------------------------------------