ENH: Replace custom LazyITKModule with native PEP 562 lazy loading#6183
ENH: Replace custom LazyITKModule with native PEP 562 lazy loading#6183thewtex wants to merge 11 commits intoInsightSoftwareConsortium:mainfrom
Conversation
Replace the LazyITKModule sys.modules-swap orchestration in itk/__init__.py with a native PEP 562 implementation: module-level __getattr__ and __dir__ resolve symbols on first access, gated by a single threading.RLock. itkConfig.LazyLoading is no longer consulted; lazy is the only mode. multiprocessing.RLock is deliberately avoided so the multiprocessing start method stays pickable after `import itk`. Per-submodule LazyITKModule instances and the __init_<modulename>__.py discovery hook are preserved unchanged so the build stays green; Phase 03 will migrate them. itk_base_global_lazy_attributes (consumed by support/template_class._LoadModules) is still populated with the full set of owners per attribute, alongside the new first-owner-only _lazy_attribute_to_module that drives the top-level __getattr__. Validated via py_compile and a mock-driver smoke harness covering PEP 366 __package__, dir-without-load, lazy-load-with-cache, first-owner precedence, AttributeError on miss, dunder short-circuit, and a 6-thread first-touch race.
The new PEP 562 module-level __dir__ called the unqualified `set` name,
which Python resolves through `itk.__dict__`. Once any SWIG submodule is
loaded (e.g. ITKCommon on first `itk.Image` access), `itk.set` is
populated as an `itkTemplate` binding `std::set` and shadows the
builtin. Subsequent calls to `dir(itk)` then raised
`TemplateTypeError: itk.set is not wrapped for input type None`.
This is the same alias hazard that the existing _initialize_module
already guards against by importing `_builtin_set` from `builtins`.
Switch __dir__ to a set-literal form `{*globals().keys(), ...}` which
constructs through the C-level set type with no name lookup, so it is
unaffected by names introduced into module globals at lazy-load time.
Introduce itk.support._lazy_submodule with a builder that returns a plain types.ModuleType for itk.<Module> wired with module-level __getattr__ / __dir__ closures, registered in sys.modules['itk.<Module>'], and carrying a one-line __reduce_ex__ shim for pickle / cloudpickle by-name round-trip. This is the replacement target for the legacy LazyITKModule subclass; the next commit swaps the construction site in itk/__init__.py.
Replace the LazyITKModule(types.ModuleType) construction in _initialize_module() with the PEP 562 helper added in the previous commit. Each itk.<Module> namespace is now a plain types.ModuleType with __getattr__/__dir__ closures and a sys.modules registration, so cloudpickle round-trips by dotted name (Tests/lazy.py). The shared _lazy_load_lock is passed into the helper so top-level and per-submodule lazy loads continue to serialise on a single RLock. After this change __init__.py no longer references LazyITKModule; the class itself remains in support/lazy.py for now and will be removed in a follow-up commit.
The PEP 562 mechanism is now the only lazy-loading path so the legacy itkConfig.LazyLoading flag and its ITK_PYTHON_LAZYLOADING environment variable are no-ops. Drop the flag definition, the docstring paragraph documenting it, and the three test-side overrides that toggled it. Specifically: * Wrapping/Generators/Python/itkConfig.template.in.py: drop the LazyLoading docstring paragraph and the LazyLoading bool assignment driven by ITK_PYTHON_LAZYLOADING. * Wrapping/Generators/Python/itk/__init__.py: refresh a leading comment that used LazyLoading as the canonical example of a mutable itkConfig flag; use DefaultFactoryLoading instead so the example still names a real attribute. * Wrapping/Generators/Python/Tests/lazy.py: remove the itkConfig.LazyLoading=True override at the top of the test and refresh the now-stale "PEP 366 compliance of LazyITKModule" comment to reference the per-submodule namespaces that replaced the class. * Wrapping/Generators/Python/Tests/multiprocess_lazy_loading.py: remove the itkConfig.LazyLoading=True override and the now-unused import itkConfig. The descriptive header comments referencing the *concept* of lazy loading remain intact. * Modules/Filtering/ImageIntensity/wrapping/test/itkImageFilterNumPyInputsTest.py: remove the itkConfig.LazyLoading=False override and the associated import itkConfig (otherwise the test would raise AttributeError once the flag is gone).
Tests/nolazy.py was the eager-mode counterpart to Tests/lazy.py and forced itkConfig.LazyLoading=False to exercise the non-lazy import path. With the LazyLoading flag removed in the previous commit the PEP 562 lazy mechanism is the only import path, so the eager test is no longer meaningful and would raise AttributeError on import. Delete the file and its CMake registration. * Wrapping/Generators/Python/Tests/nolazy.py: deleted. * Wrapping/Generators/Python/Tests/CMakeLists.txt: drop the itk_python_add_test(NAME PythonNoLazyModule ...) block.
Remove the legacy custom-subclass-of-types.ModuleType implementation now that the itk package and its per-submodule namespaces resolve attributes via module-level __getattr__ / __dir__. - Delete Wrapping/Generators/Python/itk/support/lazy.py (LazyITKModule, ITKLazyLoadLock, _lazy_itk_module_reconstructor, not_loaded sentinel). - Drop support/lazy from ITK_PYTHON_SUPPORT_MODULES in Wrapping/Generators/Python/CMakeLists.txt so the wheel no longer installs the deleted file. A repo-wide grep confirms no live import of from itk.support import lazy, from itk.support.lazy, support.lazy, _lazy., or LazyITKModule remains.
Extend Tests/lazy.py beyond PEP 366 + cloudpickle to assert the three remaining contracts the PEP 562 lazy mechanism must preserve: - PEP 562 __dir__: "Image" appears in dir(itk) and dir(ITKCommon) but is absent from vars(itk) / vars(ITKCommon) until first access -- the lazy attribute map is enumerable without forcing a SWIG load. - Factory hook: under DefaultFactoryLoading, accessing a class whose module declares a needed factory (ITKIOImageBase -> ImageIO) grows ObjectFactoryBase.GetRegisteredFactories(); the disabled path is already covered by nodefaultfactories.py. - stdlib pickle: pickle.loads(pickle.dumps(itk.ITKCommon)) returns the same instance, exercising the per-submodule __reduce_ex__ shim wired by _make_itk_lazy_submodule (a bare types.ModuleType is unpicklable on CPython 3.12+ without it). Remove unused symbols: - _lazy_submodule.py: docstring named the deleted LazyITKModule subclass; rephrased to describe the module's behavior directly. - multiprocess_lazy_loading.py: header comments used CamelCase "LazyLoading" as a symbol; rephrased to "lazy module loading" / "PEP 562 __getattr__ hook" while keeping the threading contract the test still enforces. - Wrapping/Generators/Python/CMakeLists.txt: directory-layout comment listed itk(...|LazyLoading|...).py as a static support file; updated to the actual filenames currently shipped. Comment- and docstring-only; no functional change. PythonLazyModule, PythonMultiprocessLazyLoad, and PythonLazyLoadingImage all still pass.
Add a section to the ITK 6 migration guide describing the removal of itkConfig.LazyLoading and the ITK_PYTHON_LAZYLOADING environment variable, which were dropped when the Python lazy-loading mechanism was rewritten on top of PEP 562. The section distinguishes per-symbol behavior: assignment to itkConfig.LazyLoading is silently ignored, any subsequent read raises AttributeError, and the environment variable is no longer consulted.
|
| Filename | Overview |
|---|---|
| Wrapping/Generators/Python/itk/init.py | Rewritten with module-level PEP 562 getattr/dir, a single threading.RLock, and correct double-check locking; _initialize_module now builds the flat _lazy_attribute_to_module map; eager-mode and reload-guard branches removed. |
| Wrapping/Generators/Python/itk/support/_lazy_submodule.py | New file providing _make_itk_lazy_submodule; overall logic is sound but loaded_modules is dead code (written but never read). |
| Wrapping/Generators/Python/itk/support/lazy.py | Deleted (181 lines): LazyITKModule subclass, ITKLazyLoadLock, _lazy_itk_module_reconstructor, and not_loaded sentinel; all consumers updated. |
| Wrapping/Generators/Python/itkConfig.template.in.py | Removes LazyLoading bool and ITK_PYTHON_LAZYLOADING env-var reader; DefaultFactoryLoading is preserved unchanged. |
| Documentation/docs/migration_guides/itk_6_migration_guide.md | Adds a well-written migration section, but the inline comment '# this read now raises AttributeError' is incorrect for the two-line example shown (the preceding assignment prevents the error). |
| Wrapping/Generators/Python/Tests/lazy.py | Extended with PEP 562 dir assertions, factory-hook test, and stdlib pickle round-trip for itk.ITKCommon; itkConfig.LazyLoading override removed. |
| Wrapping/Generators/Python/Tests/nolazy.py | Deleted — the eager-mode branch it exercised no longer exists; corresponding PythonNoLazyModule CMake test also removed. |
| Wrapping/Generators/Python/Tests/multiprocess_lazy_loading.py | Drops itkConfig.LazyLoading = True override and updates comments; threading test logic is unchanged. |
| Modules/Filtering/ImageIntensity/wrapping/test/itkImageFilterNumPyInputsTest.py | Removes itkConfig.LazyLoading = False guard that is no longer meaningful. |
| Wrapping/Generators/Python/CMakeLists.txt | Updates support file list from support/lazy to support/_lazy_submodule. |
Sequence Diagram
sequenceDiagram
participant Caller
participant itk as "itk module __getattr__"
participant Sub as "itk.ITKCommon __getattr__"
participant Base as "itk.support.base"
participant Lock as "threading.RLock"
Caller->>itk: "itk.Image (first access)"
itk->>Lock: "acquire()"
itk->>Base: "itk_load_swig_module(ITKCommon, namespace)"
Base-->>itk: "namespace populated"
itk->>itk: "globals().update(namespace)"
itk->>Lock: "release()"
itk-->>Caller: "Image class"
Caller->>itk: "itk.Image (subsequent access)"
Note over itk: "Found in __dict__, __getattr__ skipped"
itk-->>Caller: "Image class via fast path"
Caller->>Sub: "itk.ITKCommon.Image (first access)"
Sub->>Lock: "acquire() shared RLock"
Sub->>Base: "itk_load_swig_module(ITKCommon, namespace)"
Base-->>Sub: "namespace populated"
Sub->>Sub: "m.__dict__.update(namespace)"
Sub->>Lock: "release()"
Sub-->>Caller: "Image class"
Caller->>Sub: "pickle.dumps(itk.ITKCommon)"
Sub-->>Caller: "(importlib.import_module, itk.ITKCommon)"
Caller->>Sub: "pickle.loads resolves via sys.modules"
Sub-->>Caller: "same itk.ITKCommon instance"
Comments Outside Diff (1)
-
Documentation/docs/migration_guides/itk_6_migration_guide.md, line 33-43 (link)Misleading
AttributeErrorcomment in migration guideThe inline comment
# this read now raises AttributeErroris incorrect for the specific two-line sequence shown. Because the precedingitkConfig.LazyLoading = Falseassignment successfully sets the attribute on the module object (Python module attributes are mutable), the subsequentif itkConfig.LazyLoading:read returnsFalse— it does not raiseAttributeError. TheAttributeErroronly arises when readingLazyLoadingwithout having first written it (i.e., on a freshimport itkConfigwith no prior assignment). The current example would mislead users who try to reproduce the error to confirm they need to migrate.
Reviews (1): Last reviewed commit: "DOC: Note itkConfig.LazyLoading removal ..." | Re-trigger Greptile
Via pixi run pre-commit-run
|
Addressed Greptile's two P2 findings in ca131c5:
|
Two follow-ups from the Greptile review of PR 6183: * Drop the dead `loaded_modules: set[str]` declaration and its `loaded_modules.add(target)` write in `_make_itk_lazy_submodule`. The set was a vestige of `LazyITKModule.__getstate__`/`__setstate__`; the new `__reduce_ex__` shim delegates to `importlib.import_module` and never consults it. * Reword the migration-guide example for `itkConfig.LazyLoading`. The previous inline comment claimed the second line raised `AttributeError`, which was wrong: the preceding assignment had already set the attribute, so the read returned `False`. Split the example into separately commented write and read patterns and clarify that the `AttributeError` only occurs on a fresh `import itkConfig` with no prior assignment.
ca131c5 to
9e33a81
Compare
The custom
LazyITKModule(types.ModuleType)subclass that poweredITK's Python lazy-loading layer is replaced with the standard-library
PEP 562 (Python 3.7+) module-level
__getattr__/__dir__protocol.The same observable contract is preserved: lazy first-touch resolution,
thread-safe first-load via a single recursive lock, factory-loading
hook gated on
itkConfig.DefaultFactoryLoading, PEP 366__package__,and pickle / cloudpickle round-trip identity through
sys.modulesregistration.
Motivation:
LazyITKModule.__getattribute__intercepted everyattribute read just to check a sentinel; PEP 562 only fires on
miss, so steady-state reads now hit the standard module fast path
for free.
ITKLazyLoadLockcombinedthreading.RLockandmultiprocessing.RLock. Constructing the multiprocessing half pinnedthe multiprocessing start method at
import itktime, breakingdownstream callers that wanted
multiprocessing.set_start_method(...)after import. The new implementation uses only
threading.RLock;process boundary isolation already handles the cross-process case
(each child process re-runs
itk/__init__.pyagainst its ownsys.modules)._lazy_itk_module_reconstructor),__getstate__,__setstate__, and the eager-mode dispatch branchare deleted; standard module pickling and
sys.modules['itk.<Mod>']registration cover the round-trip with a single 3-line
__reduce_ex__shim per synthetic submodule (vanillapickle.dumps(types.ModuleType(...))raises on CPython 3.12+).What changes:
Wrapping/Generators/Python/itk/__init__.pyrewritten withmodule-level
__getattr__/__dir__(PEP 562) and a singlethreading.RLock. Reload-guard branch and eager-mode branchremoved.
Wrapping/Generators/Python/itk/support/_lazy_submodule.pyexports
_make_itk_lazy_submodule(module_name, lazy_attributes, lazy_load_lock)returning a plaintypes.ModuleTypecarryingPEP 562
__getattr__/__dir__/__reduce_ex__.Wrapping/Generators/Python/itk/support/lazy.pydeleted (181 lines:custom subclass, lock, reconstructor, sentinel).
Wrapping/Generators/Python/Tests/nolazy.pydeleted; the eagerbranch it exercised no longer exists.
itkConfig.LazyLoadingand theITK_PYTHON_LAZYLOADINGenvironment-variable reader removed from
itkConfig.template.in.py. Lazy is now the single mode.Tests/lazy.py,Tests/multiprocess_lazy_loading.py, andModules/Filtering/ImageIntensity/wrapping/test/itkImageFilterNumPyInputsTest.pydrop
itkConfig.LazyLoading = ...overrides.Tests/lazy.pyextended to cover PEP 562__dir__(visiblebefore load), the factory-loading hook, and stdlib
pickleround-trip on
itk.ITKCommon.Net diff: 10 files changed, +267 / −335 (net -68 lines).
Validated end-to-end with
pixi run --as-is build-pythonandpixi run --as-is ctest --test-dir build-python -R Python -E "PythonNoLazyModule"— 172/172 tests pass, including thePythonLazyModule,PythonMultiprocessLazyLoad, andPythonGILReleaseSafetyTestregression triplet.Lazy-only is now the single mode: callers that previously set
itkConfig.LazyLoading = Falseshould remove the assignment (itwill raise
AttributeError).