[maintenance] lazy load dpnp.tensor/dpnp and prepare for array_api lazy importing #2509
Conversation
…rn-intelex into dev/lazy_load
sklearnex/utils/validation.py
Outdated
| try: | ||
| too_small = X.size < 32768 | ||
| except TypeError: | ||
| too_small = math.prod(X.shape) < 32768 |
There was a problem hiding this comment.
Could also use np.prod, since numpy is already imported throughout the codebase.
There was a problem hiding this comment.
https://github.com/scikit-learn/scikit-learn/blob/73a8a656b8df6d02cf88ef8f9cf98373a3f42051/sklearn/utils/_array_api.py#L215 Not entirely sure how numpy would interact with pytorch in that case. Could check that if you want, but its following the precedent set by sklearn itself
There was a problem hiding this comment.
Pull Request Overview
This PR implements lazy-loading for dpnp, dpctl.tensor, and array_api support to mitigate import-time performance overhead and decouple heavy dependencies from estimator initialization. Key changes include refactoring import paths from deprecated helper modules to a new _third_party module, updating functions in several modules (e.g. logistic regression, ensemble forests, device offload) to use lazy evaluation, and adding a test in test_common.py to verify that only numpy and pandas are loaded on estimator import.
Reviewed Changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/run_examples.py | Updated import path for dpctl availability. |
| sklearnex/tests/test_memory_usage.py | Removed unused dpctl/dpnp imports from dpep_helpers. |
| sklearnex/tests/test_common.py | Added new test to validate lazy import behavior for data frameworks. |
| onedal/ensemble/_forest.py | Replaced get_unique_values_with_dpep with new inline unique extraction. |
| onedal/_device_offload.py | Updated handling of output conversion and lazy data extraction. |
| onedal/utils/_third_party.py | Introduced new helper functions for lazy importing and third-party checks. |
| onedal/utils/_array_api.py | Added caching mechanism for mapping array types to SYCL namespaces. |
| onedal/tests/utils/_dataframes_support.py | Modified dpnp availability checks using try/except. |
| onedal/linear_model/logistic_regression.py | Updated unique value extraction using _get_sycl_namespace. |
| onedal/datatypes/* | Various adjustments to imports and data conversion functions. |
| onedal/common/tests/test_sycl.py | Updated dpctl availability checks to use new module. |
Comments suppressed due to low confidence (1)
onedal/ensemble/forest.py:321
- The variable 'xp' is used without being defined. It should be initialized by extracting the array namespace from X (e.g. by adding '_, xp, _ = _get_sycl_namespace(X)' before using xp.unique).
self.classes_ = xp.unique(y)
Low confidence recommendation for |
|
/intelci: run |
|
Will rerun because of private-CI issues. |
|
/intelci: run |
2 similar comments
|
/intelci: run |
|
/intelci: run |
|
We are having private CI infrastructure issues, especially with the GPU runners, so this will be on hold until those run properly. |
|
/intelci: run |
1 similar comment
|
/intelci: run |
Description
Dpctl and dpnp are quasi-dependencies which will silently error out if not installed. This is done at import time throughout the codebase, meaning that it is mixed into the codebase in a difficult manner. As the number of supported data frameworks are increased, such a strategy is unsustainable. Lazy loading of the necessary packages must be done, as the load time of follow-on frameworks like PyTorch are non-negligible (>1s). If we were to follow the same strategy, load times of sklearnex would be even longer even if pytorch isn't used but is available. This will compound as we would add framework support. Cleanly separating and isolating their use is necessary.
Therefore we need to first move dpnp and dpctl.tensor support to a lazy loading approach which will then be extended by follow-on frameworks. The next step will be pytorch queue extraction, which will require this infrastructure.
The strategy will follow that of
array_api_compatwhich can check for namespaces without importing the actual modules, and for the direct use of the frameworks, a depedency injection + monkeypatching scheme is used with decoratorlazy_import.A new test is added to
test_common.pywhich verifies for all estimators that when they are imported, that they nor the underlying infrastructure actively load data frameworks which aren'tnumpyorpandas.NOTE TO REVIEWERS: Let me know if I should do a performance benchmarks for this.
PR should start as a draft, then move to ready for review state after CI is passed and all applicable checkboxes are closed.
This approach ensures that reviewers don't spend extra time asking for regular requirements.
You can remove a checkbox as not applicable only if it doesn't relate to this PR in any way.
For example, PR with docs update doesn't require checkboxes for performance while PR with any change in actual code should have checkboxes and justify how this code change is expected to affect performance (or justification should be self-evident).
Checklist to comply with before moving PR from draft:
PR completeness and readability
Testing
Performance