Replace pkg_resources #7943
Merged
Replace pkg_resources #7943
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rokm
force-pushed
the
use_importlib_metadata
branch
from
718a3a1
to
9e93fc3
Compare
Import `importlib.metadata` as `importlib_metadata` if stdlib version is recent enough (python >= 3.10). Otherwise import the `importlib_metadata`, but explicitly verify the version of `importlib-metadata` dist, which must be >= 4.6. This version is needed for extended functionality, such as `group` and `name` keyword arguments having been added to the `entry_points()` function. Add `importlib-metadata >= 4.6` to requirements if python < 3.10. Add `packaging` >= 20.0 to requirements; used for parsing of version and requirements strings.
Replace the `PYINSTALLER_NO_PYWIN32_FAILURE` environment variable set by `setup.py` into more general `_PYINSTALLER_SETUP_PY`, which now covers the following run-time dependencies in the `compat` module: * `importlib_metadata` on python < 3.10 * `pywin32-ctypes` on Windows This "setup.py mode" prevents `compat` from trying to import afore-mentioned run-time dependencies, which may not be present when PyInstaller wheels are being built (nor are required for the `compat` functionality used by the `setup.py`).
Switch to `importlib.metadata` (or `importlib_metadata`) for entry point enumeration during the discovery of hook directories.
Add two test that try to query metadata that was collected from (non-zipped) .egg. In such cases, we collect the EGG-INFO directory containing the metadata under its original parent directory (the package_name-version.egg) to prevent clashes between metadata from multiple .egg sources. The first test queries the metadata using `pkg_resources`, while the second uses `importlib.metadata` / `importlib_metadata`.
It seems that `pkg_resources` is able to discover metadata collected from eggs as long as the parent directory of the .egg directory is in `sys.path`. This is automatically true in our case, because we collect the `package_name-version.egg` directories (that contain `EGG-INFO` metadata directories) into `sys._MEIPASS`. In contrast, `importlib.metadata` does not seem to pick up such metadata unless the `.egg` directories themselves are in `sys.path`. So have the bootstrap script scan for directories with `.egg` suffix and append them so `sys.path`.
Port metadata collection (`copy_metadata` hook utility function) from `pkg_resources` to `importlib.metadata` and `packaging`. The latter is required for parsing package requirements and evaluating markers to exclude the requirements that are not applicable to the running python version and/or platform. Simplify the metadata destination handling code; the only special case we need to care about is EGG-INFO directory in .egg directories, where we need to preserve the parent directory instead of collecting into top-level application directory. `importlib.metadata` seems to properly resolve the path to egg-info files used by Debian/Ubuntu packages, so we can also remove the legacy name/path handling. Have the remaining hook utility functions that make use of `pkg_resources` import it locally, so we can port them one-by-one.
Port the `get_installer` helper from `pkg_resources` to `importlib.metadata`. Requires python >= 3.10 or equivalent `importlib-metadata` (>= 4.6) due to use of the `packages_distributions()` convenience function.
Since the `copy_metadata` calls in `_metadata_from` now use `importlib.metadata` instead of `pkg_resources`, we now need to handle the `importlib.metadata.PackageNotFoundError` instead of `pkg_resources.PackageNotFoundError`. Use `compat.importlib_metadata` to properly resolve `importlib.metadata` vs `importlib_metadata` regardless if the latter is available on python > 3.10, where our `compat` prefers the stdlib version.
The only use for `modulegraph.replacePackage` is for the `_xmplus` module from python2-era PyXML package. Remove the function and associated code.
In contemporary python, namespace packages should be handled without having to resort to parsing the *-nspkg.pth files.
Rework the support for legacy setuptools-based namespace packages and port it to `importlib.metadata`. The legacy namespace packages are regular packages with `__init__.py` that calls deprecated `pkg_resources.declare_namespace(__name__)`. We could get rid of this altogether once we are fine with not supporting legacy namespace packages that are split across multiple locations. Currently, our test suite contains such an example (e.g., `test_import.py::test_nspkg1`). The code has been refactored so that dropping support will also be easier. To properly support legacy namespace sub-packages that are split across different locations, we need to scan the metadata of all dists, similarly to how it was done by the original `pkg_resources` based implementation. However, within the life-time of a module graph instance, we can perform the scan only once, and cache the results.
Port the entry-point scan for test discovery from `pkg_resources` to `importlib.metadata`.
rokm
force-pushed
the
use_importlib_metadata
branch
3 times, most recently
from
4c6d225
to
1a08985
Compare
bwoodsend
reviewed
Sep 20, 2023
Replace `is_module_satisfies` with `check_requirement`, which uses `packaging` to parse the requirement and check the version against the optional specifier(s), and `importlib.metadata` to query the dist version. Keep `is_module_satisfies` as an alias for `check_requirements` to maintain compatibility with existing hooks (from the contrib hooks repository) that call the function with only the `requirement` string. The calls with additional `version` and `version_attr` arguments will not work anymore, nor will the fall back to checking module's version when the specified requirement is not satisfied. Have the @requires test decorator use `check_requirement`. Update hooks in this repository to use `check_requirement` instead of `is_module_satisfies`. Update Qt hook utilities, where we need to account for the fact that `check_requirement` does not attempt to fall back to checking module's version (which in this case was undesirable and had to be worked around).
Have `collect_all` look up the dist name for the given package name before attempting to call `copy_metadata` and `requirements_for_package`. This fixes the collection (and spurious warnings) for cases when dist name does not match the importable/package name.
Adjust the behavior of `collect_data_files` when the `include_py_files` flag is enabled: collect only `.py` and `.pyc` files, and never collect `.pyc` files from the `__pycache__` directory. Update the description to note that module collection mode is preferable way of ensuring that source .py files are collected (in addition or in lieu to byte-compiled modules in PYZ). Adjust the `test_collect_data_all_included` accordingly.
Remove `requirements_for_package` hook utility, which was used by `collect_all`. Consequently, the latter does not include the top-level modules of metadata-declared requirements among the returned hidden imports anymore.
rokm
force-pushed
the
use_importlib_metadata
branch
from
1a08985
to
86ae9b2
Compare
bwoodsend
approved these changes
Sep 21, 2023
|
Guess we can close #6458 then too. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replace the use of deprecated
pkg_resourcesthroughout the codebase with combination ofimportlib.metadata/importlib_metadataandpackaging.Closes #6458.