Switch to importlib-metadata #199

asottile · 2019-04-07T00:30:51Z

This attempts to switch the backing metadata looking from pkg_resources (slow) to importlib-metadata (fast)

I had to expose one derived property on top of the importlib_metadata Distribution class to make it look like the pkg_resources one -- but otherwise this should be drop-in.

At least for pytest this isn't enough to get the full performance boost since pytest imports pkg_resources itself -- but baby steps! I'll be following up with the pytest branch in pytest-dev/pytest#5063

asottile · 2019-04-07T00:32:05Z

This is WIP:

pytest uses dist.project_name which isn't a thing with importlib-metadata Distribution objects

need to make a shim? dist.metadata['name'] ~= dist.project_name

asottile · 2019-04-20T23:31:17Z

This is no longer WIP, technically as far as I can tell this version is compatible with pytest as-is (due to the Distribution shim)

asottile · 2019-04-21T01:05:07Z

switched the last commit from python -m tox to tox directly, it looks like until pluggy is the src layout tox is importing it directly 🤦‍♂️

codecov-io · 2019-04-21T01:16:58Z

Codecov Report

Merging #199 into master will decrease coverage by 0.08%.
The diff coverage is 92.3%.

@@            Coverage Diff             @@
##           master     #199      +/-   ##
==========================================
- Coverage   92.65%   92.57%   -0.09%     
==========================================
  Files          14       14              
  Lines        1553     1549       -4     
  Branches      115      116       +1     
==========================================
- Hits         1439     1434       -5     
- Misses         96       98       +2     
+ Partials       18       17       -1

Impacted Files	Coverage Δ
testing/test_pluginmanager.py	`95.67% <100%> (-0.12%)`	⬇️
pluggy/manager.py	`94.56% <84.21%> (-0.36%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 59b3cf2...0706fba. Read the comment docs.

jayvdb · 2019-05-02T07:44:27Z

testing/test_pluginmanager.py

    num = pm.load_setuptools_entrypoints("hello")
    assert num == 0  # no plugin loaded by this call


-def test_load_setuptools_version_conflict(monkeypatch, pm):


runtime checking of dependency versions is not possible without the old and slow pkg_resources? ;-(

hmmm I guess that's not the way I saw that. any time I've dealt with pkg_resources the side-effect of looking up package metadata triggering VersionConflict I always saw as a nuisance. It's not really protecting you against anything, you're already in the broken state.

It would be possible to do with importlib-metadata, but traversing the dependency graph is wasted cpu cycles :)

nicoddemus · 2019-05-04T14:32:00Z

Anything else to be done here @asottile? Also, can we check how this affects tox and devpy?

cc @gaborbernat @fschulze

asottile · 2019-05-04T15:03:46Z

I think testing the integration with the other big users is all that's left here. my pytest PR passes tests when coupled with this PR (unfortunately pytest has some tests which test the full end-to-end integration and worry about some specifics of the implementation of pluggy and so changes were necessary there)

asottile · 2019-05-04T15:13:19Z

tox's testsuite passes, there's not much that tox does directly with pkg_resources and would be easy to factor that out 👍

fschulze · 2019-05-04T16:18:14Z

Does it affect users of the pluggy library in any way? By the looks of it, it shouldn't. And the things we use from pkg_resources in devpi should be covered by importlib-metadata as well, so maybe we will also switch. I think only in tests are we using some things that aren't available yet.

asottile · 2019-05-04T16:25:18Z

this shouldn't affect users in any way (other than potentially speeding things up if pkg_resources gets factored out of the rest of the things) -- the only changes I had to make for pytest to make the tests pass was adjusting the tests (though I did also factor out most of the rest of the pytest usages of pkg_resources to get the full speed boost)

nicoddemus · 2019-05-05T11:38:01Z

Great!

I propose we merge this and make a 0.10 release then. We can then update pytest's PR and get this out with pytest 4.5. What do you think @asottile?

asottile · 2019-05-05T12:46:47Z

Sounds good! Let's do this Tuesday at the earliest (busy with pycon today and Monday and would want to be a available for potential fixes)

gaborbernat · 2019-05-05T13:24:10Z

tox will follow probably with a release on Thursday, to take advantage of the pytest release lessons 😎

nicoddemus · 2019-05-05T14:11:57Z

Let's do this Tuesday at the earliest (busy with pycon today and Monday and would want to be a available for potential fixes)

You mean pluggy, pytest or both? My original intention was to release pluggy soon only. pytest still has a few open PRs I would like to see merged before we release 4.5.

nicoddemus · 2019-05-05T14:12:49Z

(btw jealous of you guys at PyCon ❤️ 😁)

asottile · 2019-05-05T15:08:09Z

Let's do this Tuesday at the earliest (busy with pycon today and Monday and would want to be a available for potential fixes)

You mean pluggy, pytest or both? My original intention was to release pluggy soon only. pytest still has a few open PRs I would like to see merged before we release 4.5.

I was mostly referring to pluggy (and with the hope that pytest would follow soon after, I can help finishing up the other things we want to get into 4.5 as well :D)

nicoddemus · 2019-05-05T18:02:37Z

OK, let's wait until Tuesday then! 👍

nicoddemus · 2019-05-05T18:03:52Z

Oops forgot about the changelog entry.

fschulze · 2019-05-06T05:52:48Z

What about making it a 1.0.0? It's been in use for quite a while now.

goodboy · 2019-05-06T12:47:51Z

@fschulze we're trying to get in a bunch of "breaking" changes for 1.0 which I can hopefully get to at some point in the next month. Unfortunately been really busy with non-software stuff atm..

nicoddemus · 2019-05-06T13:20:52Z

We are tracking 1.0 here: #140

Add changelog for #199

1887: Scheduled weekly dependency update for week 19 r=rehandalal a=pyup-bot ### Update [botocore](https://pypi.org/project/botocore) from **1.12.142** to **1.12.146**. <details> <summary>Changelog</summary> ### 1.12.146 ``` ======== * api-change:``glue``: Update glue client to latest version * api-change:``sts``: Update sts client to latest version ``` ### 1.12.145 ``` ======== * api-change:``servicecatalog``: Update servicecatalog client to latest version * api-change:``eks``: Update eks client to latest version * api-change:``sagemaker``: Update sagemaker client to latest version * api-change:``kinesisanalytics``: Update kinesisanalytics client to latest version * api-change:``kinesisanalyticsv2``: Update kinesisanalyticsv2 client to latest version ``` ### 1.12.144 ``` ======== * api-change:``appsync``: Update appsync client to latest version * api-change:``storagegateway``: Update storagegateway client to latest version * api-change:``ssm``: Update ssm client to latest version * api-change:``alexaforbusiness``: Update alexaforbusiness client to latest version ``` ### 1.12.143 ``` ======== * api-change:``config``: Update config client to latest version * api-change:``iam``: Update iam client to latest version * api-change:``sts``: Update sts client to latest version * api-change:``codepipeline``: Update codepipeline client to latest version ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/botocore - Changelog: https://pyup.io/changelogs/botocore/ - Repo: https://github.com/boto/botocore </details> ### Update [Faker](https://pypi.org/project/Faker) from **1.0.5** to **1.0.6**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/faker - Changelog: https://pyup.io/changelogs/faker/ - Repo: https://github.com/joke2k/faker </details> ### Update [pluggy](https://pypi.org/project/pluggy) from **0.9.0** to **0.11.0**. <details> <summary>Changelog</summary> ### 0.11.0 ``` ========================== Bug Fixes --------- - `205 <https://github.com/pytest-dev/pluggy/issues/205>`_: Revert changes made in 0.10.0 release breaking ``.egg`` installs. ``` ### 0.10.0 ``` ========================== Features -------- - `199 <https://github.com/pytest-dev/pluggy/issues/199>`_: Switch from ``pkg_resources`` to ``importlib-metadata`` for entrypoint detection for improved performance and import time. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pluggy - Changelog: https://pyup.io/changelogs/pluggy/ - Repo: https://github.com/pytest-dev/pluggy </details> ### Update [Pygments](https://pypi.org/project/Pygments) from **2.3.1** to **2.4.0**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pygments - Homepage: http://pygments.org/ </details> ### Update [pyrsistent](https://pypi.org/project/pyrsistent) from **0.15.1** to **0.15.2**. <details> <summary>Changelog</summary> ### 0.15.2 ``` * Fix 166, Propagate 'ignore_extra' param in hierarchy. Thanks ss18 for this! * Fix 167, thaw typing. Thanks nattofriends for this! * Fix 154, not possible to insert empty pmap as leaf node with transform. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pyrsistent - Changelog: https://pyup.io/changelogs/pyrsistent/ - Repo: http://github.com/tobgu/pyrsistent/ </details> ### Update [boto3](https://pypi.org/project/boto3) from **1.9.142** to **1.9.146**. <details> <summary>Changelog</summary> ### 1.9.146 ``` ======= * api-change:``glue``: [``botocore``] Update glue client to latest version * api-change:``sts``: [``botocore``] Update sts client to latest version ``` ### 1.9.145 ``` ======= * api-change:``servicecatalog``: [``botocore``] Update servicecatalog client to latest version * api-change:``eks``: [``botocore``] Update eks client to latest version * api-change:``sagemaker``: [``botocore``] Update sagemaker client to latest version * api-change:``kinesisanalytics``: [``botocore``] Update kinesisanalytics client to latest version * api-change:``kinesisanalyticsv2``: [``botocore``] Update kinesisanalyticsv2 client to latest version ``` ### 1.9.144 ``` ======= * api-change:``appsync``: [``botocore``] Update appsync client to latest version * api-change:``storagegateway``: [``botocore``] Update storagegateway client to latest version * api-change:``ssm``: [``botocore``] Update ssm client to latest version * api-change:``alexaforbusiness``: [``botocore``] Update alexaforbusiness client to latest version ``` ### 1.9.143 ``` ======= * api-change:``config``: [``botocore``] Update config client to latest version * api-change:``iam``: [``botocore``] Update iam client to latest version * api-change:``sts``: [``botocore``] Update sts client to latest version * api-change:``codepipeline``: [``botocore``] Update codepipeline client to latest version ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/boto3 - Changelog: https://pyup.io/changelogs/boto3/ - Repo: https://github.com/boto/boto3 </details> ### Update [django-cors-headers](https://pypi.org/project/django-cors-headers) from **2.5.3** to **3.0.1**. <details> <summary>Changelog</summary> ### 3.0.0 ``` ------------------ * ``CORS_ORIGIN_WHITELIST`` now requires URI schemes, and optionally ports. This is part of the CORS specification (`Section 3.2 <https://tools.ietf.org/html/rfc6454section-3.2>`_) that was not implemented in this library, except from with the ``CORS_ORIGIN_REGEX_WHITELIST`` setting. It fixes a security issue where the CORS middleware would allow requests between schemes, for example from insecure ``http://`` Origins to a secure ``https://`` site. You will need to update your whitelist to include schemes, for example from this: .. code-block:: python CORS_ORIGIN_WHITELIST = ['example.com'] ...to this: .. code-block:: python CORS_ORIGIN_WHITELIST = ['https://example.com'] * Removed the ``CORS_MODEL`` setting, and associated class. It seems very few, or no users were using it, since there were no bug reports since its move to abstract in version 2.0.0 (2017-01-07). If you *are* using this functionality, you can continue by changing your model to not inherit from the abstract one, and add a signal handler for ``check_request_enabled`` that reads from your model. Note you'll need to handle the move to include schemes for Origins. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/django-cors-headers - Changelog: https://pyup.io/changelogs/django-cors-headers/ - Repo: https://github.com/ottoyiu/django-cors-headers </details> ### Update [djangorestframework](https://pypi.org/project/djangorestframework) from **3.9.3** to **3.9.4**. *The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)* <details> <summary>Links</summary> - PyPI: https://pypi.org/project/djangorestframework - Changelog: https://pyup.io/changelogs/djangorestframework/ - Homepage: https://www.django-rest-framework.org/ </details> ### Update [factory_boy](https://pypi.org/project/factory_boy) from **2.11.1** to **2.12.0**. <details> <summary>Changelog</summary> ### 2.12.0 ``` ------------------- *New:* - Add support for Python 3.7 - Add support for Django 2.1 - Add :attr:`~factory.fuzzy.FuzzyChoice.getter` to :class:`~factory.fuzzy.FuzzyChoice` that mimics the behavior of ``getter`` in :class:`~factory.Iterator` - Make the ``extra_kwargs`` parameter of :meth:`~factory.faker.Faker.generate` optional - Add :class:`~factory.RelatedFactoryList` class for one-to-many support, thanks `Sean Harrington <https://github.com/seanharr11>`_. - Make the `locale` argument for :class:`~factory.faker.Faker` keyword-only *Bugfix:* - Allow renamed arguments to be optional, thanks to `Justin Crown <https://github.com/mrname>`_. - Fix `django_get_or_create` behavior when using multiple fields with `unique=True`, thanks to `YPCrumble <https://github.com/YPCrumble>` ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/factory-boy - Changelog: https://pyup.io/changelogs/factory-boy/ - Repo: https://github.com/FactoryBoy/factory_boy </details> ### Update [kinto-http](https://pypi.org/project/kinto-http) from **10.3.0** to **10.4.0**. <details> <summary>Changelog</summary> ### 10.4.0 ``` =================== - Add support for Bearer tokens in the CLI utilities. - Use black for code formatting. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/kinto-http - Changelog: https://pyup.io/changelogs/kinto-http/ - Repo: https://github.com/Kinto/kinto-http.py/ </details> ### Update [pytest](https://pypi.org/project/pytest) from **4.4.1** to **4.5.0**. <details> <summary>Changelog</summary> ### 4.4.2 ``` ========================= Bug Fixes --------- - `5089 <https://github.com/pytest-dev/pytest/issues/5089>`_: Fix crash caused by error in ``__repr__`` function with both ``showlocals`` and verbose output enabled. - `5139 <https://github.com/pytest-dev/pytest/issues/5139>`_: Eliminate core dependency on 'terminal' plugin. - `5229 <https://github.com/pytest-dev/pytest/issues/5229>`_: Require ``pluggy>=0.11.0`` which reverts a dependency to ``importlib-metadata`` added in ``0.10.0``. The ``importlib-metadata`` package cannot be imported when installed as an egg and causes issues when relying on ``setup.py`` to install test dependencies. Improved Documentation ---------------------- - `5171 <https://github.com/pytest-dev/pytest/issues/5171>`_: Doc: ``pytest_ignore_collect``, ``pytest_collect_directory``, ``pytest_collect_file`` and ``pytest_pycollect_makemodule`` hooks's 'path' parameter documented type is now ``py.path.local`` - `5188 <https://github.com/pytest-dev/pytest/issues/5188>`_: Improve help for ``--runxfail`` flag. Trivial/Internal Changes ------------------------ - `5182 <https://github.com/pytest-dev/pytest/issues/5182>`_: Removed internal and unused ``_pytest.deprecated.MARK_INFO_ATTRIBUTE``. ``` </details> <details> <summary>Links</summary> - PyPI: https://pypi.org/project/pytest - Changelog: https://pyup.io/changelogs/pytest/ - Homepage: https://docs.pytest.org/en/latest/ </details> Co-authored-by: pyup-bot <github-bot@pyup.io> Co-authored-by: Michael Cooper <mythmon@gmail.com>

asottile · 2019-05-25T22:41:27Z

I've re-applied this in #215 🎉

gaborbernat · 2019-05-28T09:48:43Z

cool times

Switch to importlib-metadata

9550fc3

asottile mentioned this pull request Apr 7, 2019

Switch to importlib-metadata pytest-dev/pytest#5063

Merged

asottile added 2 commits April 7, 2019 13:47

Add a Facade to pretend to be pkg_resources

491cefe

Fix test for DistFacade

e387587

asottile changed the title ~~[WIP] Switch to importlib-metadata~~ Switch to importlib-metadata Apr 20, 2019

asottile added 2 commits April 20, 2019 17:37

Expose the ImportError / AttributeError to the caller

f282a42

Invoke tox directly (?)

0706fba

jayvdb reviewed May 2, 2019

View reviewed changes

nicoddemus approved these changes May 4, 2019

View reviewed changes

nicoddemus merged commit 8749791 into pytest-dev:master May 5, 2019

asottile deleted the importlib_metadata_v2 branch May 6, 2019 14:08

nicoddemus added a commit that referenced this pull request May 6, 2019

Add changelog for #199 (#201)

f4eb6b4

Add changelog for #199

nicoddemus mentioned this pull request May 7, 2019

importlib_metadata breaks TravisCI builds #205

Closed

asottile mentioned this pull request May 7, 2019

Revert "Switch to importlib-metadata" #206

Merged

asottile mentioned this pull request May 25, 2019

importlib-metadata again #215

Merged

anthrotype mentioned this pull request Oct 16, 2019

drop or relax dependency on pkg_resources? PyFilesystem/pyfilesystem2#356

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to importlib-metadata #199

Switch to importlib-metadata #199

asottile commented Apr 7, 2019 •

edited

asottile commented Apr 7, 2019 •

edited

asottile commented Apr 20, 2019

asottile commented Apr 21, 2019

codecov-io commented Apr 21, 2019 •

edited

jayvdb May 2, 2019

asottile May 2, 2019

nicoddemus commented May 4, 2019

asottile commented May 4, 2019

asottile commented May 4, 2019

fschulze commented May 4, 2019

asottile commented May 4, 2019 •

edited

nicoddemus commented May 5, 2019

asottile commented May 5, 2019

gaborbernat commented May 5, 2019

nicoddemus commented May 5, 2019

nicoddemus commented May 5, 2019

asottile commented May 5, 2019

nicoddemus commented May 5, 2019

nicoddemus commented May 5, 2019

fschulze commented May 6, 2019

goodboy commented May 6, 2019

nicoddemus commented May 6, 2019

asottile commented May 25, 2019

gaborbernat commented May 28, 2019

Switch to importlib-metadata #199

Switch to importlib-metadata #199

Conversation

asottile commented Apr 7, 2019 • edited

asottile commented Apr 7, 2019 • edited

asottile commented Apr 20, 2019

asottile commented Apr 21, 2019

codecov-io commented Apr 21, 2019 • edited

Codecov Report

jayvdb May 2, 2019

Choose a reason for hiding this comment

asottile May 2, 2019

Choose a reason for hiding this comment

nicoddemus commented May 4, 2019

asottile commented May 4, 2019

asottile commented May 4, 2019

fschulze commented May 4, 2019

asottile commented May 4, 2019 • edited

nicoddemus commented May 5, 2019

asottile commented May 5, 2019

gaborbernat commented May 5, 2019

nicoddemus commented May 5, 2019

nicoddemus commented May 5, 2019

asottile commented May 5, 2019

nicoddemus commented May 5, 2019

nicoddemus commented May 5, 2019

fschulze commented May 6, 2019

goodboy commented May 6, 2019

nicoddemus commented May 6, 2019

asottile commented May 25, 2019

gaborbernat commented May 28, 2019

asottile commented Apr 7, 2019 •

edited

asottile commented Apr 7, 2019 •

edited

codecov-io commented Apr 21, 2019 •

edited

asottile commented May 4, 2019 •

edited