Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move to importlib-metadata for performance improvements #1324

Merged
merged 1 commit into from May 28, 2019
Merged

Move to importlib-metadata for performance improvements #1324

merged 1 commit into from May 28, 2019

Conversation

@asottile
Copy link
Member

@asottile asottile commented May 28, 2019

setup

(with ~some amount of pip requirements installed -- I used https://github.com/Yelp/paasta as it installs a bunch of things and I know they all build on my machine)

$ pip freeze -l | wc -l
165

I used "best of 5" when picking the outputs below -- the version 0.0.0 is because I didn't have setuptools-scm when I built the dist -- just ignore that ;)

before

$ time tox --version
3.12.1 imported from /tmp/paasta/.tox/py/lib/python3.6/site-packages/tox/__init__.py

real	0m0.298s
user	0m0.270s
sys	0m0.029s

after

$ time tox --version
0.0.0 imported from /tmp/paasta/.tox/py/lib/python3.6/site-packages/tox/__init__.py

real	0m0.208s
user	0m0.200s
sys	0m0.008s

savings

~90ms -- so somewhere between 30-40%

Contribution checklist:

(also see CONTRIBUTING.rst for details)

  • wrote descriptive pull request text
  • added/updated test(s)
  • updated/extended the documentation
  • added relevant issue keyword
    in message body
  • added news fragment in changelog folder
    • fragment name: <issue number>.<type>.rst for example (588.bugfix.rst)
    • <type> is must be one of bugfix, feature, deprecation,breaking, doc, misc
    • if PR has no issue: consider creating one first or change it to the PR number after creating the PR
    • "sign" fragment with "by :user:<your username>"
    • please use full sentences with correct case and punctuation, for example: "Fix issue with non-ascii contents in doctest text files - by :user:superuser."
    • also see examples
  • added yourself to CONTRIBUTORS (preserving alphabetical order)
@@ -38,8 +38,9 @@ classifiers =
packages = find:
python_requires = >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
install_requires =
setuptools >= 30.0.0
pluggy >= 0.3.0, <1
importlib-metadata
Copy link
Member

@gaborbernat gaborbernat May 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we version guard from above?

Copy link
Member Author

@asottile asottile May 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yessss I wasn't sure what to put here, I'll go with >=0.12,<1

pkg = Requirement(require)
pkg_name = canonicalize_name(pkg.name)
if (
pkg.marker is None or pkg.marker.evaluate({"extra": ""})
Copy link
Member

@gaborbernat gaborbernat May 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you explain this check here 🤔 maybe put a comment because reading it seems kinda cryptic; why we need the extra?

Copy link
Member Author

@asottile asottile May 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah this is a little bit weird

it's essentially a reimplement of pkg_resources.Requirement.requires but using the importlib-metadata / packaging primitives

the pkg_resources api has:

    def requires(self, extras=()):

so by default it does not pass extras at all.

the importlib_metadata requires ends up looking like this for tox:

>>> for req in importlib_metadata.distribution('tox').requires:
...     print(f'- {req}')
... 
- setuptools>=30.0.0
- pluggy<1,>=0.3.0
- py<2,>=1.4.17
- six<2,>=1.0.0
- virtualenv>=14.0.0
- toml>=0.9.4
- filelock<4,>=3.0.0
- sphinx<3,>=2.0.0; extra == "docs"
- towncrier>=18.5.0; extra == "docs"
- pygments-github-lexers>=0.0.5; extra == "docs"
- sphinxcontrib-autoprogram>=0.1.5; extra == "docs"
- freezegun<1,>=0.3.11; extra == "testing"
- pathlib2<3,>=2.3.3; extra == "testing"
- pytest<5,>=3.0.0; extra == "testing"
- pytest-cov<3,>=2.5.1; extra == "testing"
- pytest-mock<2,>=1.10.0; extra == "testing"
- pytest-xdist<2,>=1.22.2; extra == "testing"
- pytest-randomly<2,>=1.2.3; extra == "testing"
- flaky<4,>=3.4.0; extra == "testing"
- psutil<6,>=5.6.1; (python_version != "3.4") and extra == "testing"

Converting one of those later ones into packaging primitives:

>>> from packaging.requirements import Requirement
>>> req = Requirement('pytest-randomly<2,>=1.2.3; extra == "testing"')
>>> req.marker
<Marker('extra == "testing"')>
>>> req.marker.evaluate()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/tox/venv/lib/python3.6/site-packages/packaging/markers.py", line 296, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/tmp/tox/venv/lib/python3.6/site-packages/packaging/markers.py", line 221, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/tmp/tox/venv/lib/python3.6/site-packages/packaging/markers.py", line 203, in _get_env
    "{0!r} does not exist in evaluation environment.".format(name)
packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.
>>> req.marker.evaluate({'extra': ''})
False
>>> req.marker.evaluate({'extra': 'testing'})
True

@gaborbernat gaborbernat merged commit afe11da into tox-dev:master May 28, 2019
1 check passed
@asottile asottile deleted the importlib_metadata branch May 28, 2019
@asottile
Copy link
Member Author

@asottile asottile commented May 28, 2019

oh, heh, I was going to add the bounds for importlib-metadata -- I'll do that in another PR 😆

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants