Skip to content

Conversation

@anntzer
Copy link
Contributor

@anntzer anntzer commented Jan 6, 2021

Locally, this speeds up distribution("ipython") on a large-ish
environment by nearly 2-fold. (The GHA benchmark goes from ~1.6ms
to ~0.8ms.)

All the following changes matter for the performance gain:

  • avoid needlessly reinstanciating Prepareds.
  • switch from os.path.splitext (very slow) to str.rpartition
  • avoid having to add back a dot to the ext part given by rpartition
  • precompute egg_prefix and versionless_egg_name

(See https://gitlab.com/python-devs/importlib_metadata/-/issues/95 for
rationale why performance matters.)

@jaraco
Copy link
Member

jaraco commented Jan 10, 2021

Thanks for this. It's excellent. The only real wish I have (for future consideration) is if these optimizations had been committed and pushed separately, we could see the impact of each change independently.

Any idea why the Python 3.6 tests are timing out?

@jaraco
Copy link
Member

jaraco commented Jan 10, 2021

Any idea why the Python 3.6 tests are timing out?

Oh, dear. I see the Python 3.6 tests have been timing out for some time, although apparently only intermittently. I'll bet it's some bug with pips resolver and selective dependencies on self (importlib_metadata). I'll deal with that in another issue/pr.

@anntzer
Copy link
Contributor Author

anntzer commented Jan 10, 2021

if these optimizations had been committed and pushed separately, we could see the impact of each change independently.

Done.

@jaraco jaraco closed this Jan 10, 2021
@jaraco jaraco reopened this Jan 10, 2021
@jaraco
Copy link
Member

jaraco commented Jan 10, 2021

Looks like the second and third commits had the most impact:

importlib_metadata HEAD $ git checkout -q c1af1d6~1; tox -q -e perf
200 loops, best of 5: 1.47 msec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

importlib_metadata HEAD $ git checkout -q c1af1d6; tox -q -e perf
200 loops, best of 5: 1.45 msec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

importlib_metadata HEAD $ git checkout -q c9c0909; tox -q -e perf
200 loops, best of 5: 1.14 msec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

importlib_metadata HEAD $ git checkout -q 7fc4f11; tox -q -e perf
200 loops, best of 5: 928 usec per loop
________________________________________________________________ summary ________________________________________________________________
  perf: commands succeeded
  congratulations :)

@jaraco jaraco closed this Jan 10, 2021
@jaraco jaraco reopened this Jan 10, 2021
@jaraco jaraco merged commit c93a9f7 into python:main Jan 10, 2021
@anntzer anntzer deleted the moreperf branch January 10, 2021 19:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants