Investigate performance concerns raised between 1.4 and 4.3 #324
Labels
invalid
This doesn't seem right
Comments
|
In jaraco/pytest-perf 0.2, I've made some progress toward having a pytest plugin that can take a suite of tests and compare them against the main branch. I'm hoping to use that to evaluate options to address the reported concerns. |
|
In e796bcc, I've added a test that models the reported regression. The test added here runs the aforementioned commands through However, on both my machine and in CI runs such as this one, the output from the test looks something like this: That result indicates that the local code took 4.2ms and that was 69% faster than the code in importlib_metadata 1.4, contradicting the original report. To be sure, |
jaraco
added a commit
that referenced
this issue
Jun 27, 2021
|
Re-reading the report, I see that there were additional dependencies indicated that weren't present in my attempt to repro. Adding these dependencies did add a couple of ms to the load time, but did not affect the percentage difference: |
|
If you want to try it for yourself, just clone the |
|
Just to check my assumptions on how the test is measuring and reporting, I added the following diff to inject 10ms of slowness into each trial: diff --git a/exercises.py b/exercises.py
index 0c997be3582..f1a83a90aa4 100644
--- a/exercises.py
+++ b/exercises.py
@@ -46,6 +46,8 @@ def entry_points_selected_perf():
try:
eps.select(group="flake8.extension")
eps.select(group="flake8.report")
+ import time
+ time.sleep(.01)
except AttributeError:
eps["flake8.extension"]
eps["flake8.report"]And then ran the tests again. This time, pytest-perf reports that the experiment took 21ms and was 50% slower than the control: That's fairly consistent, at least within an order of magnitude. |
|
In 4de84f8, I add some assertions. I was going to assert that the lengths of the entry points were the same, but they're not the same, possibly due to deduplication of distributions (although that shouldn't be the case either, so maybe that's a clue). |
|
Running the tests on Python 3.10, the performance is even better, so that doesn't explain the reported regression: |
|
Re-running the exact commands from the OP using Python 3.9.5 and Python 3.10.0b3, I get the opposite results to those reported (Python 3.10 is > 4x faster): At this point, I don't believe the report is valid, or maybe it was fixed by incorporating the performance fixes that I'd advised were added in Python 3.10b3 and importlib_metadata 4.3. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
In bpo-44246, @asottile writes:
here's the performance regressions, they affect any callers of
distributions()and are even worse on callers of the new apis.a call to distributions() is about 3x slower than in 3.9
here is the setup I am using:
virtualenv venv39 -ppython3.9
venv39/bin/pip install flake8 pytest twine pre-commit
virtualenv venv310 -ppython3.10
venv310/bin/pip install flake8 pytest twine pre-commit
to test just the
distributions()call I'm using the following:this is a less-extreme example, many applications have more dependencies installed -- but even in this case this is adding ~24ms startup to any application using
entry_points()-- and it gets worsethe return value of
entry_points()alone isn't all that useful, next an application needs to retrieve its entry points. let's start for the somewhat normal case of retrieving a single category of entry points:again, 3x slower and very real time to the end user (~24-25ms)
now let's show an example usage that something like flake8 uses where multiple groups are requested (this is common for apps and plugin systems which provide multiple distinct functionalities)
also slower, but an additional ms per call to
.select(...)and it only gets worse with more and more packages installed
here's the versions I'm using to ensure they are up to date:
Python 3.10.0b2 maps to importlib_metadata 4.3 and Python 3.9.5 maps to importlib_metadata 1.4. Let's investigate which factors affect the performance and what use cases.
The text was updated successfully, but these errors were encountered: