Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cache Requirement Object Creation #10550

Merged
merged 6 commits into from Oct 9, 2021
Merged

Cache Requirement Object Creation #10550

merged 6 commits into from Oct 9, 2021

Conversation

jbylund
Copy link
Contributor

@jbylund jbylund commented Oct 6, 2021 •
edited

Requirement object creation is a large fraction of resolution time in some circumstances (~65% inclusive in the profile I'm looking at), and at least in this example there are very few distinct arguments to the Requirement constructor. Which I think makes this a decent candidate for caching.

TODO:

  • requirements don't have an equality operator... so compare attributes? or?

image

@jbylund
(joe) requirement creation is very expensive, and at least in my test…
bb88244 
… case there were only ~200 unique requirement objects created in ~5-10 minutes of resolution time
@jbylund jbylund force-pushed the joe/cache_requirement_creation branch from 44a7076 to bb88244 Compare October 6, 2021 16:42
uranusjr
uranusjr approved these changes Oct 8, 2021
View reviewed changes
Copy link
Member

@uranusjr uranusjr left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if this should be a part of 21.3; leaving the decision to RM. @pradyunsg

pradyunsg
pradyunsg requested changes Oct 8, 2021
View reviewed changes
src/pip/_internal/utils/packaging.py Outdated Show resolved Hide resolved
tests/unit/test_packaging.py Outdated Show resolved Hide resolved
news/10550.feature.rst Outdated Show resolved Hide resolved
tests/unit/test_packaging.py Show resolved Hide resolved
@pradyunsg pradyunsg added this to the 21.3 milestone Oct 8, 2021
@jbylund
Copy link
Contributor Author

jbylund commented Oct 8, 2021

Is that just a sometimes flaky test? the diff between the second to last and last commits is just adding a max-size parameter on the cache.

@pradyunsg
Copy link
Member

Yea. That test checks that a timeout of 0.001 seconds causes timeout errors. Apparently the network is too quick in some cases. :)

@jbylund
Copy link
Contributor Author

jbylund commented Oct 8, 2021

Yea. That test checks that a timeout of 0.001 seconds causes timeout errors. Apparently the network is too quick in some cases. :)

Do I need to push an empty commit or anything? or is it just ok if there are some failed tests on the pr?

@DiddiLeija
Copy link
Member

Maybe re-triggering the builds could help. I will close and reopen the PR to do it.

@DiddiLeija DiddiLeija closed this Oct 8, 2021
@DiddiLeija DiddiLeija reopened this Oct 8, 2021
@pradyunsg pradyunsg removed this from the 21.3 milestone Oct 9, 2021
@pradyunsg pradyunsg merged commit 4fac2b9 into pypa:main Oct 9, 2021
@pradyunsg
Copy link
Member

Thanks for the PR @jbylund!

inmantaci pushed a commit to inmanta/inmanta-core that referenced this pull request Oct 12, 2021
@dependabot @inmantaci
Bump pip from 21.2.4 to 21.3 (PR #3348)
a3a2158 
Bumps [pip](https://github.com/pypa/pip) from 21.2.4 to 21.3.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>21.3 (2021-10-11)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li>Improve deprecation warning regarding the copying of source trees when installing from a local directory. (<code>[#10128](pypa/pip#10128) &lt;https://github.com/pypa/pip/issues/10128&gt;</code>_)</li>
<li>Suppress location mismatch warnings when pip is invoked from a Python source
tree, so <code>ensurepip</code> does not emit warnings on CPython <code>make install</code>. (<code>[#10270](pypa/pip#10270) &lt;https://github.com/pypa/pip/issues/10270&gt;</code>_)</li>
<li>On Python 3.10 or later, the installation scheme backend has been changed to use
<code>sysconfig</code>. This is to anticipate the deprecation of <code>distutils</code> in Python
3.10, and its scheduled removal in 3.12. For compatibility considerations, pip
installations running on Python 3.9 or lower will continue to use <code>distutils</code>. (<code>[#10358](pypa/pip#10358) &lt;https://github.com/pypa/pip/issues/10358&gt;</code>_)</li>
<li>Remove the <code>--build-dir</code> option and aliases, one last time. (<code>[#10485](pypa/pip#10485) &lt;https://github.com/pypa/pip/issues/10485&gt;</code>_)</li>
<li>In-tree builds are now the default. <code>--use-feature=in-tree-build</code> is now
ignored. <code>--use-deprecated=out-of-tree-build</code> may be used temporarily to ease
the transition. (<code>[#10495](pypa/pip#10495) &lt;https://github.com/pypa/pip/issues/10495&gt;</code>_)</li>
<li>Un-deprecate source distribution re-installation behaviour. (<code>[#8711](pypa/pip#8711) &lt;https://github.com/pypa/pip/issues/8711&gt;</code>_)</li>
</ul>
<h2>Features</h2>
<ul>
<li>Replace vendored appdirs with platformdirs. (<code>[#10202](pypa/pip#10202) &lt;https://github.com/pypa/pip/issues/10202&gt;</code>_)</li>
<li>Support <code>PEP 610 &lt;https://www.python.org/dev/peps/pep-0610/&gt;</code>_ to detect
editable installs in <code>pip freeze</code> and  <code>pip list</code>. The <code>pip list</code> column output
has a new <code>Editable project location</code> column, and the JSON output has a new
<code>editable_project_location</code> field. (<code>[#10249](pypa/pip#10249) &lt;https://github.com/pypa/pip/issues/10249&gt;</code>_)</li>
<li><code>pip freeze</code> will now always fallback to reporting the editable project
location when it encounters a VCS error while analyzing an editable
requirement. Before, it sometimes reported the requirement as non-editable. (<code>[#10410](pypa/pip#10410) &lt;https://github.com/pypa/pip/issues/10410&gt;</code>_)</li>
<li><code>pip show</code> now sorts <code>Requires</code> and <code>Required-By</code> alphabetically. (<code>[#10422](pypa/pip#10422) &lt;https://github.com/pypa/pip/issues/10422&gt;</code>_)</li>
<li>Do not raise error when there are no files to remove with <code>pip cache purge/remove</code>.
Instead log a warning and continue (to log that we removed 0 files). (<code>[#10459](pypa/pip#10459) &lt;https://github.com/pypa/pip/issues/10459&gt;</code>_)</li>
<li>When backtracking during dependency resolution, prefer the dependencies which are involved in the most recent conflict. This can significantly reduce the amount of backtracking required. (<code>[#10479](pypa/pip#10479) &lt;https://github.com/pypa/pip/issues/10479&gt;</code>_)</li>
<li>Cache requirement objects, to improve performance reducing reparses of requirement strings. (<code>[#10550](pypa/pip#10550) &lt;https://github.com/pypa/pip/issues/10550&gt;</code>_)</li>
<li>Support editable installs for projects that have a <code>pyproject.toml</code> and use a
build backend that supports :pep:<code>660</code>. (<code>[#8212](pypa/pip#8212) &lt;https://github.com/pypa/pip/issues/8212&gt;</code>_)</li>
<li>When a revision is specified in a Git URL, use git's partial clone feature to speed up source retrieval. (<code>[#9086](pypa/pip#9086) &lt;https://github.com/pypa/pip/issues/9086&gt;</code>_)</li>
<li>Add a <code>--debug</code> flag, to enable a mode that doesn't log errors and propagates them to the top level instead. This is primarily to aid with debugging pip's crashes. (<code>[#9349](pypa/pip#9349) &lt;https://github.com/pypa/pip/issues/9349&gt;</code>_)</li>
<li>If a host is explicitly specified as trusted by the user (via the --trusted-host option), cache HTTP responses from it in addition to HTTPS ones. (<code>[#9498](pypa/pip#9498) &lt;https://github.com/pypa/pip/issues/9498&gt;</code>_)</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>Present a better error message, when a <code>file:</code> URL is not found. (<code>[#10263](pypa/pip#10263) &lt;https://github.com/pypa/pip/issues/10263&gt;</code>_)</li>
<li>Fix the auth credential cache to allow for the case in which
the index url contains the username, but the password comes
from an external source, such as keyring. (<code>[#10269](pypa/pip#10269) &lt;https://github.com/pypa/pip/issues/10269&gt;</code>_)</li>
<li>Fix double unescape of HTML <code>data-requires-python</code> and <code>data-yanked</code> attributes. (<code>[#10378](pypa/pip#10378) &lt;https://github.com/pypa/pip/issues/10378&gt;</code>_)</li>
<li>New resolver: Fixes depth ordering of packages during resolution, e.g. a dependency 2 levels deep will be ordered before a dependecy 3 levels deep. (<code>[#10482](pypa/pip#10482) &lt;https://github.com/pypa/pip/issues/10482&gt;</code>_)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/abec8a701bfa66aa15fedf4c898011aa2d95f29e"><code>abec8a7</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/68a70486c9224f9d25be3cbf56c73d8a33c6a713"><code>68a7048</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/9f18a403ca41f4e42fbb89d286b6571a099cb54b"><code>9f18a40</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10481">#10481</a> from notatallshaw/prefer_failures</li>
<li><a href="https://github.com/pypa/pip/commit/db496cbce518fa159476695db0cd4f1c1a8ab6f5"><code>db496cb</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10563">#10563</a> from pradyunsg/shorter-timeout</li>
<li><a href="https://github.com/pypa/pip/commit/4fac2b90a5d200b46e7b576013bb25f4ebb3f937"><code>4fac2b9</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10550">#10550</a> from jbylund/joe/cache_requirement_creation</li>
<li><a href="https://github.com/pypa/pip/commit/786957cf85a641d49b4cfcceef717ef229ac8238"><code>786957c</code></a> Use a shorter timeout, to ensure that this fails more often</li>
<li><a href="https://github.com/pypa/pip/commit/1e3c127d4a938643aca1bbc25e6581493e316476"><code>1e3c127</code></a> Avoid passing <code>.</code> to vendoring</li>
<li><a href="https://github.com/pypa/pip/commit/610424f9f8ad1f99d0a48bf9a53e7a9df4242304"><code>610424f</code></a> Quote &quot;PreferenceInformation&quot; to avoid runtime NameError</li>
<li><a href="https://github.com/pypa/pip/commit/c01b5c6d8a4858cf733408b4b020933f902dda9e"><code>c01b5c6</code></a> Update a test for resolvelib 0.8.0</li>
<li><a href="https://github.com/pypa/pip/commit/394a24eb1a5f9af5da7d4d2452ed5fe952de5db2"><code>394a24e</code></a> Upgrade resolvelib to 0.8.0</li>
<li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/21.2.4...21.3">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=21.2.4&new-version=21.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>
mergify bot pushed a commit to andrewbolster/bolster that referenced this pull request Oct 12, 2021
@dependabot
Bump pip from 21.2.4 to 21.3 in /.github/workflows (#63)
9d1dd85 
Bumps [pip](https://github.com/pypa/pip) from 21.2.4 to 21.3.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>21.3 (2021-10-11)</h1>
<h2>Deprecations and Removals</h2>
<ul>
<li>Improve deprecation warning regarding the copying of source trees when installing from a local directory. (<code>[#10128](pypa/pip#10128) &lt;https://github.com/pypa/pip/issues/10128&gt;</code>_)</li>
<li>Suppress location mismatch warnings when pip is invoked from a Python source
tree, so <code>ensurepip</code> does not emit warnings on CPython <code>make install</code>. (<code>[#10270](pypa/pip#10270) &lt;https://github.com/pypa/pip/issues/10270&gt;</code>_)</li>
<li>On Python 3.10 or later, the installation scheme backend has been changed to use
<code>sysconfig</code>. This is to anticipate the deprecation of <code>distutils</code> in Python
3.10, and its scheduled removal in 3.12. For compatibility considerations, pip
installations running on Python 3.9 or lower will continue to use <code>distutils</code>. (<code>[#10358](pypa/pip#10358) &lt;https://github.com/pypa/pip/issues/10358&gt;</code>_)</li>
<li>Remove the <code>--build-dir</code> option and aliases, one last time. (<code>[#10485](pypa/pip#10485) &lt;https://github.com/pypa/pip/issues/10485&gt;</code>_)</li>
<li>In-tree builds are now the default. <code>--use-feature=in-tree-build</code> is now
ignored. <code>--use-deprecated=out-of-tree-build</code> may be used temporarily to ease
the transition. (<code>[#10495](pypa/pip#10495) &lt;https://github.com/pypa/pip/issues/10495&gt;</code>_)</li>
<li>Un-deprecate source distribution re-installation behaviour. (<code>[#8711](pypa/pip#8711) &lt;https://github.com/pypa/pip/issues/8711&gt;</code>_)</li>
</ul>
<h2>Features</h2>
<ul>
<li>Replace vendored appdirs with platformdirs. (<code>[#10202](pypa/pip#10202) &lt;https://github.com/pypa/pip/issues/10202&gt;</code>_)</li>
<li>Support <code>PEP 610 &lt;https://www.python.org/dev/peps/pep-0610/&gt;</code>_ to detect
editable installs in <code>pip freeze</code> and  <code>pip list</code>. The <code>pip list</code> column output
has a new <code>Editable project location</code> column, and the JSON output has a new
<code>editable_project_location</code> field. (<code>[#10249](pypa/pip#10249) &lt;https://github.com/pypa/pip/issues/10249&gt;</code>_)</li>
<li><code>pip freeze</code> will now always fallback to reporting the editable project
location when it encounters a VCS error while analyzing an editable
requirement. Before, it sometimes reported the requirement as non-editable. (<code>[#10410](pypa/pip#10410) &lt;https://github.com/pypa/pip/issues/10410&gt;</code>_)</li>
<li><code>pip show</code> now sorts <code>Requires</code> and <code>Required-By</code> alphabetically. (<code>[#10422](pypa/pip#10422) &lt;https://github.com/pypa/pip/issues/10422&gt;</code>_)</li>
<li>Do not raise error when there are no files to remove with <code>pip cache purge/remove</code>.
Instead log a warning and continue (to log that we removed 0 files). (<code>[#10459](pypa/pip#10459) &lt;https://github.com/pypa/pip/issues/10459&gt;</code>_)</li>
<li>When backtracking during dependency resolution, prefer the dependencies which are involved in the most recent conflict. This can significantly reduce the amount of backtracking required. (<code>[#10479](pypa/pip#10479) &lt;https://github.com/pypa/pip/issues/10479&gt;</code>_)</li>
<li>Cache requirement objects, to improve performance reducing reparses of requirement strings. (<code>[#10550](pypa/pip#10550) &lt;https://github.com/pypa/pip/issues/10550&gt;</code>_)</li>
<li>Support editable installs for projects that have a <code>pyproject.toml</code> and use a
build backend that supports :pep:<code>660</code>. (<code>[#8212](pypa/pip#8212) &lt;https://github.com/pypa/pip/issues/8212&gt;</code>_)</li>
<li>When a revision is specified in a Git URL, use git's partial clone feature to speed up source retrieval. (<code>[#9086](pypa/pip#9086) &lt;https://github.com/pypa/pip/issues/9086&gt;</code>_)</li>
<li>Add a <code>--debug</code> flag, to enable a mode that doesn't log errors and propagates them to the top level instead. This is primarily to aid with debugging pip's crashes. (<code>[#9349](pypa/pip#9349) &lt;https://github.com/pypa/pip/issues/9349&gt;</code>_)</li>
<li>If a host is explicitly specified as trusted by the user (via the --trusted-host option), cache HTTP responses from it in addition to HTTPS ones. (<code>[#9498](pypa/pip#9498) &lt;https://github.com/pypa/pip/issues/9498&gt;</code>_)</li>
</ul>
<h2>Bug Fixes</h2>
<ul>
<li>Present a better error message, when a <code>file:</code> URL is not found. (<code>[#10263](pypa/pip#10263) &lt;https://github.com/pypa/pip/issues/10263&gt;</code>_)</li>
<li>Fix the auth credential cache to allow for the case in which
the index url contains the username, but the password comes
from an external source, such as keyring. (<code>[#10269](pypa/pip#10269) &lt;https://github.com/pypa/pip/issues/10269&gt;</code>_)</li>
<li>Fix double unescape of HTML <code>data-requires-python</code> and <code>data-yanked</code> attributes. (<code>[#10378](pypa/pip#10378) &lt;https://github.com/pypa/pip/issues/10378&gt;</code>_)</li>
<li>New resolver: Fixes depth ordering of packages during resolution, e.g. a dependency 2 levels deep will be ordered before a dependecy 3 levels deep. (<code>[#10482](pypa/pip#10482) &lt;https://github.com/pypa/pip/issues/10482&gt;</code>_)</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/abec8a701bfa66aa15fedf4c898011aa2d95f29e"><code>abec8a7</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/68a70486c9224f9d25be3cbf56c73d8a33c6a713"><code>68a7048</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/9f18a403ca41f4e42fbb89d286b6571a099cb54b"><code>9f18a40</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10481">#10481</a> from notatallshaw/prefer_failures</li>
<li><a href="https://github.com/pypa/pip/commit/db496cbce518fa159476695db0cd4f1c1a8ab6f5"><code>db496cb</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10563">#10563</a> from pradyunsg/shorter-timeout</li>
<li><a href="https://github.com/pypa/pip/commit/4fac2b90a5d200b46e7b576013bb25f4ebb3f937"><code>4fac2b9</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/pypa/pip/issues/10550">#10550</a> from jbylund/joe/cache_requirement_creation</li>
<li><a href="https://github.com/pypa/pip/commit/786957cf85a641d49b4cfcceef717ef229ac8238"><code>786957c</code></a> Use a shorter timeout, to ensure that this fails more often</li>
<li><a href="https://github.com/pypa/pip/commit/1e3c127d4a938643aca1bbc25e6581493e316476"><code>1e3c127</code></a> Avoid passing <code>.</code> to vendoring</li>
<li><a href="https://github.com/pypa/pip/commit/610424f9f8ad1f99d0a48bf9a53e7a9df4242304"><code>610424f</code></a> Quote &quot;PreferenceInformation&quot; to avoid runtime NameError</li>
<li><a href="https://github.com/pypa/pip/commit/c01b5c6d8a4858cf733408b4b020933f902dda9e"><code>c01b5c6</code></a> Update a test for resolvelib 0.8.0</li>
<li><a href="https://github.com/pypa/pip/commit/394a24eb1a5f9af5da7d4d2452ed5fe952de5db2"><code>394a24e</code></a> Upgrade resolvelib to 0.8.0</li>
<li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/21.2.4...21.3">compare view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=21.2.4&new-version=21.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)


</details>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 25, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@uranusjr uranusjr uranusjr approved these changes

@pradyunsg pradyunsg pradyunsg approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@jbylund @pradyunsg @DiddiLeija @uranusjr