Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pip downloads many versions of extra'ed dependency while the exact version is pinned via a constraint #10233

Closed
1 task done
sbidoul opened this issue Jul 29, 2021 · 2 comments · Fixed by #10239
Closed
1 task done

pip downloads many versions of extra'ed dependency while the exact version is pinned via a constraint #10233

sbidoul opened this issue Jul 29, 2021 · 2 comments · Fixed by #10239
Labels
C: dependency resolution About choosing which dependencies to install
Milestone
21.2.2

Comments

@sbidoul
Copy link
Member

sbidoul commented Jul 29, 2021

Description

Pip looks at many version of an extra'ed dependency, even if the version is pinned with a constraint.

Expected behavior

Since there is a constraint that specifies an exact version for google-api-core, could the resolver use that information to avoid looking at many versions? If the [grpc] extra is not specified it seems pip immediately selects the correct version.

pip version

21.2.1

Python version

3.8

OS

Linux, probably not relevant

How to Reproduce

Assume the following setup.py, with a dependency on google-api-core with grpc extra:

from setuptools import setup

setup(
    name="pkga",
    install_requires=[
        "google-api-core[grpc]",
    ],
)

and requirements.txt which pins google-api-core:

google-api-core==1.14.3
grpcio==1.39.0

Running pip install -c requirements.txt -e . shows that pip is trying several versions of google-api-core:

Collecting google-api-core[grpc]
  Using cached google_api_core-1.31.1-py2.py3-none-any.whl (93 kB)
  Using cached google_api_core-1.31.0-py2.py3-none-any.whl (93 kB)
  Using cached google_api_core-1.30.0-py2.py3-none-any.whl (93 kB)
  Using cached google_api_core-1.29.0-py2.py3-none-any.whl (93 kB)
  Using cached google_api_core-1.28.0-py2.py3-none-any.whl (92 kB)
  Using cached google_api_core-1.27.0-py2.py3-none-any.whl (93 kB)
  Using cached google_api_core-1.26.3-py2.py3-none-any.whl (93 kB)
  Using cached google_api_core-1.26.2-py2.py3-none-any.whl (93 kB)
  Using cached google_api_core-1.26.1-py2.py3-none-any.whl (92 kB)
  Using cached google_api_core-1.26.0-py2.py3-none-any.whl (92 kB)
  Using cached google_api_core-1.25.1-py2.py3-none-any.whl (92 kB)
  Using cached google_api_core-1.25.0-py2.py3-none-any.whl (92 kB)
  Using cached google_api_core-1.24.1-py2.py3-none-any.whl (92 kB)
  Using cached google_api_core-1.24.0-py2.py3-none-any.whl (91 kB)
  Using cached google_api_core-1.23.0-py2.py3-none-any.whl (91 kB)
  Using cached google_api_core-1.22.4-py2.py3-none-any.whl (91 kB)
  Using cached google_api_core-1.22.3-py2.py3-none-any.whl (91 kB)
  Using cached google_api_core-1.22.2-py2.py3-none-any.whl (91 kB)
  Using cached google_api_core-1.22.1-py2.py3-none-any.whl (91 kB)
  Using cached google_api_core-1.22.0-py2.py3-none-any.whl (91 kB)
  Using cached google_api_core-1.21.0-py2.py3-none-any.whl (90 kB)
  Using cached google_api_core-1.20.1-py2.py3-none-any.whl (90 kB)
  Using cached google_api_core-1.19.1-py2.py3-none-any.whl (90 kB)
  Using cached google_api_core-1.19.0-py2.py3-none-any.whl (90 kB)
  Using cached google_api_core-1.18.0-py2.py3-none-any.whl (90 kB)
  Using cached google_api_core-1.17.0-py2.py3-none-any.whl (70 kB)
  Using cached google_api_core-1.16.0-py2.py3-none-any.whl (70 kB)
  Using cached google_api_core-1.15.0-py2.py3-none-any.whl (69 kB)
  Using cached google_api_core-1.14.3-py2.py3-none-any.whl (68 kB)
Collecting protobuf>=3.4.0
  Using cached protobuf-3.17.3-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
Collecting googleapis-common-protos<2.0dev,>=1.6.0
  Using cached googleapis_common_protos-1.53.0-py2.py3-none-any.whl (198 kB)
Collecting google-auth<2.0dev,>=0.4.0
  Using cached google_auth-1.34.0-py2.py3-none-any.whl (152 kB)
Collecting six>=1.10.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting requests<3.0.0dev,>=2.18.0
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Requirement already satisfied: setuptools>=34.0.0 in /home/sbi-local/.virtualenvs/tempenv-12e7210736546/lib/python3.8/site-packages (from google-api-core[grpc]->pkga==0.0.0) (57.0.0)
Collecting pytz
  Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting grpcio<2.0dev,>=1.8.2
  Using cached grpcio-1.39.0-cp38-cp38-manylinux2014_x86_64.whl (4.3 MB)
Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting cachetools<5.0,>=2.0.0
  Using cached cachetools-4.2.2-py3-none-any.whl (11 kB)
Collecting rsa<5,>=3.1.4
  Using cached rsa-4.7.2-py3-none-any.whl (34 kB)
Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.6-py2.py3-none-any.whl (138 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.3-py3-none-any.whl (35 kB)
Installing collected packages: six, pyasn1, urllib3, rsa, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, requests, pytz, googleapis-common-protos, google-auth, grpcio, google-api-core, pkga
  Running setup.py develop for pkga
Successfully installed cachetools-4.2.2 certifi-2021.5.30 charset-normalizer-2.0.3 google-api-core-1.14.3 google-auth-1.34.0 googleapis-common-protos-1.53.0 grpcio-1.39.0 idna-3.2 pkga-0.0.0 protobuf-3.17.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pytz-2021.1 requests-2.26.0 rsa-4.7.2 six-1.16.0 urllib3-1.26.6

Output

No response

Code of Conduct
@sbidoul sbidoul added the C: dependency resolution About choosing which dependencies to install label Jul 29, 2021
@pfmoore
Copy link
Member

pfmoore commented Jul 29, 2021

This is likely because the resolver treats google-api-core and google-api-core[grpc] as different candidates, with an implied constraint that they must both have the same version. But the explicit version constraint is on google-api-core and the resolver is "driving" the search from google-api-core[grpc], which therefore doesn't see that.

Extras are hard. And yucky.

It should be possible to "push" the explicit constraint down to the candidate with the extra, but it might be tricky to implement.

@uranusjr uranusjr added this to the 21.2.2 milestone Jul 30, 2021
@uranusjr uranusjr mentioned this issue Jul 30, 2021
Merged
@uranusjr
Copy link
Member

PR up in #10239.

inmantaci added a commit to inmanta/inmanta-core that referenced this issue Aug 2, 2021
@inmantaci
Bump pip from 21.2.1 to 21.2.2 (PR #3152)
787beed 
Bumps [pip](https://github.com/pypa/pip) from 21.2.1 to 21.2.2.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a href="https://github.com/pypa/pip/blob/main/NEWS.rst">pip's changelog</a>.</em></p>
<blockquote>
<h1>21.2.2 (2021-07-31)</h1>
<h2>Bug Fixes</h2>
<ul>
<li>New resolver: When a package is specified with extras in constraints, and with
extras in non-constraint requirements, the resolver now correctly identifies the
constraint's existence and avoids backtracking. (<code>[#10233](pypa/pip#10233) &lt;https://github.com/pypa/pip/issues/10233&gt;</code>_)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a href="https://github.com/pypa/pip/commit/f7d912a21902d11448d8fb19cbb84ba1854bcdc4"><code>f7d912a</code></a> Bump for release</li>
<li><a href="https://github.com/pypa/pip/commit/c7f3f197685d074855cfcd8e92234e7b93af3d3f"><code>c7f3f19</code></a> Update AUTHORS.txt</li>
<li><a href="https://github.com/pypa/pip/commit/08eee0b1c0b4805e7a2963e5b455fd7d22adf283"><code>08eee0b</code></a> Correctly normalize relative paths for 'pip show'</li>
<li><a href="https://github.com/pypa/pip/commit/aaba4997e4f6084bd9f0a360b39c14c0f1bd0278"><code>aaba499</code></a> Post a deprecation warning for distutils configs</li>
<li><a href="https://github.com/pypa/pip/commit/a2cbacfe1cb0a8c1a149385a819e39d4c1bd6eee"><code>a2cbacf</code></a> Respect the base's constraint for extra-ed package</li>
<li><a href="https://github.com/pypa/pip/commit/3d7b9c50342b1b0d4cba96016dcf82970d902d62"><code>3d7b9c5</code></a> Correctly ignore osx_framework_user mismatches</li>
<li><a href="https://github.com/pypa/pip/commit/5fc6d16050adbb8ad17f399b332de1fdad6c769d"><code>5fc6d16</code></a> Suppress location warning on abiflag differences</li>
<li><a href="https://github.com/pypa/pip/commit/cc5563e25172372f26de17663951c7b76223ab69"><code>cc5563e</code></a> Kill location warning on Deb and RH system Python</li>
<li><a href="https://github.com/pypa/pip/commit/ec41e0ca0c32688b42ed3bcded5c3d0cf3c12c4f"><code>ec41e0c</code></a> Patch tenacity to quote typing.NoReturn</li>
<li><a href="https://github.com/pypa/pip/commit/8a38cc61555673353952e260ff4b3ca9168c47ab"><code>8a38cc6</code></a> Fix broken link in <code>README.rst</code></li>
<li>Additional commits viewable in <a href="https://github.com/pypa/pip/compare/21.2.1...21.2.2">compare view</a></li>
</ul>
</details>
<br />

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pip&package-manager=pip&previous-version=21.2.1&new-version=21.2.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

</details>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 24, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
C: dependency resolution About choosing which dependencies to install
Projects
None yet
Milestone
21.2.2
Development

Successfully merging a pull request may close this issue.

Respect the base's constraint for extra-ed package uranusjr/pip
4 participants
@uranusjr @sbidoul @pfmoore and others