Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--require-hashes does not correctly handle pinned package with extras #9644

Open
tonybajan opened this issue Feb 22, 2021 · 57 comments · Fixed by #9995
Open

--require-hashes does not correctly handle pinned package with extras #9644

tonybajan opened this issue Feb 22, 2021 · 57 comments · Fixed by #9995
Labels
C: dependency resolution About choosing which dependencies to install type: bug A confirmed bug or unintended behavior

Comments

@tonybajan
Copy link

tonybajan commented Feb 22, 2021

What did you want to do?

Install this requirements.txt file with pip 21.0.1 or master in a new virtual environment.

A package is pinned to a non-latest version with an extra (here, requirements[security]) and another dependency requires this package without specifying the extra.

Output

Install fails with:

...
Collecting requests[security]==2.24.0
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3==1.25.11
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting requests<3
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    requests<3 from https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl#sha256=c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e (from requests-extra==1.0.0b3->-r requirements.txt (line 116))

The resolver does not recognise that requests[security]==2.24.0 fulfils requests<3, and tries to collect latest requests. This fails as it is has no pinned hash.

Additional information

Installation succeeds with --use-deprecated=legacy-resolver.

If the requirements file has no hashes, installation succeeds with the new resolver: Collecting requests<3 resolves to latest requests (2.25.1) in the install output, but the pinned version (2.24.0) is what ends up installed.

If all packages are already installed in the environment, pip install succeeds (with Requirement already satisfied) even with the new resolver.

@uranusjr uranusjr added C: new resolver type: bug A confirmed bug or unintended behavior labels Feb 22, 2021
@maljub01
Copy link

maljub01 commented Mar 1, 2021

One workaround for this is to use --no-deps, which is a good idea by itself anyway (See the second warning here).

@cas--
Copy link

cas-- commented Apr 5, 2021

I think this might be the same issue I have encountered with pinned and hash-checking botframework-connector requirements:

Broken requirements file: requirements.txt

Steps to reproduce:

pip install -U pip pip-tools
pip -V
> pip 21.0.1
echo "botframework-connector==4.12.0" > requirements.in
pip-compile requirements.in --generate-hashes
pip install -r requirements.txt

Actual result:

Collecting adal==1.2.1
  Using cached adal-1.2.1-py2.py3-none-any.whl (52 kB)
Collecting botbuilder-schema==4.12.0
  Using cached botbuilder_schema-4.12.0-py2.py3-none-any.whl (34 kB)
Collecting botframework-connector==4.12.0
  Using cached botframework_connector-4.12.0-py2.py3-none-any.whl (67 kB)
Collecting certifi==2020.12.5
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting cffi==1.14.5
  Using cached cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl (411 kB)
Collecting chardet==3.0.4
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting cryptography==3.3.2
  Using cached cryptography-3.3.2-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
Collecting idna==2.10
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting isodate==0.6.0
  Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting msal==1.6.0
  Using cached msal-1.6.0-py2.py3-none-any.whl (50 kB)
Collecting msrest==0.6.10
  Using cached msrest-0.6.10-py2.py3-none-any.whl (82 kB)
Collecting oauthlib==3.1.0
  Using cached oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
Collecting pycparser==2.20
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting pyjwt[crypto]==1.5.3
  Using cached PyJWT-1.5.3-py2.py3-none-any.whl (17 kB)
Collecting python-dateutil==2.8.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting requests-oauthlib==1.3.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting requests==2.23.0
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3==1.25.11
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting PyJWT>=1.0.0
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    PyJWT>=1.0.0 from https://files.pythonhosted.org/packages/b4/9b/8850f99027ed029af6828199cc87179eaccbbf1f9e6e373e7f0177d32dad/PyJWT-2.0.1-py3-none-any.whl#sha256=b70b15f89dc69b993d8a8d32c299032d5355c82f9b5b7e851d1a6d706dffe847 (from adal==1.2.1->-r requirements.txt (line 7))

Workaround

Either install pip <= 20.3.1, use tonybajan --use-deprecated=legacy-resolver option or use maljub01 --no-deps option:

pip install -r requirements.txt --use-deprecated=legacy-resolver
> Successfully installed adal-1.2.1 botbuilder-schema-4.12.0 botframework-connector-4.12.0 certifi-2020.12.5 cffi-1.14.5 chardet-3.0.4 cryptography-3.3.2 idna-2.10 isodate-0.6.0 msal-1.6.0 msrest-0.6.10 oauthlib-3.1.0 pycparser-2.20 pyjwt-1.5.3 python-dateutil-2.8.1 requests-2.23.0 requests-oauthlib-1.3.0 six-1.15.0 urllib3-1.25.11

pip install -r requirements.txt --no-deps
> Successfully installed adal-1.2.1 botbuilder-schema-4.12.0 botframework-connector-4.12.0 certifi-2020.12.5 cffi-1.14.5 chardet-3.0.4 cryptography-3.3.2 idna-2.10 isodate-0.6.0 msal-1.6.0 msrest-0.6.10 oauthlib-3.1.0 pycparser-2.20 pyjwt-1.5.3 python-dateutil-2.8.1 requests-2.23.0 requests-oauthlib-1.3.0 six-1.15.0 urllib3-1.25.11

Edit: I thought I had posted this as off-topic but after looking at the requirements.txt generated by pip-tools I see that it is specifying extras: pyjwt[crypto]

@di
Copy link
Sponsor Member

di commented Apr 24, 2021

If I understand correctly, #9775 did not fix this, but made it possible to fix this?

@uranusjr
Copy link
Member

uranusjr commented Apr 24, 2021

More accurately, the basis of #9775 (#9771) makes it possible to fix this.

@di
Copy link
Sponsor Member

di commented Apr 25, 2021

Thanks @uranusjr. This is currently blocking CI and deployment for pypa/warehouse, is there any workaround? If not, could you explain at a high level what I'd need to do to fix this issue in pip?

@uranusjr
Copy link
Member

I believe it’s possible to work around this by manually “unpacking” extras, e.g. add a line to also provide hashes to requests in addition to requests[security] (using the example provided by OP).

@di
Copy link
Sponsor Member

di commented May 18, 2021

I tried to create a failing test for this:

def test_new_resolver_hash_with_extras(script):
    parent_path = create_basic_wheel_for_package(
        script, "parent", "0.1.0", depends=["child"]
    )
    parent_hash = hashlib.sha256(parent_path.read_bytes()).hexdigest()

    child_path = create_basic_wheel_for_package(
        script, "child", "0.1.0", extras={"extra": ["extra"]}
    )
    child_hash = hashlib.sha256(child_path.read_bytes()).hexdigest()

    extra_path = create_basic_wheel_for_package(script, "extra", "0.1.0")
    extra_hash = hashlib.sha256(extra_path.read_bytes()).hexdigest()

    requirements_txt = script.scratch_path / "requirements.txt"
    requirements_txt.write_text(
        """
        child[extra]==0.1.0 --hash=sha256:{child_hash}
        parent==0.1.0 --hash=sha256:{parent_hash}
        extra==0.1.0 --hash=sha256:{extra_hash}
        """.format(
            child_hash=child_hash,
            parent_hash=parent_hash,
            extra_hash=extra_hash,
        ),
    )

    script.pip(
        "install",
        "--no-cache-dir",
        "--no-index",
        "--find-links", script.scratch_path,
        "--requirement", requirements_txt,
    )

    assert_installed(script, parent="0.1.0", child="0.1.0", extra="0.1.0")

but it's passing on the current main branch at e6414d6.

Is it possible that this has already been fixed, or is my test not properly exercising this bug?

@uranusjr
Copy link
Member

uranusjr commented May 18, 2021

I think the failure case should be like this:

child==0.1.0 --hash=sha256:{child_hash}
parent==0.1.0 --hash=sha256:{parent_hash}
extra==0.1.0 --hash=sha256:{extra_hash}

and change parent to depend on child[extra].

@di
Copy link
Sponsor Member

di commented May 18, 2021

That still passes, but after looking more closely I was able to figure out what will cause it to fail:

  • two parent dependencies
    • one with a dependency on a child with an extra
    • one with a dependency on a child without an extra
  • a newer version of the child dependency available than is pinned

It seems like what's happening is that pip is not considering the child dependency with the extra to be equivalent to the child dependency without the extra, and attempting to install the latest version of the child dependency, and not using the existing pinned/hashed requirement when doing this, which causes the hash checking to fail.

I made a draft PR with the failing test here: #9995, but I don't have time at the moment to continue working on it to fix the issue.

@cas--
Copy link

cas-- commented May 18, 2021

@di I have confirmed your theory by specifying another dependency without the extras package and everything installs

pyjwt[crypto]==1.5.3 \
    --hash=sha256:500be75b17a63f70072416843dc80c8821109030be824f4d14758f114978bae7 \
    --hash=sha256:a4e5f1441e3ca7b382fd0c0b416777ced1f97c64ef0c33bfa39daf38505cfd2f
+pyjwt==1.5.3 \
+    --hash=sha256:500be75b17a63f70072416843dc80c8821109030be824f4d14758f114978bae7 \
+    --hash=sha256:a4e5f1441e3ca7b382fd0c0b416777ced1f97c64ef0c33bfa39daf38505cfd2f

@di
Copy link
Sponsor Member

di commented May 18, 2021

@cas-- It may be preferable to update all the dependencies that the extra includes to the latest versions instead. By removing the extra you are possibly removing sub-dependencies your application (or a dependency of your application) needs.

bskinn added a commit to bskinn/warehouse that referenced this issue May 18, 2021
Should be removed before merging pypi#8615
bskinn added a commit to bskinn/warehouse that referenced this issue May 18, 2021
@cas--
Copy link

cas-- commented May 18, 2021

@di Yeah all dependencies are specified with pip-tools as demonstrated in my previous comment.

In my example it seems that child dependency pyjwt is specified with extras in msal and without in adal.

@di
Copy link
Sponsor Member

di commented May 18, 2021

@cas-- Ah, sorry, I missed that you were including both. That works too!

yozachar added a commit to yozachar/waka-readme that referenced this issue Jul 19, 2023
- upstream issue: pypa/pip#9644
- runs tests within a container, uses the same `dockerfile`
- ignores `pdm.lock`, removes `requirement.txt`
- update manual contribution steps
- bumps project version

**Related Items**

_Issues_

- Closes athul#137
yozachar added a commit to yozachar/waka-readme that referenced this issue Jul 19, 2023
- upstream issue: pypa/pip#9644
- runs tests within a container, uses the same `dockerfile`
- ignores `pdm.lock`, removes `requirement.txt`
- update manual contribution steps
- bumps project version

**Related Items**

_Issues_

- Closes athul#137
yozachar added a commit to yozachar/waka-readme that referenced this issue Jul 19, 2023
- upstream issue: pypa/pip#9644
- runs tests within a container, uses the same `dockerfile`
- ignores `pdm.lock`, removes `requirement.txt`
- update manual contribution steps
- bumps project version

**Related Items**

_Issues_

- Closes athul#137
yozachar added a commit to yozachar/waka-readme that referenced this issue Jul 19, 2023
- upstream issue: pypa/pip#9644
- runs tests within a container, uses the same `dockerfile`
- ignores `pdm.lock`, removes `requirement.txt`
- update manual contribution steps
- bumps project version

**Related Items**

_Issues_

- Closes athul#137
yozachar added a commit to yozachar/waka-readme that referenced this issue Jul 19, 2023
- upstream issue: pypa/pip#9644
- runs tests within a container, uses the same `dockerfile`
- ignores `pdm.lock`, removes `requirement.txt`
- update manual contribution steps
- bumps project version

**Related Items**

_Issues_

- Closes athul#137
yozachar added a commit to yozachar/waka-readme that referenced this issue Jul 19, 2023
- upstream issue: pypa/pip#9644
- runs tests within a container, uses the same `dockerfile`
- ignores `pdm.lock`, removes `requirement.txt`
- update manual contribution steps
- bumps project version

**Related Items**

_Issues_

- Closes athul#137
@thomas-riccardi
Copy link

FWIW #9995 did not fix the following case we encountered: one constraint with 3 extra, and another dependency constraint with 1 extra, subset of the previous 3 extra:

absl-py==1.4.0 \
--hash=sha256:0d3fe606adfa4f7db64792dd4c7aee4ee0c38ab75dfd353b7a83ed3e957fcb47 \
--hash=sha256:d2c244d01048ba476e7c080bd2c6df5e141d211de80223460d5b3b8a2a58433d
# via
#   array-record
#   tensorflow-datasets
#   [... others]

array-record==0.2.0 \
    --hash=sha256:12ce6844f8acb2e65f0bc4d8bcecbe19ac45a39cd2ba5bb56828668f118b1e87 \
    --hash=sha256:4b9335c7e21b54f559bada68b26f79309903015ff65101d4a3c3c42c62658398 \
    --hash=sha256:d3b9a3a0d11f43a06a37fd8129d78e2894d7ff65b5fa53def198698c5592562a

etils[enp,epath,epy]==1.3.0 \
--hash=sha256:0a695ec45a982ae7c9deb437f1f251346d88b43ca59be67e961f61fe8bc8cae4 \
--hash=sha256:809a92ff72f12149441492cf4d9a26b56a4741dffb4dfb9c4c7b7afe055c2d28
# via
#   array-record
#   tensorflow-datasets [removed from minimal repro, but could explain the current state of this pip-freeze requirements.txt]

With pip 23.2.1:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    etils[epath] from https://files.pythonhosted.org/packages/4a/6a/d58ec120f5e4babbf5001c144266ba623dcdae8e81dc6cdb422a98d0e0ce/etils-1.4.1-py3-none-any.whl (from array-record==0.2.0->-r requirements.txt (line 9))

etils[epath] and etils[enp,epath,epy] should be considered identical for the hash PoV, like the initial bug report: with vs without extra.

exhuma added a commit to exhuma/httpbin that referenced this issue Aug 18, 2023
fbuireu added a commit to fbuireu/waka-readme that referenced this issue Sep 10, 2023
* chore: monthly updates for July'23

- migrate to `pdm` from `poetry`
- update dependencies
- improvises `dockerfile` (drops `containerfile`)
- only `python` & `pip` are required in ci/cd
- adds `compose.yml`
- simplifies local & containerized development

* fix: license format from choosealicense.com

* feat: flags to tweak commit info; bump version

- flags are now available to modify commit information
- adds these flags to `dockerfile` & `action.yml`
- fixes a few typos
- improves `Readme.md`
- updates secondary dependencies
- bumps project version

**Related Items**

_Issues_

- Closes athul#103
- Closes athul#104
- Closes athul#125

* patch: work around pip's secure-install bug; bump version

- upstream issue: pypa/pip#9644
- runs tests within a container, uses the same `dockerfile`
- ignores `pdm.lock`, removes `requirement.txt`
- update manual contribution steps
- bumps project version

**Related Items**

_Issues_

- Closes athul#137

* chore: monthly updates for Aug'23; bump version

- revised readme, better theming support, improved wording where needed
- granular config for ruff linter & pyright type checker, re-format code
- update dependencies, bump version

---------

Co-authored-by: Jovial Joe Jayarson <jovial7joe@hotmail.com>
crungehottman added a commit to crungehottman/exodus-lambda that referenced this issue Sep 29, 2023
The tomli package is required by coverage (a dependency of pytest-cov
and coveralls), but the version is unpinned when upgrading to python
3.11.

`ERROR: In --require-hashes mode, all requirements must have their
versions pinned with ==. These do not:
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 572.7/572.7 kB 86.5 MB/s eta 0:00:00
    tomli from
https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl
(from coverage[toml]==6.5.0->-r test-requirements.txt (line 173))`

This seems to be caused by a bug in pip [0], which results in
incorrect handling of extra dependencies. To work around this issue,
the tomli package version must be pinned.

[0] pypa/pip#9644
crungehottman added a commit to crungehottman/exodus-lambda that referenced this issue Sep 29, 2023
The tomli package is required by coverage (a dependency of pytest-cov
and coveralls), but the version is unpinned when upgrading to python
3.11.

`ERROR: In --require-hashes mode, all requirements must have their
versions pinned with ==. These do not:
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 572.7/572.7 kB 86.5 MB/s eta 0:00:00
    tomli from
https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl
(from coverage[toml]==6.5.0->-r test-requirements.txt (line 173))`

This seems to be caused by a bug in pip [0], which results in
incorrect handling of extra dependencies. To work around this issue,
the tomli package version must be pinned.

[0] pypa/pip#9644
@Niorlys
Copy link

Niorlys commented Oct 3, 2023

I am still getting this issue with urllib3 suddenly:

Collecting zipp==3.16.2
Downloading zipp-3.16.2-py3-none-any.whl (7.2 kB)
Requirement already satisfied: tomli in /home/userx/.pyenv/versions/3.10.8/lib/python3.10/site-packages (from autopep8==2.0.2->-r /tmp/tmpx2vsi8n_ (line 115)) (2.0.1)
Requirement already satisfied: packaging>=22.0 in /home/userx/.pyenv/versions/3.10.8/lib/python3.10/site-packages (from black==23.7.0->-r /tmp/tmpx2vsi8n_ (line 121)) (23.1)
Collecting urllib3<2.0
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
urllib3<2.0 from https://files.pythonhosted.org/packages/48/fe/a5c6cc46e9fe9171d7ecf0f33ee7aae14642f8d74baa7af4d7840f9358be/urllib3-1.26.17-py2.py3-none-any.whl (from google-auth==2.22.0->-r /tmp/tmpx2vsi8n_ (line 527))

@anentropic

This comment was marked as off-topic.

@pradyunsg

This comment was marked as off-topic.

@anentropic

This comment was marked as off-topic.

@pfmoore

This comment was marked as off-topic.

@anentropic

This comment was marked as off-topic.

@pradyunsg

This comment was marked as off-topic.

@anentropic

This comment was marked as off-topic.

@merwok

This comment was marked as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C: dependency resolution About choosing which dependencies to install type: bug A confirmed bug or unintended behavior
Projects
None yet