Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2020 resolver does not consider hashes in the constraints file #8792

Closed
jwhitlock opened this issue Aug 21, 2020 · 15 comments · Fixed by #8839
Closed

2020 resolver does not consider hashes in the constraints file #8792

jwhitlock opened this issue Aug 21, 2020 · 15 comments · Fixed by #8839
Assignees
Labels
type: bug A confirmed bug or unintended behavior

Comments

@jwhitlock
Copy link

jwhitlock commented Aug 21, 2020

See:

Update 1:
The issue was not the projects multiple requirements file, but instead the combinations of using hashes and a constraints files. The current resolver uses hashes on a requirement in a constraints file, while the 2020 resolver ignores them, and fails to install because they do not have hashes. The description below describes the more complex version.

Update 2:
Merged in the changes, so the default branch of pip-resolver-demo describes the simpler version, and includes the django.txt / django-versions.txt example as well.

What did you want to do?

Our requirements files include other files, as a way to only specify a requirement once for two different environments (development and building in ReadTheDocs.org):

  • default.txt: -c constraints.txt, -r docs.txt, -r shared.txt
  • docs.txt: -c constraints.txt, -r shared.txt
  • shared.txt: None
  • constraints.txt: None

All of our requirements are specified with hashes, populated with hashin.

This works, without warnings, when installing with pip install -r default.txt and pip 20.2.2.

When installing with pip 20.2.2 or pip-20.3.dev0 (today's in-development version), this fails:

pip install -r default.txt --use-feature=2020-resolver
....
Collecting idna<3,>=2.5
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    idna<3,>=2.5 from https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl#sha256=b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 (from requests[security]==2.24.0->-r shared.txt (line 6))

The idna==2.10 requirement is in constraints.txt, with the sha256 hash.

A similar error occurs when installing docs.txt:

pip install -r docs.txt --use-feature=2020-resolver
...
Collecting chardet<4,>=3.0.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    chardet<4,>=3.0.2 from https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl#sha256=fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 (from requests[security]==2.24.0->-r shared.txt (line 6))

Output

Here's pip install -r default.txt --use-feature=2020-resolver:

Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting requests[security]==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting urllib3[secure]==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting Sphinx==3.1.2
  Downloading Sphinx-3.1.2-py3-none-any.whl (2.9 MB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from Sphinx==3.1.2->-r docs.txt (line 6)) (49.3.1)
Collecting geoip2==4.0.2
  Downloading geoip2-4.0.2-py2.py3-none-any.whl (25 kB)
Collecting maxminddb==2.0.2
  Downloading maxminddb-2.0.2.tar.gz (285 kB)
Collecting idna<3,>=2.5
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    idna<3,>=2.5 from https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl#sha256=b97d804b1e9b523befed77c48dacec60e6dcb0b5391d57af6a65a312a90648c0 (from requests[security]==2.24.0->-r shared.txt (line 6))

Here's pip install -r docs.txt --use-feature=2020-resolver:

Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting requests[security]==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting urllib3[secure]==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting Sphinx==3.1.2
  Downloading Sphinx-3.1.2-py3-none-any.whl (2.9 MB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from Sphinx==3.1.2->-r docs.txt (line 6)) (49.3.1)
Collecting chardet<4,>=3.0.2
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    chardet<4,>=3.0.2 from https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl#sha256=fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691 (from requests[security]==2.24.0->-r shared.txt (line 6))

For comparison, here's pip install -r default.txt:

Collecting certifi==2020.6.20
  Downloading certifi-2020.6.20-py2.py3-none-any.whl (156 kB)
Collecting requests[security]==2.24.0
  Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting urllib3[secure]==1.25.10
  Downloading urllib3-1.25.10-py2.py3-none-any.whl (127 kB)
Collecting Sphinx==3.1.2
  Downloading Sphinx-3.1.2-py3-none-any.whl (2.9 MB)
Collecting geoip2==4.0.2
  Downloading geoip2-4.0.2-py2.py3-none-any.whl (25 kB)
Collecting maxminddb==2.0.2
  Downloading maxminddb-2.0.2.tar.gz (285 kB)
Collecting idna==2.10
  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting chardet==3.0.4
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting pyOpenSSL==19.1.0
  Downloading pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
Collecting cryptography==3.0
  Downloading cryptography-3.0-cp35-abi3-manylinux2010_x86_64.whl (2.7 MB)
Collecting packaging==20.4
  Downloading packaging-20.4-py2.py3-none-any.whl (37 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.8/site-packages (from Sphinx==3.1.2->-r docs.txt (line 6)) (49.3.1)
Collecting Babel==2.8.0
  Downloading Babel-2.8.0-py2.py3-none-any.whl (8.6 MB)
Collecting docutils==0.15.2
  Downloading docutils-0.15.2-py3-none-any.whl (547 kB)
Collecting sphinxcontrib-applehelp==1.0.2
  Downloading sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl (121 kB)
Collecting imagesize==1.2.0
  Downloading imagesize-1.2.0-py2.py3-none-any.whl (4.8 kB)
Collecting Jinja2==2.11.2
  Downloading Jinja2-2.11.2-py2.py3-none-any.whl (125 kB)
Collecting sphinxcontrib-devhelp==1.0.2
  Downloading sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
Collecting sphinxcontrib-htmlhelp==1.0.3
  Downloading sphinxcontrib_htmlhelp-1.0.3-py2.py3-none-any.whl (96 kB)
Collecting alabaster==0.7.12
  Downloading alabaster-0.7.12-py2.py3-none-any.whl (14 kB)
Collecting sphinxcontrib-jsmath==1.0.1
  Downloading sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
Collecting Pygments==2.6.1
  Downloading Pygments-2.6.1-py3-none-any.whl (914 kB)
Collecting snowballstemmer==2.0.0
  Downloading snowballstemmer-2.0.0-py2.py3-none-any.whl (97 kB)
Collecting sphinxcontrib-serializinghtml==1.1.4
  Downloading sphinxcontrib_serializinghtml-1.1.4-py2.py3-none-any.whl (89 kB)
Collecting sphinxcontrib-qthelp==1.0.3
  Downloading sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
Collecting aiohttp==3.6.2
  Downloading aiohttp-3.6.2-py3-none-any.whl (441 kB)
Collecting six==1.15.0
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting cffi==1.14.1
  Downloading cffi-1.14.1-cp38-cp38-manylinux1_x86_64.whl (409 kB)
Collecting pyparsing==2.4.7
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting pytz==2020.1
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Collecting MarkupSafe==1.1.1
  Downloading MarkupSafe-1.1.1.tar.gz (19 kB)
Collecting multidict==4.7.6
  Downloading multidict-4.7.6-cp38-cp38-manylinux1_x86_64.whl (162 kB)
Collecting yarl==1.5.1
  Downloading yarl-1.5.1-cp38-cp38-manylinux1_x86_64.whl (262 kB)
Collecting async-timeout==3.0.1
  Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
Collecting attrs==19.3.0
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting pycparser==2.20
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Building wheels for collected packages: MarkupSafe, maxminddb
  Building wheel for MarkupSafe (setup.py): started
  Building wheel for MarkupSafe (setup.py): finished with status 'done'
  Created wheel for MarkupSafe: filename=MarkupSafe-1.1.1-py3-none-any.whl size=12629 sha256=2f6a74f92d30e9e5c58a60f6b78a85229e279ee2229e4fced3d0c2c4e25c3d11
  Stored in directory: /root/.cache/pip/wheels/0c/61/d6/4db4f4c28254856e82305fdb1f752ed7f8482e54c384d8cb0e
  Building wheel for maxminddb (setup.py): started
  Building wheel for maxminddb (setup.py): finished with status 'done'
  Created wheel for maxminddb: filename=maxminddb-2.0.2-py3-none-any.whl size=15259 sha256=015ad8503c939cf424e3fedf475b0a3e81dd4dffbc95ed04e55235b8ce8d9f3c
  Stored in directory: /root/.cache/pip/wheels/8f/e8/68/7354267262db5ca21a7f869d544e703dcd7227b244db984b52
Successfully built MarkupSafe maxminddb
Installing collected packages: alabaster, pytz, Babel, pycparser, cffi, chardet, six, cryptography, docutils, idna, imagesize, MarkupSafe, Jinja2, Pygments, pyOpenSSL, snowballstemmer, sphinxcontrib-applehelp, sphinxcontrib-htmlhelp, sphinxcontrib-jsmath, sphinxcontrib-devhelp, sphinxcontrib-serializinghtml, pyparsing, packaging, sphinxcontrib-qthelp, attrs, multidict, yarl, async-timeout, aiohttp, certifi, urllib3, requests, Sphinx, maxminddb, geoip2
Successfully installed Babel-2.8.0 Jinja2-2.11.2 MarkupSafe-1.1.1 Pygments-2.6.1 Sphinx-3.1.2 aiohttp-3.6.2 alabaster-0.7.12 async-timeout-3.0.1 attrs-19.3.0 certifi-2020.6.20 cffi-1.14.1 chardet-3.0.4 cryptography-3.0 docutils-0.15.2 geoip2-4.0.2 idna-2.10 imagesize-1.2.0 maxminddb-2.0.2 multidict-4.7.6 packaging-20.4 pyOpenSSL-19.1.0 pycparser-2.20 pyparsing-2.4.7 pytz-2020.1 requests-2.24.0 six-1.15.0 snowballstemmer-2.0.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 urllib3-1.25.10 yarl-1.5.1

Additional information

Full dependency tree:

geoip2==4.0.2
  - aiohttp [required: >=3.6.2,<4.0.0, installed: 3.6.2]
    - async-timeout [required: >=3.0,<4.0, installed: 3.0.1]
    - attrs [required: >=17.3.0, installed: 19.3.0]
    - chardet [required: >=2.0,<4.0, installed: 3.0.4]
    - multidict [required: >=4.5,<5.0, installed: 4.7.6]
    - yarl [required: >=1.0,<2.0, installed: 1.5.1]
      - idna [required: >=2.0, installed: 2.10]
      - multidict [required: >=4.0, installed: 4.7.6]
  - maxminddb [required: >=2.0.0,<3.0.0, installed: 2.0.2]
  - requests [required: >=2.24.0,<3.0.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - urllib3 [required: >=1.25.2,<2.0.0, installed: 1.25.10]
pipdeptree==1.0.0
  - pip [required: >=6.0.0, installed: 20.2.2]
pyOpenSSL==19.1.0
  - cryptography [required: >=2.8, installed: 3.0]
    - cffi [required: >=1.8,!=1.11.3, installed: 1.14.1]
      - pycparser [required: Any, installed: 2.20]
    - six [required: >=1.4.1, installed: 1.15.0]
  - six [required: >=1.5.2, installed: 1.15.0]
Sphinx==3.1.2
  - alabaster [required: >=0.7,<0.8, installed: 0.7.12]
  - babel [required: >=1.3, installed: 2.8.0]
    - pytz [required: >=2015.7, installed: 2020.1]
  - docutils [required: >=0.12, installed: 0.15.2]
  - imagesize [required: Any, installed: 1.2.0]
  - Jinja2 [required: >=2.3, installed: 2.11.2]
    - MarkupSafe [required: >=0.23, installed: 1.1.1]
  - packaging [required: Any, installed: 20.4]
    - pyparsing [required: >=2.0.2, installed: 2.4.7]
    - six [required: Any, installed: 1.15.0]
  - Pygments [required: >=2.0, installed: 2.6.1]
  - requests [required: >=2.5.0, installed: 2.24.0]
    - certifi [required: >=2017.4.17, installed: 2020.6.20]
    - chardet [required: >=3.0.2,<4, installed: 3.0.4]
    - idna [required: >=2.5,<3, installed: 2.10]
    - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.10]
  - setuptools [required: Any, installed: 49.6.0]
  - snowballstemmer [required: >=1.1, installed: 2.0.0]
  - sphinxcontrib-applehelp [required: Any, installed: 1.0.2]
  - sphinxcontrib-devhelp [required: Any, installed: 1.0.2]
  - sphinxcontrib-htmlhelp [required: Any, installed: 1.0.3]
  - sphinxcontrib-jsmath [required: Any, installed: 1.0.1]
  - sphinxcontrib-qthelp [required: Any, installed: 1.0.3]
  - sphinxcontrib-serializinghtml [required: Any, installed: 1.1.4]
wheel==0.35.1
@uranusjr
Copy link
Member

uranusjr commented Aug 22, 2020

We completely overlooked this 🙂

Constraints went through a minor re-design in the new resolver, so there are a few decisions we need to make:

  1. Do we want to continue to support this usage? I’m inclined to say yes since this usage essentially adds further restrictions on what pip can choose when downloading distributions, which is not unlike version specifiers. Discussions in What does a URL constraint mean? #8253 seems like we are open to applying this kind of “adding further restrictions” semantic to constraints. (Note: we’ll still need to add an error message if we decide to not support this. The current implementation silently ignores hashes.)
  2. What should we do when hashes are specified in both constraints and requirements? There are three choices I can think of: requirements override constraints, union, and intersection. My intuition goes to intersection (i.e. the constraints restrict the list of hashes further) since that’s how version specifiers are merged. what does the current resolver do? What happens when a package appear multiple times in an install command with different list of hashes?

@jwhitlock
Copy link
Author

jwhitlock commented Aug 22, 2020

Interesting, I did not expect that hashes in the constraints file wouldn't be processed, so I was wrong about the cause of my issue. I've made my example simpler in a branch (https://github.com/jwhitlock/pip-resolver-demo/tree/just-constraints). I've added a note to my original bug, and plan to change the title, but I haven't re-written my original bug entirely.

My mental model of a constraints file is that it is the requirements of the requirements that I care about. For example, I use Sphinx to generate documentation, but I don't particularly care about sphinxcontrib-qthelp, it just comes along for the ride. I've evangelized for this model with https://stackoverflow.com/questions/34645821/pip-constraints-files/36848206#36848206.

This separation supports the way I approach package updates:

  • When updating the project requirements:
    • Update a few weeks after each release
    • Read the changelog, and think about project changes before and after the update
    • Run automated tests after updating
    • Run some manual tests around the project functionality that uses that requirement
    • Run the project in a staging environment overnight
  • When updating the requirements of requirements (which I stick in constraints)
    • Batch a bunch at once, otherwise only update for security issues
    • Skip the changelog unless it is easy to find and skim
    • Let continuous integration run automated tests
    • Skip manual tests
    • Briefly run on stage before going to production

The other nice feature of constraints is that, if a project requirement stopped requiring them, then they didn't get installed. Otherwise, it seems like I could solve my issue by changing -c constraints.txt to -r constraints.txt.

With that preamble, my answers are:

  1. Do we want to continue to support this usage? I hope so. When installing with hashes, the hash is as important as a version for the constraint. If someone is going to do something funky, I could imagine a new build for a widely used but somewhat boring requirement-of-requirements, and I'd want the hash to signal the shenanigans.
  2. What should we do when hashes are specified in both constraints and requirements? For my use case, an intersection (only hashes in all specs) would work. I'd go a step further, and say the hash list must be identical if present. This treats the version number + sorted hash list as the full package restriction.

I think other people's legit use would be a requirements.txt file with:

Django>=3.0

and a constraints.txt with

Django==3.1 \
    --hash=sha256:1a63f5bb6ff4d7c42f62a519edc2adbb37f9b78068a5a862beff858b68e3dc8b \
    --hash=sha256:2d390268a13c655c97e0e2ede9d117007996db692c1bb93eabebd4fb7ea7012b
pytz==2020.1 \
    --hash=sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed \
    --hash=sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048
sqlparse==0.3.1 \
    --hash=sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e \
    --hash=sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548
asgiref==3.2.10 \
    --hash=sha256:7e51911ee147dd685c3c8b805c0ad0cb58d360987b56953878f8c06d2d1c6f1a \
    --hash=sha256:9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aed

so that

pip install -r requirements.txt -c constraints.txt --use-feature=2020-resolver

works and securely installs the listed packages.

@jwhitlock jwhitlock changed the title 2020 resolver fails to find hashes in constraint file from included requirement 2020 resolver does not consider hashes in the constraints file Aug 22, 2020
@pradyunsg pradyunsg added this to Needs triage in Resolver Rollout via automation Aug 23, 2020
@pradyunsg pradyunsg added this to To do before prod release in New Resolver Implementation Aug 23, 2020
@pradyunsg pradyunsg moved this from Needs triage to To do before prod release in Resolver Rollout Aug 23, 2020
@pradyunsg pradyunsg added S: needs triage Issues/PRs that need to be triaged C: new resolver labels Aug 30, 2020
@pradyunsg
Copy link
Member

pradyunsg commented Aug 30, 2020

@pfmoore Do you have concerns with (re?)introducing this behaviour, for this use-case?

@pfmoore
Copy link
Member

pfmoore commented Aug 30, 2020

tl;dr; I feel that we should be cautious, but I don't have a strong objection.

I wouldn't describe the new resolver changes as a "minor redesign" - I think we did a fairly comprehensive overhaul and stripped constraints files down to being purely a way to specify global (version) limits for packages. That gave us a very specific behaviour, that translates well to how pip decides which version to install (don't even consider versions that don't match the limits in the constraint file).

I'm fine with adding extra functionality to constraint files that is in the same vein - removing certain candidates from consideration up front. As @uranusjr says, hashes can be viewed in this way, and so I feel that it's a reasonable extension. However, that logic does mean that IMO the only reasonable interpretation of the case when hashes are in both constraints and requirements is "intersection" - pip never considers anything that doesn't match the constraints, so if a requirement has a conflicting hash, pip will see nothing and hence will fail. I'd be unhappy with "requirements override" or "union" because they don't fit that model.

My mental model of a constraints file is that it is the requirements of the requirements that I care about.

That's not how I viewed them when I designed the new implementation. As I said above, the intended interpretation is that constraints limit what candidates pip sees. Most of the time, I don't expect there to be much difference in the viewpoints (which has positive and negative aspects - we won't conflict with people's expectations, which is good, but people may want to push constraints in a direction we didn't intend, which is less so...)

@uranusjr
Copy link
Member

uranusjr commented Aug 31, 2020

Some additional context:

pip currently merges hash lists (i.e. a union operation) if a requirement is specified multiple times. So this

# This one points to the sdist.
six==1.15.0 --hash=sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259

# This points to the wheel.
six==1.15.0 --hash=sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced

has the same effect as

six==1.15.0 \
    --hash=sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259 \
    --hash=sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced

The ordering of hashes does not matter, nor whether the line is a constraint or requirement. So if we’re going to change the constraint file’s behaviour, we should probably also change the merge logic in requirements.txt as well. But the problem is, the merge is currently done during the parsing stage, and shared between both resolvers. This can be tricky to change 😥

@pfmoore
Copy link
Member

pfmoore commented Aug 31, 2020

Ah. Yes, that will be tricky. But IMO having a consistent model is the important thing (something pip has traditionally not been very good at doing 🙁) so we should either come up with a different model for constraints that makes union the "natural" interpretation¹, or bite the bullet and make the tricky change. Personally, I like the current model as it's very easy to explain.

¹ And, of course, review the existing constraints code to make sure it conforms to the changed model 🤷

@pradyunsg
Copy link
Member

pradyunsg commented Aug 31, 2020

Where does the merging occur?

My understanding was that we were handling the hash-checking-related rejections in Provider.find_matches (and, finally, in a loop in whatever the relevant method on Factory is).

@pradyunsg pradyunsg self-assigned this Sep 2, 2020
@pradyunsg pradyunsg removed this from To do before prod release in Resolver Rollout Sep 2, 2020
@uranusjr
Copy link
Member

uranusjr commented Sep 3, 2020

The merging is done in the Hashes class (which is on InstallRequirement). I traced the code and believe it’s possible to only change the new resolver implementation to do the intersection logic (and keep the legacy resolver as-is, performing union). PR coming shortly.

@pradyunsg
Copy link
Member

pradyunsg commented Sep 3, 2020

I traced the code and believe it’s possible to only change the new resolver implementation to do the intersection logic (and keep the legacy resolver as-is, performing union).

I did this too, and I was wondering whether I was missing something. :)

PR coming shortly.

\o/

@uranusjr uranusjr added C: debug The debug command type: bug A confirmed bug or unintended behavior and removed S: needs triage Issues/PRs that need to be triaged C: debug The debug command labels Sep 3, 2020
@sbidoul
Copy link
Member

sbidoul commented Sep 4, 2020

Intersection sounds the most natural to me too. Just one question: what if we have foo and foo --hash .... Does it result in an empty hash set? Or is it simply not allowed because hash checking mode rejects requirements with no hashes?

@pfmoore
Copy link
Member

pfmoore commented Sep 4, 2020

Do you have a real world use case? If you're asking in a purely theoretical sense, then I'd say:

  1. foo must appear in a genuine requirement (a constraint of foo by itself makes no sense).
  2. So we're talking about foo in a requirement and foo --hash in a constraint.

I'd say that the result should be that only versions of foo with the given hash can be selected (i.e., intersection) but it wouldn't trigger full hash checking mode, because hashes in constraint files don't do that.

But I don't really like that behaviour, because it would encourage people to use constraints for "only hash check one project" mode. An alternative would be to disallow hashes in constraints files unless hash checking mode is in force (which would mean that a constraint file with hashes in it couldn't be used with requirements that omit hashes).

It's hard to say which is best without a proper use case, though.

The bigger question is whether constraints files can even include some projects with hashes and others without. Hash checking is supposed to be all-or-nothing, and having constraints files that could add hashes to some projects but not others could be very confusing. Maybe what we should really be doing is saying that hashes in constraint files should also be all-or-nothing, and therefore that if a constraint file contains hashes, all projects in that constraint file must have hashes, and using the constraint file triggers hash checking mode, meaning that any requirements not limited by a constraint must have a hash specified in the requirements.

But all of this is guessing. We don't have a good enough conceptual model of has checking (IMO - or if we do, it's not documented well enough) and we don't have enough real use cases to infer what rules would be useful.

@sbidoul
Copy link
Member

sbidoul commented Sep 4, 2020

Do you have a real world use case?

Not really. The current hash checking mode is not practical for my use cases because it does not work with VCS URLs and it disables the wheel cache.

So I'm probing the corner cases to try to make myself a mental model of what would work and what not. And the model is not clear yet, for sure :)

What could work is to say that when a constraint gets activated and we are in hash checking mode, that constraint must have hashes, just like other requirements. And then do the intersection.

@uranusjr
Copy link
Member

uranusjr commented Sep 4, 2020

what if we have foo and foo --hash .... Does it result in an empty hash set? Or is it simply not allowed because hash checking mode rejects requirements with no hashes?

You would get

Hashes are required in --require-hashes mode, but they are
missing from some requirements. Here is a list of those
requirements along with the hashes their downloaded archives
actually had. Add lines like these to your requirements files to
prevent tampering. (If you did not enable --require-hashes
manually, note that it turns on automatically when any package
has a hash.)

Indeed there is a theoratical use case, but I’m happy to keep it as-is and not worry about it.

bors bot added a commit to duckinator/emanate that referenced this issue Oct 18, 2020
192: Update pip to 20.2.4 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.3** to **20.2.4**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.2.4
   ```
   ===================

Deprecations and Removals
-------------------------

- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)

Features
--------

- New resolver: Avoid accessing indexes when the installed candidate is preferred
  and considered good enough. (`8023 &lt;https://github.com/pypa/pip/issues/8023&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)

Bug Fixes
---------

- New resolver: Correctly respect ``Requires-Python`` metadata to reject
  incompatible packages in ``--no-deps`` mode. (`8758 &lt;https://github.com/pypa/pip/issues/8758&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)

Improved Documentation
----------------------

- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
@jwhitlock
Copy link
Author

jwhitlock commented Oct 20, 2020

The 20.2.4 release didn't fix my issue. Should I open a new bug or re-open this one?

Consider this requirements.txt:

-c constraints.txt
Django==3.1 \
    --hash=sha256:1a63f5bb6ff4d7c42f62a519edc2adbb37f9b78068a5a862beff858b68e3dc8b \
    --hash=sha256:2d390268a13c655c97e0e2ede9d117007996db692c1bb93eabebd4fb7ea7012b

and this constraints.txt:

pytz==2020.1 \
    --hash=sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed \
    --hash=sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048
sqlparse==0.3.1 \
    --hash=sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e \
    --hash=sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548
asgiref==3.2.10 \
    --hash=sha256:7e51911ee147dd685c3c8b805c0ad0cb58d360987b56953878f8c06d2d1c6f1a \
    --hash=sha256:9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aed

This works:

pip install -r requirements.txt

and this fails:

pip install -r requirements.txt --use-feature=2020-resolver

with:

ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    asgiref~=3.2.10 from https://files.pythonhosted.org/packages/d5/eb/64725b25f991010307fd18a9e0c1f0e6dff2f03622fc4bcbcdb2244f60d6/asgiref-3.2.10-py3-none-any.whl#sha256=9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aed (from Django==3.1->-r requirements.txt (line 2))

@uranusjr
Copy link
Member

uranusjr commented Oct 20, 2020

@jwhitlock I think the fix to the issue is surfacing another different bug (that was not triggered previously since pip fails before hitting it). It would be best to track it in another issue.

brainwane added a commit to brainwane/pip that referenced this issue Oct 27, 2020
brainwane added a commit to brainwane/pip that referenced this issue Oct 27, 2020
brainwane added a commit to brainwane/pip that referenced this issue Oct 27, 2020
brainwane added a commit to brainwane/pip that referenced this issue Oct 28, 2020
brainwane added a commit to brainwane/pip that referenced this issue Oct 29, 2020
brainwane added a commit to brainwane/pip that referenced this issue Oct 29, 2020
bors bot added a commit to duckinator/emanate that referenced this issue Dec 1, 2020
195: Update pip to 20.3 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.3
   ```
   - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 &lt;https://github.com/pypa/pip/issues/8546&gt;`_, `8377 &lt;https://github.com/pypa/pip/issues/8377&gt;`_)
- Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 &lt;https://github.com/pypa/pip/issues/6741&gt;`_)
- Warn if index pages have unexpected content-type (`6754 &lt;https://github.com/pypa/pip/issues/6754&gt;`_)
- Allow specifying ``--prefer-binary`` option in a requirements file (`7693 &lt;https://github.com/pypa/pip/issues/7693&gt;`_)
- Generate PEP 376 REQUESTED metadata for user supplied requirements installed
  by pip. (`7811 &lt;https://github.com/pypa/pip/issues/7811&gt;`_)
- Warn if package url is a vcs or an archive url with invalid scheme (`8128 &lt;https://github.com/pypa/pip/issues/8128&gt;`_)
- Parallelize network operations in ``pip list``. (`8504 &lt;https://github.com/pypa/pip/issues/8504&gt;`_)
- Allow the new resolver to obtain dependency information through wheels
  lazily downloaded using HTTP range requests.  To enable this feature,
  invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 &lt;https://github.com/pypa/pip/issues/8588&gt;`_)
- Support ``--use-feature`` in requirements files (`8601 &lt;https://github.com/pypa/pip/issues/8601&gt;`_)

Bug Fixes
---------

- Use canonical package names while looking up already installed packages. (`5021 &lt;https://github.com/pypa/pip/issues/5021&gt;`_)
- Fix normalizing path on Windows when installing package on another logical disk. (`7625 &lt;https://github.com/pypa/pip/issues/7625&gt;`_)
- The VCS commands run by pip as subprocesses don&#39;t merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 &lt;https://github.com/pypa/pip/issues/7968&gt;`_)
- Correctly treat non-ASCII entry point declarations in wheels so they can be
  installed on Windows. (`8342 &lt;https://github.com/pypa/pip/issues/8342&gt;`_)
- Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 &lt;https://github.com/pypa/pip/issues/8454&gt;`_)
- Headers provided by wheels in .data directories are now correctly installed
  into the user-provided locations, such as ``--prefix``, instead of the virtual
  environment pip is running in. (`8521 &lt;https://github.com/pypa/pip/issues/8521&gt;`_)

Vendored Libraries
------------------

- Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3.
- Upgrade appdirs to 1.4.4
- Upgrade certifi to 2020.6.20
- Upgrade distlib to 0.3.1
- Upgrade html5lib to 1.1
- Upgrade idna to 2.10
- Upgrade packaging to 20.4
- Upgrade requests to 2.24.0
- Upgrade six to 1.15.0
- Upgrade toml to 0.10.1
- Upgrade urllib3 to 1.25.9

Improved Documentation
----------------------

- Add ``--no-input`` option to pip docs (`7688 &lt;https://github.com/pypa/pip/issues/7688&gt;`_)
- List of options supported in requirements file are extracted from source of truth,
  instead of being maintained manually. (`7908 &lt;https://github.com/pypa/pip/issues/7908&gt;`_)
- Fix pip config docstring so that the subcommands render correctly in the docs (`8072 &lt;https://github.com/pypa/pip/issues/8072&gt;`_)
- replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 &lt;https://github.com/pypa/pip/issues/8353&gt;`_)
- Fix example for defining multiple values for options which support them (`8373 &lt;https://github.com/pypa/pip/issues/8373&gt;`_)
- Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 &lt;https://github.com/pypa/pip/issues/8459&gt;`_)
- Add feature flags to docs (`8512 &lt;https://github.com/pypa/pip/issues/8512&gt;`_)
- Document how to install package extras from git branch and source distributions. (`8576 &lt;https://github.com/pypa/pip/issues/8576&gt;`_)
   ```
   
  
  
   ### 20.3b1
   ```
   ===================

Deprecations and Removals
-------------------------

- ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version.
  To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Deprecate support for Python 3.5 (`8181 &lt;https://github.com/pypa/pip/issues/8181&gt;`_)
- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)
- Deprecate ``--find-links`` option in ``pip freeze`` (`9069 &lt;https://github.com/pypa/pip/issues/9069&gt;`_)

Features
--------

- Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Allow multiple values for --abi and --platform. (`6121 &lt;https://github.com/pypa/pip/issues/6121&gt;`_)
- Add option ``--format`` to subcommand ``list`` of ``pip  cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 &lt;https://github.com/pypa/pip/issues/8355&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Make the ``setup.py install`` deprecation warning less noisy. We warn only
  when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as
  situations where both fails are most probably irrelevant to this deprecation. (`8752 &lt;https://github.com/pypa/pip/issues/8752&gt;`_)
- Check the download directory for existing wheels to possibly avoid
  fetching metadata when the ``fast-deps`` feature is used with
  ``pip wheel`` and ``pip download``. (`8804 &lt;https://github.com/pypa/pip/issues/8804&gt;`_)
- When installing a git URL that refers to a commit that is not available locally
  after git clone, attempt to fetch it from the remote. (`8815 &lt;https://github.com/pypa/pip/issues/8815&gt;`_)
- Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 &lt;https://github.com/pypa/pip/issues/8892&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)
- Support Python 3.9. (`8971 &lt;https://github.com/pypa/pip/issues/8971&gt;`_)
- Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 &lt;https://github.com/pypa/pip/issues/8975&gt;`_)
- Switch to the new dependency resolver by default. (`9019 &lt;https://github.com/pypa/pip/issues/9019&gt;`_)
- Remove the ``--build-dir`` option, as per the deprecation. (`9049 &lt;https://github.com/pypa/pip/issues/9049&gt;`_)

Bug Fixes
---------

- Propagate ``--extra-index-url`` from requirements file properly to session auth,
  so that keyring auth will work as expected. (`8103 &lt;https://github.com/pypa/pip/issues/8103&gt;`_)
- Allow specifying verbosity and quiet level via configuration files
  and environment variables. Previously these options were treated as
  boolean values when read from there while through CLI the level can be
  specified. (`8578 &lt;https://github.com/pypa/pip/issues/8578&gt;`_)
- Only converts Windows path to unicode on Python 2 to avoid regressions when a
  POSIX environment does not configure the file system encoding correctly. (`8658 &lt;https://github.com/pypa/pip/issues/8658&gt;`_)
- List downloaded distributions before exiting ``pip download``
  when using the new resolver to make the behavior the same as
  that on the legacy resolver. (`8696 &lt;https://github.com/pypa/pip/issues/8696&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- Avoid polluting the destination directory by resolution artifacts
  when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 &lt;https://github.com/pypa/pip/issues/8827&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)
- Tweak the output during dependency resolution in the new resolver. (`8861 &lt;https://github.com/pypa/pip/issues/8861&gt;`_)
- Correctly search for installed distributions in new resolver logic in order
  to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 &lt;https://github.com/pypa/pip/issues/8963&gt;`_)
- Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file
  with editable=True. Render it as a non-editable ``file://`` URL until modern
  editable installs are standardized and supported. (`8996 &lt;https://github.com/pypa/pip/issues/8996&gt;`_)

Vendored Libraries
------------------

- Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed.
  It is mandatory for ``pip debug`` command.

Improved Documentation
----------------------

- Add documentation for &#39;.netrc&#39; support. (`7231 &lt;https://github.com/pypa/pip/issues/7231&gt;`_)
- Add OS tabs for OS-specific commands. (`7311 &lt;https://github.com/pypa/pip/issues/7311&gt;`_)
- Add note and example on keyring support for index basic-auth (`8636 &lt;https://github.com/pypa/pip/issues/8636&gt;`_)
- Added initial UX feedback widgets to docs. (`8783 &lt;https://github.com/pypa/pip/issues/8783&gt;`_, `8848 &lt;https://github.com/pypa/pip/issues/8848&gt;`_)
- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
- Update user docs to reflect new resolver as default in 20.3. (`9044 &lt;https://github.com/pypa/pip/issues/9044&gt;`_)
- Improve migration guide to reflect changes in new resolver behavior. (`9056 &lt;https://github.com/pypa/pip/issues/9056&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
bors bot added a commit to duckinator/emanate that referenced this issue Dec 1, 2020
195: Update pip to 20.3 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.3
   ```
   - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 &lt;https://github.com/pypa/pip/issues/8546&gt;`_, `8377 &lt;https://github.com/pypa/pip/issues/8377&gt;`_)
- Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 &lt;https://github.com/pypa/pip/issues/6741&gt;`_)
- Warn if index pages have unexpected content-type (`6754 &lt;https://github.com/pypa/pip/issues/6754&gt;`_)
- Allow specifying ``--prefer-binary`` option in a requirements file (`7693 &lt;https://github.com/pypa/pip/issues/7693&gt;`_)
- Generate PEP 376 REQUESTED metadata for user supplied requirements installed
  by pip. (`7811 &lt;https://github.com/pypa/pip/issues/7811&gt;`_)
- Warn if package url is a vcs or an archive url with invalid scheme (`8128 &lt;https://github.com/pypa/pip/issues/8128&gt;`_)
- Parallelize network operations in ``pip list``. (`8504 &lt;https://github.com/pypa/pip/issues/8504&gt;`_)
- Allow the new resolver to obtain dependency information through wheels
  lazily downloaded using HTTP range requests.  To enable this feature,
  invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 &lt;https://github.com/pypa/pip/issues/8588&gt;`_)
- Support ``--use-feature`` in requirements files (`8601 &lt;https://github.com/pypa/pip/issues/8601&gt;`_)

Bug Fixes
---------

- Use canonical package names while looking up already installed packages. (`5021 &lt;https://github.com/pypa/pip/issues/5021&gt;`_)
- Fix normalizing path on Windows when installing package on another logical disk. (`7625 &lt;https://github.com/pypa/pip/issues/7625&gt;`_)
- The VCS commands run by pip as subprocesses don&#39;t merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 &lt;https://github.com/pypa/pip/issues/7968&gt;`_)
- Correctly treat non-ASCII entry point declarations in wheels so they can be
  installed on Windows. (`8342 &lt;https://github.com/pypa/pip/issues/8342&gt;`_)
- Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 &lt;https://github.com/pypa/pip/issues/8454&gt;`_)
- Headers provided by wheels in .data directories are now correctly installed
  into the user-provided locations, such as ``--prefix``, instead of the virtual
  environment pip is running in. (`8521 &lt;https://github.com/pypa/pip/issues/8521&gt;`_)

Vendored Libraries
------------------

- Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3.
- Upgrade appdirs to 1.4.4
- Upgrade certifi to 2020.6.20
- Upgrade distlib to 0.3.1
- Upgrade html5lib to 1.1
- Upgrade idna to 2.10
- Upgrade packaging to 20.4
- Upgrade requests to 2.24.0
- Upgrade six to 1.15.0
- Upgrade toml to 0.10.1
- Upgrade urllib3 to 1.25.9

Improved Documentation
----------------------

- Add ``--no-input`` option to pip docs (`7688 &lt;https://github.com/pypa/pip/issues/7688&gt;`_)
- List of options supported in requirements file are extracted from source of truth,
  instead of being maintained manually. (`7908 &lt;https://github.com/pypa/pip/issues/7908&gt;`_)
- Fix pip config docstring so that the subcommands render correctly in the docs (`8072 &lt;https://github.com/pypa/pip/issues/8072&gt;`_)
- replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 &lt;https://github.com/pypa/pip/issues/8353&gt;`_)
- Fix example for defining multiple values for options which support them (`8373 &lt;https://github.com/pypa/pip/issues/8373&gt;`_)
- Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 &lt;https://github.com/pypa/pip/issues/8459&gt;`_)
- Add feature flags to docs (`8512 &lt;https://github.com/pypa/pip/issues/8512&gt;`_)
- Document how to install package extras from git branch and source distributions. (`8576 &lt;https://github.com/pypa/pip/issues/8576&gt;`_)
   ```
   
  
  
   ### 20.3b1
   ```
   ===================

Deprecations and Removals
-------------------------

- ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version.
  To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Deprecate support for Python 3.5 (`8181 &lt;https://github.com/pypa/pip/issues/8181&gt;`_)
- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)
- Deprecate ``--find-links`` option in ``pip freeze`` (`9069 &lt;https://github.com/pypa/pip/issues/9069&gt;`_)

Features
--------

- Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Allow multiple values for --abi and --platform. (`6121 &lt;https://github.com/pypa/pip/issues/6121&gt;`_)
- Add option ``--format`` to subcommand ``list`` of ``pip  cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 &lt;https://github.com/pypa/pip/issues/8355&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Make the ``setup.py install`` deprecation warning less noisy. We warn only
  when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as
  situations where both fails are most probably irrelevant to this deprecation. (`8752 &lt;https://github.com/pypa/pip/issues/8752&gt;`_)
- Check the download directory for existing wheels to possibly avoid
  fetching metadata when the ``fast-deps`` feature is used with
  ``pip wheel`` and ``pip download``. (`8804 &lt;https://github.com/pypa/pip/issues/8804&gt;`_)
- When installing a git URL that refers to a commit that is not available locally
  after git clone, attempt to fetch it from the remote. (`8815 &lt;https://github.com/pypa/pip/issues/8815&gt;`_)
- Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 &lt;https://github.com/pypa/pip/issues/8892&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)
- Support Python 3.9. (`8971 &lt;https://github.com/pypa/pip/issues/8971&gt;`_)
- Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 &lt;https://github.com/pypa/pip/issues/8975&gt;`_)
- Switch to the new dependency resolver by default. (`9019 &lt;https://github.com/pypa/pip/issues/9019&gt;`_)
- Remove the ``--build-dir`` option, as per the deprecation. (`9049 &lt;https://github.com/pypa/pip/issues/9049&gt;`_)

Bug Fixes
---------

- Propagate ``--extra-index-url`` from requirements file properly to session auth,
  so that keyring auth will work as expected. (`8103 &lt;https://github.com/pypa/pip/issues/8103&gt;`_)
- Allow specifying verbosity and quiet level via configuration files
  and environment variables. Previously these options were treated as
  boolean values when read from there while through CLI the level can be
  specified. (`8578 &lt;https://github.com/pypa/pip/issues/8578&gt;`_)
- Only converts Windows path to unicode on Python 2 to avoid regressions when a
  POSIX environment does not configure the file system encoding correctly. (`8658 &lt;https://github.com/pypa/pip/issues/8658&gt;`_)
- List downloaded distributions before exiting ``pip download``
  when using the new resolver to make the behavior the same as
  that on the legacy resolver. (`8696 &lt;https://github.com/pypa/pip/issues/8696&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- Avoid polluting the destination directory by resolution artifacts
  when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 &lt;https://github.com/pypa/pip/issues/8827&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)
- Tweak the output during dependency resolution in the new resolver. (`8861 &lt;https://github.com/pypa/pip/issues/8861&gt;`_)
- Correctly search for installed distributions in new resolver logic in order
  to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 &lt;https://github.com/pypa/pip/issues/8963&gt;`_)
- Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file
  with editable=True. Render it as a non-editable ``file://`` URL until modern
  editable installs are standardized and supported. (`8996 &lt;https://github.com/pypa/pip/issues/8996&gt;`_)

Vendored Libraries
------------------

- Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed.
  It is mandatory for ``pip debug`` command.

Improved Documentation
----------------------

- Add documentation for &#39;.netrc&#39; support. (`7231 &lt;https://github.com/pypa/pip/issues/7231&gt;`_)
- Add OS tabs for OS-specific commands. (`7311 &lt;https://github.com/pypa/pip/issues/7311&gt;`_)
- Add note and example on keyring support for index basic-auth (`8636 &lt;https://github.com/pypa/pip/issues/8636&gt;`_)
- Added initial UX feedback widgets to docs. (`8783 &lt;https://github.com/pypa/pip/issues/8783&gt;`_, `8848 &lt;https://github.com/pypa/pip/issues/8848&gt;`_)
- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
- Update user docs to reflect new resolver as default in 20.3. (`9044 &lt;https://github.com/pypa/pip/issues/9044&gt;`_)
- Improve migration guide to reflect changes in new resolver behavior. (`9056 &lt;https://github.com/pypa/pip/issues/9056&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
bors bot added a commit to duckinator/emanate that referenced this issue Dec 1, 2020
195: Update pip to 20.3 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.3
   ```
   - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 &lt;https://github.com/pypa/pip/issues/8546&gt;`_, `8377 &lt;https://github.com/pypa/pip/issues/8377&gt;`_)
- Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 &lt;https://github.com/pypa/pip/issues/6741&gt;`_)
- Warn if index pages have unexpected content-type (`6754 &lt;https://github.com/pypa/pip/issues/6754&gt;`_)
- Allow specifying ``--prefer-binary`` option in a requirements file (`7693 &lt;https://github.com/pypa/pip/issues/7693&gt;`_)
- Generate PEP 376 REQUESTED metadata for user supplied requirements installed
  by pip. (`7811 &lt;https://github.com/pypa/pip/issues/7811&gt;`_)
- Warn if package url is a vcs or an archive url with invalid scheme (`8128 &lt;https://github.com/pypa/pip/issues/8128&gt;`_)
- Parallelize network operations in ``pip list``. (`8504 &lt;https://github.com/pypa/pip/issues/8504&gt;`_)
- Allow the new resolver to obtain dependency information through wheels
  lazily downloaded using HTTP range requests.  To enable this feature,
  invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 &lt;https://github.com/pypa/pip/issues/8588&gt;`_)
- Support ``--use-feature`` in requirements files (`8601 &lt;https://github.com/pypa/pip/issues/8601&gt;`_)

Bug Fixes
---------

- Use canonical package names while looking up already installed packages. (`5021 &lt;https://github.com/pypa/pip/issues/5021&gt;`_)
- Fix normalizing path on Windows when installing package on another logical disk. (`7625 &lt;https://github.com/pypa/pip/issues/7625&gt;`_)
- The VCS commands run by pip as subprocesses don&#39;t merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 &lt;https://github.com/pypa/pip/issues/7968&gt;`_)
- Correctly treat non-ASCII entry point declarations in wheels so they can be
  installed on Windows. (`8342 &lt;https://github.com/pypa/pip/issues/8342&gt;`_)
- Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 &lt;https://github.com/pypa/pip/issues/8454&gt;`_)
- Headers provided by wheels in .data directories are now correctly installed
  into the user-provided locations, such as ``--prefix``, instead of the virtual
  environment pip is running in. (`8521 &lt;https://github.com/pypa/pip/issues/8521&gt;`_)

Vendored Libraries
------------------

- Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3.
- Upgrade appdirs to 1.4.4
- Upgrade certifi to 2020.6.20
- Upgrade distlib to 0.3.1
- Upgrade html5lib to 1.1
- Upgrade idna to 2.10
- Upgrade packaging to 20.4
- Upgrade requests to 2.24.0
- Upgrade six to 1.15.0
- Upgrade toml to 0.10.1
- Upgrade urllib3 to 1.25.9

Improved Documentation
----------------------

- Add ``--no-input`` option to pip docs (`7688 &lt;https://github.com/pypa/pip/issues/7688&gt;`_)
- List of options supported in requirements file are extracted from source of truth,
  instead of being maintained manually. (`7908 &lt;https://github.com/pypa/pip/issues/7908&gt;`_)
- Fix pip config docstring so that the subcommands render correctly in the docs (`8072 &lt;https://github.com/pypa/pip/issues/8072&gt;`_)
- replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 &lt;https://github.com/pypa/pip/issues/8353&gt;`_)
- Fix example for defining multiple values for options which support them (`8373 &lt;https://github.com/pypa/pip/issues/8373&gt;`_)
- Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 &lt;https://github.com/pypa/pip/issues/8459&gt;`_)
- Add feature flags to docs (`8512 &lt;https://github.com/pypa/pip/issues/8512&gt;`_)
- Document how to install package extras from git branch and source distributions. (`8576 &lt;https://github.com/pypa/pip/issues/8576&gt;`_)
   ```
   
  
  
   ### 20.3b1
   ```
   ===================

Deprecations and Removals
-------------------------

- ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version.
  To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Deprecate support for Python 3.5 (`8181 &lt;https://github.com/pypa/pip/issues/8181&gt;`_)
- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)
- Deprecate ``--find-links`` option in ``pip freeze`` (`9069 &lt;https://github.com/pypa/pip/issues/9069&gt;`_)

Features
--------

- Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Allow multiple values for --abi and --platform. (`6121 &lt;https://github.com/pypa/pip/issues/6121&gt;`_)
- Add option ``--format`` to subcommand ``list`` of ``pip  cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 &lt;https://github.com/pypa/pip/issues/8355&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Make the ``setup.py install`` deprecation warning less noisy. We warn only
  when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as
  situations where both fails are most probably irrelevant to this deprecation. (`8752 &lt;https://github.com/pypa/pip/issues/8752&gt;`_)
- Check the download directory for existing wheels to possibly avoid
  fetching metadata when the ``fast-deps`` feature is used with
  ``pip wheel`` and ``pip download``. (`8804 &lt;https://github.com/pypa/pip/issues/8804&gt;`_)
- When installing a git URL that refers to a commit that is not available locally
  after git clone, attempt to fetch it from the remote. (`8815 &lt;https://github.com/pypa/pip/issues/8815&gt;`_)
- Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 &lt;https://github.com/pypa/pip/issues/8892&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)
- Support Python 3.9. (`8971 &lt;https://github.com/pypa/pip/issues/8971&gt;`_)
- Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 &lt;https://github.com/pypa/pip/issues/8975&gt;`_)
- Switch to the new dependency resolver by default. (`9019 &lt;https://github.com/pypa/pip/issues/9019&gt;`_)
- Remove the ``--build-dir`` option, as per the deprecation. (`9049 &lt;https://github.com/pypa/pip/issues/9049&gt;`_)

Bug Fixes
---------

- Propagate ``--extra-index-url`` from requirements file properly to session auth,
  so that keyring auth will work as expected. (`8103 &lt;https://github.com/pypa/pip/issues/8103&gt;`_)
- Allow specifying verbosity and quiet level via configuration files
  and environment variables. Previously these options were treated as
  boolean values when read from there while through CLI the level can be
  specified. (`8578 &lt;https://github.com/pypa/pip/issues/8578&gt;`_)
- Only converts Windows path to unicode on Python 2 to avoid regressions when a
  POSIX environment does not configure the file system encoding correctly. (`8658 &lt;https://github.com/pypa/pip/issues/8658&gt;`_)
- List downloaded distributions before exiting ``pip download``
  when using the new resolver to make the behavior the same as
  that on the legacy resolver. (`8696 &lt;https://github.com/pypa/pip/issues/8696&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- Avoid polluting the destination directory by resolution artifacts
  when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 &lt;https://github.com/pypa/pip/issues/8827&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)
- Tweak the output during dependency resolution in the new resolver. (`8861 &lt;https://github.com/pypa/pip/issues/8861&gt;`_)
- Correctly search for installed distributions in new resolver logic in order
  to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 &lt;https://github.com/pypa/pip/issues/8963&gt;`_)
- Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file
  with editable=True. Render it as a non-editable ``file://`` URL until modern
  editable installs are standardized and supported. (`8996 &lt;https://github.com/pypa/pip/issues/8996&gt;`_)

Vendored Libraries
------------------

- Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed.
  It is mandatory for ``pip debug`` command.

Improved Documentation
----------------------

- Add documentation for &#39;.netrc&#39; support. (`7231 &lt;https://github.com/pypa/pip/issues/7231&gt;`_)
- Add OS tabs for OS-specific commands. (`7311 &lt;https://github.com/pypa/pip/issues/7311&gt;`_)
- Add note and example on keyring support for index basic-auth (`8636 &lt;https://github.com/pypa/pip/issues/8636&gt;`_)
- Added initial UX feedback widgets to docs. (`8783 &lt;https://github.com/pypa/pip/issues/8783&gt;`_, `8848 &lt;https://github.com/pypa/pip/issues/8848&gt;`_)
- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
- Update user docs to reflect new resolver as default in 20.3. (`9044 &lt;https://github.com/pypa/pip/issues/9044&gt;`_)
- Improve migration guide to reflect changes in new resolver behavior. (`9056 &lt;https://github.com/pypa/pip/issues/9056&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
bors bot added a commit to duckinator/emanate that referenced this issue Dec 4, 2020
194: Update pytest-pylint to 0.18.0 r=duckinator a=pyup-bot


This PR updates [pytest-pylint](https://pypi.org/project/pytest-pylint) from **0.17.0** to **0.18.0**.



*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest-pylint
  - Changelog: https://pyup.io/changelogs/pytest-pylint/
  - Repo: https://github.com/carsongee/pytest-pylint
</details>



197: Update pip to 20.3.1 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.3.1
   ```
   ===================

Deprecations and Removals
-------------------------

- The --build-dir option has been restored as a no-op, to soften the transition
  for tools that still used it. (`9193 &lt;https://github.com/pypa/pip/issues/9193&gt;`_)
   ```
   
  
  
   ### 20.3
   ```
   - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 &lt;https://github.com/pypa/pip/issues/8546&gt;`_, `8377 &lt;https://github.com/pypa/pip/issues/8377&gt;`_)
- Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 &lt;https://github.com/pypa/pip/issues/6741&gt;`_)
- Warn if index pages have unexpected content-type (`6754 &lt;https://github.com/pypa/pip/issues/6754&gt;`_)
- Allow specifying ``--prefer-binary`` option in a requirements file (`7693 &lt;https://github.com/pypa/pip/issues/7693&gt;`_)
- Generate PEP 376 REQUESTED metadata for user supplied requirements installed
  by pip. (`7811 &lt;https://github.com/pypa/pip/issues/7811&gt;`_)
- Warn if package url is a vcs or an archive url with invalid scheme (`8128 &lt;https://github.com/pypa/pip/issues/8128&gt;`_)
- Parallelize network operations in ``pip list``. (`8504 &lt;https://github.com/pypa/pip/issues/8504&gt;`_)
- Allow the new resolver to obtain dependency information through wheels
  lazily downloaded using HTTP range requests.  To enable this feature,
  invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 &lt;https://github.com/pypa/pip/issues/8588&gt;`_)
- Support ``--use-feature`` in requirements files (`8601 &lt;https://github.com/pypa/pip/issues/8601&gt;`_)

Bug Fixes
---------

- Use canonical package names while looking up already installed packages. (`5021 &lt;https://github.com/pypa/pip/issues/5021&gt;`_)
- Fix normalizing path on Windows when installing package on another logical disk. (`7625 &lt;https://github.com/pypa/pip/issues/7625&gt;`_)
- The VCS commands run by pip as subprocesses don&#39;t merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 &lt;https://github.com/pypa/pip/issues/7968&gt;`_)
- Correctly treat non-ASCII entry point declarations in wheels so they can be
  installed on Windows. (`8342 &lt;https://github.com/pypa/pip/issues/8342&gt;`_)
- Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 &lt;https://github.com/pypa/pip/issues/8454&gt;`_)
- Headers provided by wheels in .data directories are now correctly installed
  into the user-provided locations, such as ``--prefix``, instead of the virtual
  environment pip is running in. (`8521 &lt;https://github.com/pypa/pip/issues/8521&gt;`_)

Vendored Libraries
------------------

- Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3.
- Upgrade appdirs to 1.4.4
- Upgrade certifi to 2020.6.20
- Upgrade distlib to 0.3.1
- Upgrade html5lib to 1.1
- Upgrade idna to 2.10
- Upgrade packaging to 20.4
- Upgrade requests to 2.24.0
- Upgrade six to 1.15.0
- Upgrade toml to 0.10.1
- Upgrade urllib3 to 1.25.9

Improved Documentation
----------------------

- Add ``--no-input`` option to pip docs (`7688 &lt;https://github.com/pypa/pip/issues/7688&gt;`_)
- List of options supported in requirements file are extracted from source of truth,
  instead of being maintained manually. (`7908 &lt;https://github.com/pypa/pip/issues/7908&gt;`_)
- Fix pip config docstring so that the subcommands render correctly in the docs (`8072 &lt;https://github.com/pypa/pip/issues/8072&gt;`_)
- replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 &lt;https://github.com/pypa/pip/issues/8353&gt;`_)
- Fix example for defining multiple values for options which support them (`8373 &lt;https://github.com/pypa/pip/issues/8373&gt;`_)
- Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 &lt;https://github.com/pypa/pip/issues/8459&gt;`_)
- Add feature flags to docs (`8512 &lt;https://github.com/pypa/pip/issues/8512&gt;`_)
- Document how to install package extras from git branch and source distributions. (`8576 &lt;https://github.com/pypa/pip/issues/8576&gt;`_)
   ```
   
  
  
   ### 20.3b1
   ```
   ===================

Deprecations and Removals
-------------------------

- ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version.
  To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Deprecate support for Python 3.5 (`8181 &lt;https://github.com/pypa/pip/issues/8181&gt;`_)
- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)
- Deprecate ``--find-links`` option in ``pip freeze`` (`9069 &lt;https://github.com/pypa/pip/issues/9069&gt;`_)

Features
--------

- Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Allow multiple values for --abi and --platform. (`6121 &lt;https://github.com/pypa/pip/issues/6121&gt;`_)
- Add option ``--format`` to subcommand ``list`` of ``pip  cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 &lt;https://github.com/pypa/pip/issues/8355&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Make the ``setup.py install`` deprecation warning less noisy. We warn only
  when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as
  situations where both fails are most probably irrelevant to this deprecation. (`8752 &lt;https://github.com/pypa/pip/issues/8752&gt;`_)
- Check the download directory for existing wheels to possibly avoid
  fetching metadata when the ``fast-deps`` feature is used with
  ``pip wheel`` and ``pip download``. (`8804 &lt;https://github.com/pypa/pip/issues/8804&gt;`_)
- When installing a git URL that refers to a commit that is not available locally
  after git clone, attempt to fetch it from the remote. (`8815 &lt;https://github.com/pypa/pip/issues/8815&gt;`_)
- Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 &lt;https://github.com/pypa/pip/issues/8892&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)
- Support Python 3.9. (`8971 &lt;https://github.com/pypa/pip/issues/8971&gt;`_)
- Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 &lt;https://github.com/pypa/pip/issues/8975&gt;`_)
- Switch to the new dependency resolver by default. (`9019 &lt;https://github.com/pypa/pip/issues/9019&gt;`_)
- Remove the ``--build-dir`` option, as per the deprecation. (`9049 &lt;https://github.com/pypa/pip/issues/9049&gt;`_)

Bug Fixes
---------

- Propagate ``--extra-index-url`` from requirements file properly to session auth,
  so that keyring auth will work as expected. (`8103 &lt;https://github.com/pypa/pip/issues/8103&gt;`_)
- Allow specifying verbosity and quiet level via configuration files
  and environment variables. Previously these options were treated as
  boolean values when read from there while through CLI the level can be
  specified. (`8578 &lt;https://github.com/pypa/pip/issues/8578&gt;`_)
- Only converts Windows path to unicode on Python 2 to avoid regressions when a
  POSIX environment does not configure the file system encoding correctly. (`8658 &lt;https://github.com/pypa/pip/issues/8658&gt;`_)
- List downloaded distributions before exiting ``pip download``
  when using the new resolver to make the behavior the same as
  that on the legacy resolver. (`8696 &lt;https://github.com/pypa/pip/issues/8696&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- Avoid polluting the destination directory by resolution artifacts
  when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 &lt;https://github.com/pypa/pip/issues/8827&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)
- Tweak the output during dependency resolution in the new resolver. (`8861 &lt;https://github.com/pypa/pip/issues/8861&gt;`_)
- Correctly search for installed distributions in new resolver logic in order
  to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 &lt;https://github.com/pypa/pip/issues/8963&gt;`_)
- Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file
  with editable=True. Render it as a non-editable ``file://`` URL until modern
  editable installs are standardized and supported. (`8996 &lt;https://github.com/pypa/pip/issues/8996&gt;`_)

Vendored Libraries
------------------

- Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed.
  It is mandatory for ``pip debug`` command.

Improved Documentation
----------------------

- Add documentation for &#39;.netrc&#39; support. (`7231 &lt;https://github.com/pypa/pip/issues/7231&gt;`_)
- Add OS tabs for OS-specific commands. (`7311 &lt;https://github.com/pypa/pip/issues/7311&gt;`_)
- Add note and example on keyring support for index basic-auth (`8636 &lt;https://github.com/pypa/pip/issues/8636&gt;`_)
- Added initial UX feedback widgets to docs. (`8783 &lt;https://github.com/pypa/pip/issues/8783&gt;`_, `8848 &lt;https://github.com/pypa/pip/issues/8848&gt;`_)
- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
- Update user docs to reflect new resolver as default in 20.3. (`9044 &lt;https://github.com/pypa/pip/issues/9044&gt;`_)
- Improve migration guide to reflect changes in new resolver behavior. (`9056 &lt;https://github.com/pypa/pip/issues/9056&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
Co-authored-by: Ellen Marie Dash <me@duckie.co>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 9, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
type: bug A confirmed bug or unintended behavior
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

5 participants