Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Resolver: Be less eager about hitting the network #8905

Closed
pradyunsg opened this issue Sep 23, 2020 · 15 comments
Closed

New Resolver: Be less eager about hitting the network #8905

pradyunsg opened this issue Sep 23, 2020 · 15 comments
Labels
!release blocker Hold a release until this is resolved

Comments

@pradyunsg
Copy link
Member

pradyunsg commented Sep 23, 2020

pip's new resolver is hitting the network even when the currently installed version does satisfy the version requested. Further, it's also includes hitting the same index page (i.e. https://pypi.org/simple/{project}) each time we see it during the graph exploration, which is obviously the wrong thing to do.

Originally posted by @pradyunsg in #8664 (comment)

@pradyunsg pradyunsg added this to Needs triage in Resolver Rollout via automation Sep 23, 2020
@pradyunsg pradyunsg added !release blocker Hold a release until this is resolved C: new resolver labels Sep 23, 2020
@uranusjr
Copy link
Member

uranusjr commented Sep 23, 2020

it's also includes hitting the same index page (i.e. https://pypi.org/simple/{project}) each time we see it during the graph exploration, which is obviously the wrong thing to do.

I thought the networking implementation already handles caching on this? I recall reviewing a PR to add lru_cache on the index page parsing function.

@pradyunsg
Copy link
Member Author

pradyunsg commented Sep 23, 2020

Output from running w/ -r /tmp/foo.txt with pinned packages being already installed.

/tmp/foo.txt is from #8664 (comment).

Even though the page parsing code has lru_cache, what's happening is we're reaching out over the network even though we've seen the page, due to the way we're handling max_age on the pages in the download cache stuff.

@pfmoore
Copy link
Member

pfmoore commented Sep 23, 2020

I'm not sure it's "just" network hits. The new resolver checks multiple times for the same requirement:

>pip install -r .\repro.txt --use-feature=2020-resolver | wc -l
370
>pip install -r .\repro.txt | wc -l
164

An example:

>pip install -r .\repro.txt | grep automat
Requirement already satisfied: automat==20.2.0 in d:\work\projects\pip\.venv\lib\site-packages (from -r .\repro.txt (line 16)) (20.2.0)
>pip install -r .\repro.txt --use-feature=2020-resolver | grep automat
Requirement already satisfied: automat==20.2.0 in d:\work\projects\pip\.venv\lib\site-packages (from -r .\repro.txt (line 16)) (20.2.0)
Requirement already satisfied: automat==20.2.0 in d:\work\projects\pip\.venv\lib\site-packages (from -r .\repro.txt (line 16)) (20.2.0)
Requirement already satisfied: automat==20.2.0 in d:\work\projects\pip\.venv\lib\site-packages (from -r .\repro.txt (line 16)) (20.2.0)

Also, putting a memory cache on _get_html_response in collector.py (which should in theory remove the bulk of repeated network request costs) made very little difference to the runtime.

@pradyunsg
Copy link
Member Author

pradyunsg commented Sep 23, 2020

The new resolver checks multiple times for the same requirement:

That's expected, since it'll reach the requirement from multiple edges of the graph.

putting a memory cache on _get_html_response in collector.py (which should in theory remove the bulk of repeated network request costs) made very little difference to the runtime.

OH, huh.

@pradyunsg
Copy link
Member Author

pradyunsg commented Sep 23, 2020

filtered-resolver-log.txt

Adding a debugging reporter to resolvelib (one that does a logger.debug("Reporter.{function_name}({arguments})", {arguments})) results in the above file.

pip install --log /tmp/log.txt -r /tmp/requirements.txt --use-feature=2020-resolver
grep -E "Getting page|Reporter" /tmp/log.txt > /tmp/filtered-resolver-log.txt

Notably, the file is empty when used without the 2020-resolver -- we're not hitting the network at all with the old resolver code. :)

@pfmoore
Copy link
Member

pfmoore commented Sep 23, 2020

Consider the following:

>pip install automat==20.2.0
Requirement already satisfied: automat==20.2.0 in d:\work\projects\pip\.venv\lib\site-packages (20.2.0)
Requirement already satisfied: attrs>=19.2.0 in d:\work\projects\pip\.venv\lib\site-packages (from automat==20.2.0) (20.2.0)
Requirement already satisfied: six in d:\work\projects\pip\.venv\lib\site-packages (from automat==20.2.0) (1.15.0)

Old resolver checks automat and its 2 dependencies.

>pip install automat==20.2.0 --use-feature=2020-resolver
Requirement already satisfied: automat==20.2.0 in d:\work\projects\pip\.venv\lib\site-packages (20.2.0)
Requirement already satisfied: six in d:\work\projects\pip\.venv\lib\site-packages (from automat==20.2.0) (1.15.0)
Requirement already satisfied: attrs>=19.2.0 in d:\work\projects\pip\.venv\lib\site-packages (from automat==20.2.0) (20.2.0)

So does new resolver.

>pip install automat==20.2.0 six==1.15.0
Requirement already satisfied: automat==20.2.0 in d:\work\projects\pip\.venv\lib\site-packages (20.2.0)
Requirement already satisfied: six==1.15.0 in d:\work\projects\pip\.venv\lib\site-packages (1.15.0)
Requirement already satisfied: attrs>=19.2.0 in d:\work\projects\pip\.venv\lib\site-packages (from automat==20.2.0) (20.2.0)

Old resolver doesn't re-check six, because it "did it earlier". (Oversimplification, but basically accurate)

>pip install automat==20.2.0 six==1.15.0 --use-feature=2020-resolver
Requirement already satisfied: automat==20.2.0 in d:\work\projects\pip\.venv\lib\site-packages (20.2.0)
Requirement already satisfied: six==1.15.0 in d:\work\projects\pip\.venv\lib\site-packages (1.15.0)
Requirement already satisfied: six==1.15.0 in d:\work\projects\pip\.venv\lib\site-packages (1.15.0)
Requirement already satisfied: attrs>=19.2.0 in d:\work\projects\pip\.venv\lib\site-packages (from automat==20.2.0) (20.2.0)

The new resolver does re-check.

Now multiply that by 161 requirements in the actual example. That's why the old resolver has 160-odd lines and the new has twice as many.

On my machine, the old resolver (160-odd checks) takes around 1-2 seconds. The new resolver with --no-deps takes 15 seconds and also does around 160 checks. That seems to be the cost of the extra complexity (plus the fact that we haven't tuned the new resolver much, yet). Without --no-deps the new resolver outputs twice as many lines, equating to twice as many checks, and takes around 30 seconds.

So I see two main components to the slowdown:

  1. Extra complexity and possible wasted time in the new resolver. That doesn't seem linear in the number of requirements. I need to do some more analysis to estimate how it grows, but it's about a 50% increase on one requirement, up to 15x on 160.
  2. Additional time because the new resolver checks everything (as it should) whereas the old resolver ignored a lot of requirements.

For (1), it's not surprising that the performance gets worse non-linearly. Dependency resolution algorithms are essentially exponential in performance, so big dependency graphs will be pretty nasty. That's fairly fundamental. The best we can likely do here is to work hard to improve the base performance so that the exponential factor doesn't hurt us quite so soon. But we're going to remain exponential.

For (2), that's fundamentally the cost of correctness. We can't avoid it because the old resolver was "getting away with" simply dropping requirements. Unfortunately (!), it was often safe to do so, so the old resolver seemed to be fast and accurate in many cases (whereas it was actually just fast and lucky!)

Notably, the file is empty when used without the 2020-resolver

OK, it looks like a significant component of (1) is the network hit. That means we probably can significantly improve that base performance by removing the network cost.

@pradyunsg
Copy link
Member Author

pradyunsg commented Sep 23, 2020

Yup to everything you've said. :)

@uranusjr
Copy link
Member

uranusjr commented Sep 24, 2020

what's happening is we're reaching out over the network even though we've seen the page, due to the way we're handling max_age on the pages in the download cache stuff.

Huh. Is there a way to configure requests so we cache responses during one pip command run, but drop the cache once the command ends? Similar to how the browser can set the cache header to “end of session” (iow each pip command run is a session, and cache clears when session ends). This is probably a good thing to do even out of the context of this performance problem—it would be very confusing if a new version becomes available right in the middle of a dependency resolution run.

@pfmoore
Copy link
Member

pfmoore commented Sep 24, 2020

That's what I was hoping that putting a LRU cache on _get_html_response would do but I messed it up somehow. +1 on having everything fetched at most once per run.

@pradyunsg
Copy link
Member Author

pradyunsg commented Sep 24, 2020

I think we prefer the already-installed version, when it satisfies a requirement except when upgrading. Looks like the entire "don't hit the network when the already installed version works" discussion was discussed in #8023.

@uranusjr
Copy link
Member

uranusjr commented Sep 24, 2020

Slapping _lru_cache on PackageFinder.find_all_candidates instead gets me

real  0m47.327s
user  0m35.412s
sys	  0m1.267s

Current performace for comparison:

real  1m25.259s
user  0m59.505s
sys	  0m1.542s

Not nearly as good as the legacy resolver (which is 2s), but probably a good start?

@pradyunsg
Copy link
Member Author

pradyunsg commented Sep 24, 2020

I think skipping the network requests would be give us a much bigger chunk of performance improvement -- but obviously both is best. :P

@pfmoore
Copy link
Member

pfmoore commented Sep 24, 2020

Agreed. But the logic of caching find_all_candidates in my view is that it asserts very clearly that the set of candidates for a project is fixed (for the duration of a pip run) and we can rely on that. So adding a cache makes a statement about semantics.

Not calling find_all_candidates if we don't need to is a different semantic assertion. One we should also do, but logically separate.

@uranusjr
Copy link
Member

uranusjr commented Sep 24, 2020

Also the not calling find_all_candidates part is much more difficult to implement (as discussed in #8023).

@pradyunsg
Copy link
Member Author

pradyunsg commented Oct 1, 2020

Gonna go ahead and close this now, given that #8912 is merged and #8023 covers basically the rest of this issue.

Resolver Rollout automation moved this from Needs triage to Closed Oct 1, 2020
bors bot added a commit to duckinator/emanate that referenced this issue Oct 18, 2020
192: Update pip to 20.2.4 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.3** to **20.2.4**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.2.4
   ```
   ===================

Deprecations and Removals
-------------------------

- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)

Features
--------

- New resolver: Avoid accessing indexes when the installed candidate is preferred
  and considered good enough. (`8023 &lt;https://github.com/pypa/pip/issues/8023&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)

Bug Fixes
---------

- New resolver: Correctly respect ``Requires-Python`` metadata to reject
  incompatible packages in ``--no-deps`` mode. (`8758 &lt;https://github.com/pypa/pip/issues/8758&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)

Improved Documentation
----------------------

- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
bors bot added a commit to duckinator/emanate that referenced this issue Dec 1, 2020
195: Update pip to 20.3 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.3
   ```
   - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 &lt;https://github.com/pypa/pip/issues/8546&gt;`_, `8377 &lt;https://github.com/pypa/pip/issues/8377&gt;`_)
- Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 &lt;https://github.com/pypa/pip/issues/6741&gt;`_)
- Warn if index pages have unexpected content-type (`6754 &lt;https://github.com/pypa/pip/issues/6754&gt;`_)
- Allow specifying ``--prefer-binary`` option in a requirements file (`7693 &lt;https://github.com/pypa/pip/issues/7693&gt;`_)
- Generate PEP 376 REQUESTED metadata for user supplied requirements installed
  by pip. (`7811 &lt;https://github.com/pypa/pip/issues/7811&gt;`_)
- Warn if package url is a vcs or an archive url with invalid scheme (`8128 &lt;https://github.com/pypa/pip/issues/8128&gt;`_)
- Parallelize network operations in ``pip list``. (`8504 &lt;https://github.com/pypa/pip/issues/8504&gt;`_)
- Allow the new resolver to obtain dependency information through wheels
  lazily downloaded using HTTP range requests.  To enable this feature,
  invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 &lt;https://github.com/pypa/pip/issues/8588&gt;`_)
- Support ``--use-feature`` in requirements files (`8601 &lt;https://github.com/pypa/pip/issues/8601&gt;`_)

Bug Fixes
---------

- Use canonical package names while looking up already installed packages. (`5021 &lt;https://github.com/pypa/pip/issues/5021&gt;`_)
- Fix normalizing path on Windows when installing package on another logical disk. (`7625 &lt;https://github.com/pypa/pip/issues/7625&gt;`_)
- The VCS commands run by pip as subprocesses don&#39;t merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 &lt;https://github.com/pypa/pip/issues/7968&gt;`_)
- Correctly treat non-ASCII entry point declarations in wheels so they can be
  installed on Windows. (`8342 &lt;https://github.com/pypa/pip/issues/8342&gt;`_)
- Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 &lt;https://github.com/pypa/pip/issues/8454&gt;`_)
- Headers provided by wheels in .data directories are now correctly installed
  into the user-provided locations, such as ``--prefix``, instead of the virtual
  environment pip is running in. (`8521 &lt;https://github.com/pypa/pip/issues/8521&gt;`_)

Vendored Libraries
------------------

- Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3.
- Upgrade appdirs to 1.4.4
- Upgrade certifi to 2020.6.20
- Upgrade distlib to 0.3.1
- Upgrade html5lib to 1.1
- Upgrade idna to 2.10
- Upgrade packaging to 20.4
- Upgrade requests to 2.24.0
- Upgrade six to 1.15.0
- Upgrade toml to 0.10.1
- Upgrade urllib3 to 1.25.9

Improved Documentation
----------------------

- Add ``--no-input`` option to pip docs (`7688 &lt;https://github.com/pypa/pip/issues/7688&gt;`_)
- List of options supported in requirements file are extracted from source of truth,
  instead of being maintained manually. (`7908 &lt;https://github.com/pypa/pip/issues/7908&gt;`_)
- Fix pip config docstring so that the subcommands render correctly in the docs (`8072 &lt;https://github.com/pypa/pip/issues/8072&gt;`_)
- replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 &lt;https://github.com/pypa/pip/issues/8353&gt;`_)
- Fix example for defining multiple values for options which support them (`8373 &lt;https://github.com/pypa/pip/issues/8373&gt;`_)
- Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 &lt;https://github.com/pypa/pip/issues/8459&gt;`_)
- Add feature flags to docs (`8512 &lt;https://github.com/pypa/pip/issues/8512&gt;`_)
- Document how to install package extras from git branch and source distributions. (`8576 &lt;https://github.com/pypa/pip/issues/8576&gt;`_)
   ```
   
  
  
   ### 20.3b1
   ```
   ===================

Deprecations and Removals
-------------------------

- ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version.
  To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Deprecate support for Python 3.5 (`8181 &lt;https://github.com/pypa/pip/issues/8181&gt;`_)
- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)
- Deprecate ``--find-links`` option in ``pip freeze`` (`9069 &lt;https://github.com/pypa/pip/issues/9069&gt;`_)

Features
--------

- Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Allow multiple values for --abi and --platform. (`6121 &lt;https://github.com/pypa/pip/issues/6121&gt;`_)
- Add option ``--format`` to subcommand ``list`` of ``pip  cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 &lt;https://github.com/pypa/pip/issues/8355&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Make the ``setup.py install`` deprecation warning less noisy. We warn only
  when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as
  situations where both fails are most probably irrelevant to this deprecation. (`8752 &lt;https://github.com/pypa/pip/issues/8752&gt;`_)
- Check the download directory for existing wheels to possibly avoid
  fetching metadata when the ``fast-deps`` feature is used with
  ``pip wheel`` and ``pip download``. (`8804 &lt;https://github.com/pypa/pip/issues/8804&gt;`_)
- When installing a git URL that refers to a commit that is not available locally
  after git clone, attempt to fetch it from the remote. (`8815 &lt;https://github.com/pypa/pip/issues/8815&gt;`_)
- Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 &lt;https://github.com/pypa/pip/issues/8892&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)
- Support Python 3.9. (`8971 &lt;https://github.com/pypa/pip/issues/8971&gt;`_)
- Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 &lt;https://github.com/pypa/pip/issues/8975&gt;`_)
- Switch to the new dependency resolver by default. (`9019 &lt;https://github.com/pypa/pip/issues/9019&gt;`_)
- Remove the ``--build-dir`` option, as per the deprecation. (`9049 &lt;https://github.com/pypa/pip/issues/9049&gt;`_)

Bug Fixes
---------

- Propagate ``--extra-index-url`` from requirements file properly to session auth,
  so that keyring auth will work as expected. (`8103 &lt;https://github.com/pypa/pip/issues/8103&gt;`_)
- Allow specifying verbosity and quiet level via configuration files
  and environment variables. Previously these options were treated as
  boolean values when read from there while through CLI the level can be
  specified. (`8578 &lt;https://github.com/pypa/pip/issues/8578&gt;`_)
- Only converts Windows path to unicode on Python 2 to avoid regressions when a
  POSIX environment does not configure the file system encoding correctly. (`8658 &lt;https://github.com/pypa/pip/issues/8658&gt;`_)
- List downloaded distributions before exiting ``pip download``
  when using the new resolver to make the behavior the same as
  that on the legacy resolver. (`8696 &lt;https://github.com/pypa/pip/issues/8696&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- Avoid polluting the destination directory by resolution artifacts
  when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 &lt;https://github.com/pypa/pip/issues/8827&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)
- Tweak the output during dependency resolution in the new resolver. (`8861 &lt;https://github.com/pypa/pip/issues/8861&gt;`_)
- Correctly search for installed distributions in new resolver logic in order
  to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 &lt;https://github.com/pypa/pip/issues/8963&gt;`_)
- Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file
  with editable=True. Render it as a non-editable ``file://`` URL until modern
  editable installs are standardized and supported. (`8996 &lt;https://github.com/pypa/pip/issues/8996&gt;`_)

Vendored Libraries
------------------

- Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed.
  It is mandatory for ``pip debug`` command.

Improved Documentation
----------------------

- Add documentation for &#39;.netrc&#39; support. (`7231 &lt;https://github.com/pypa/pip/issues/7231&gt;`_)
- Add OS tabs for OS-specific commands. (`7311 &lt;https://github.com/pypa/pip/issues/7311&gt;`_)
- Add note and example on keyring support for index basic-auth (`8636 &lt;https://github.com/pypa/pip/issues/8636&gt;`_)
- Added initial UX feedback widgets to docs. (`8783 &lt;https://github.com/pypa/pip/issues/8783&gt;`_, `8848 &lt;https://github.com/pypa/pip/issues/8848&gt;`_)
- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
- Update user docs to reflect new resolver as default in 20.3. (`9044 &lt;https://github.com/pypa/pip/issues/9044&gt;`_)
- Improve migration guide to reflect changes in new resolver behavior. (`9056 &lt;https://github.com/pypa/pip/issues/9056&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
bors bot added a commit to duckinator/emanate that referenced this issue Dec 1, 2020
195: Update pip to 20.3 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.3
   ```
   - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 &lt;https://github.com/pypa/pip/issues/8546&gt;`_, `8377 &lt;https://github.com/pypa/pip/issues/8377&gt;`_)
- Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 &lt;https://github.com/pypa/pip/issues/6741&gt;`_)
- Warn if index pages have unexpected content-type (`6754 &lt;https://github.com/pypa/pip/issues/6754&gt;`_)
- Allow specifying ``--prefer-binary`` option in a requirements file (`7693 &lt;https://github.com/pypa/pip/issues/7693&gt;`_)
- Generate PEP 376 REQUESTED metadata for user supplied requirements installed
  by pip. (`7811 &lt;https://github.com/pypa/pip/issues/7811&gt;`_)
- Warn if package url is a vcs or an archive url with invalid scheme (`8128 &lt;https://github.com/pypa/pip/issues/8128&gt;`_)
- Parallelize network operations in ``pip list``. (`8504 &lt;https://github.com/pypa/pip/issues/8504&gt;`_)
- Allow the new resolver to obtain dependency information through wheels
  lazily downloaded using HTTP range requests.  To enable this feature,
  invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 &lt;https://github.com/pypa/pip/issues/8588&gt;`_)
- Support ``--use-feature`` in requirements files (`8601 &lt;https://github.com/pypa/pip/issues/8601&gt;`_)

Bug Fixes
---------

- Use canonical package names while looking up already installed packages. (`5021 &lt;https://github.com/pypa/pip/issues/5021&gt;`_)
- Fix normalizing path on Windows when installing package on another logical disk. (`7625 &lt;https://github.com/pypa/pip/issues/7625&gt;`_)
- The VCS commands run by pip as subprocesses don&#39;t merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 &lt;https://github.com/pypa/pip/issues/7968&gt;`_)
- Correctly treat non-ASCII entry point declarations in wheels so they can be
  installed on Windows. (`8342 &lt;https://github.com/pypa/pip/issues/8342&gt;`_)
- Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 &lt;https://github.com/pypa/pip/issues/8454&gt;`_)
- Headers provided by wheels in .data directories are now correctly installed
  into the user-provided locations, such as ``--prefix``, instead of the virtual
  environment pip is running in. (`8521 &lt;https://github.com/pypa/pip/issues/8521&gt;`_)

Vendored Libraries
------------------

- Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3.
- Upgrade appdirs to 1.4.4
- Upgrade certifi to 2020.6.20
- Upgrade distlib to 0.3.1
- Upgrade html5lib to 1.1
- Upgrade idna to 2.10
- Upgrade packaging to 20.4
- Upgrade requests to 2.24.0
- Upgrade six to 1.15.0
- Upgrade toml to 0.10.1
- Upgrade urllib3 to 1.25.9

Improved Documentation
----------------------

- Add ``--no-input`` option to pip docs (`7688 &lt;https://github.com/pypa/pip/issues/7688&gt;`_)
- List of options supported in requirements file are extracted from source of truth,
  instead of being maintained manually. (`7908 &lt;https://github.com/pypa/pip/issues/7908&gt;`_)
- Fix pip config docstring so that the subcommands render correctly in the docs (`8072 &lt;https://github.com/pypa/pip/issues/8072&gt;`_)
- replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 &lt;https://github.com/pypa/pip/issues/8353&gt;`_)
- Fix example for defining multiple values for options which support them (`8373 &lt;https://github.com/pypa/pip/issues/8373&gt;`_)
- Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 &lt;https://github.com/pypa/pip/issues/8459&gt;`_)
- Add feature flags to docs (`8512 &lt;https://github.com/pypa/pip/issues/8512&gt;`_)
- Document how to install package extras from git branch and source distributions. (`8576 &lt;https://github.com/pypa/pip/issues/8576&gt;`_)
   ```
   
  
  
   ### 20.3b1
   ```
   ===================

Deprecations and Removals
-------------------------

- ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version.
  To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Deprecate support for Python 3.5 (`8181 &lt;https://github.com/pypa/pip/issues/8181&gt;`_)
- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)
- Deprecate ``--find-links`` option in ``pip freeze`` (`9069 &lt;https://github.com/pypa/pip/issues/9069&gt;`_)

Features
--------

- Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Allow multiple values for --abi and --platform. (`6121 &lt;https://github.com/pypa/pip/issues/6121&gt;`_)
- Add option ``--format`` to subcommand ``list`` of ``pip  cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 &lt;https://github.com/pypa/pip/issues/8355&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Make the ``setup.py install`` deprecation warning less noisy. We warn only
  when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as
  situations where both fails are most probably irrelevant to this deprecation. (`8752 &lt;https://github.com/pypa/pip/issues/8752&gt;`_)
- Check the download directory for existing wheels to possibly avoid
  fetching metadata when the ``fast-deps`` feature is used with
  ``pip wheel`` and ``pip download``. (`8804 &lt;https://github.com/pypa/pip/issues/8804&gt;`_)
- When installing a git URL that refers to a commit that is not available locally
  after git clone, attempt to fetch it from the remote. (`8815 &lt;https://github.com/pypa/pip/issues/8815&gt;`_)
- Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 &lt;https://github.com/pypa/pip/issues/8892&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)
- Support Python 3.9. (`8971 &lt;https://github.com/pypa/pip/issues/8971&gt;`_)
- Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 &lt;https://github.com/pypa/pip/issues/8975&gt;`_)
- Switch to the new dependency resolver by default. (`9019 &lt;https://github.com/pypa/pip/issues/9019&gt;`_)
- Remove the ``--build-dir`` option, as per the deprecation. (`9049 &lt;https://github.com/pypa/pip/issues/9049&gt;`_)

Bug Fixes
---------

- Propagate ``--extra-index-url`` from requirements file properly to session auth,
  so that keyring auth will work as expected. (`8103 &lt;https://github.com/pypa/pip/issues/8103&gt;`_)
- Allow specifying verbosity and quiet level via configuration files
  and environment variables. Previously these options were treated as
  boolean values when read from there while through CLI the level can be
  specified. (`8578 &lt;https://github.com/pypa/pip/issues/8578&gt;`_)
- Only converts Windows path to unicode on Python 2 to avoid regressions when a
  POSIX environment does not configure the file system encoding correctly. (`8658 &lt;https://github.com/pypa/pip/issues/8658&gt;`_)
- List downloaded distributions before exiting ``pip download``
  when using the new resolver to make the behavior the same as
  that on the legacy resolver. (`8696 &lt;https://github.com/pypa/pip/issues/8696&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- Avoid polluting the destination directory by resolution artifacts
  when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 &lt;https://github.com/pypa/pip/issues/8827&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)
- Tweak the output during dependency resolution in the new resolver. (`8861 &lt;https://github.com/pypa/pip/issues/8861&gt;`_)
- Correctly search for installed distributions in new resolver logic in order
  to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 &lt;https://github.com/pypa/pip/issues/8963&gt;`_)
- Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file
  with editable=True. Render it as a non-editable ``file://`` URL until modern
  editable installs are standardized and supported. (`8996 &lt;https://github.com/pypa/pip/issues/8996&gt;`_)

Vendored Libraries
------------------

- Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed.
  It is mandatory for ``pip debug`` command.

Improved Documentation
----------------------

- Add documentation for &#39;.netrc&#39; support. (`7231 &lt;https://github.com/pypa/pip/issues/7231&gt;`_)
- Add OS tabs for OS-specific commands. (`7311 &lt;https://github.com/pypa/pip/issues/7311&gt;`_)
- Add note and example on keyring support for index basic-auth (`8636 &lt;https://github.com/pypa/pip/issues/8636&gt;`_)
- Added initial UX feedback widgets to docs. (`8783 &lt;https://github.com/pypa/pip/issues/8783&gt;`_, `8848 &lt;https://github.com/pypa/pip/issues/8848&gt;`_)
- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
- Update user docs to reflect new resolver as default in 20.3. (`9044 &lt;https://github.com/pypa/pip/issues/9044&gt;`_)
- Improve migration guide to reflect changes in new resolver behavior. (`9056 &lt;https://github.com/pypa/pip/issues/9056&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
bors bot added a commit to duckinator/emanate that referenced this issue Dec 1, 2020
195: Update pip to 20.3 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.3
   ```
   - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 &lt;https://github.com/pypa/pip/issues/8546&gt;`_, `8377 &lt;https://github.com/pypa/pip/issues/8377&gt;`_)
- Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 &lt;https://github.com/pypa/pip/issues/6741&gt;`_)
- Warn if index pages have unexpected content-type (`6754 &lt;https://github.com/pypa/pip/issues/6754&gt;`_)
- Allow specifying ``--prefer-binary`` option in a requirements file (`7693 &lt;https://github.com/pypa/pip/issues/7693&gt;`_)
- Generate PEP 376 REQUESTED metadata for user supplied requirements installed
  by pip. (`7811 &lt;https://github.com/pypa/pip/issues/7811&gt;`_)
- Warn if package url is a vcs or an archive url with invalid scheme (`8128 &lt;https://github.com/pypa/pip/issues/8128&gt;`_)
- Parallelize network operations in ``pip list``. (`8504 &lt;https://github.com/pypa/pip/issues/8504&gt;`_)
- Allow the new resolver to obtain dependency information through wheels
  lazily downloaded using HTTP range requests.  To enable this feature,
  invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 &lt;https://github.com/pypa/pip/issues/8588&gt;`_)
- Support ``--use-feature`` in requirements files (`8601 &lt;https://github.com/pypa/pip/issues/8601&gt;`_)

Bug Fixes
---------

- Use canonical package names while looking up already installed packages. (`5021 &lt;https://github.com/pypa/pip/issues/5021&gt;`_)
- Fix normalizing path on Windows when installing package on another logical disk. (`7625 &lt;https://github.com/pypa/pip/issues/7625&gt;`_)
- The VCS commands run by pip as subprocesses don&#39;t merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 &lt;https://github.com/pypa/pip/issues/7968&gt;`_)
- Correctly treat non-ASCII entry point declarations in wheels so they can be
  installed on Windows. (`8342 &lt;https://github.com/pypa/pip/issues/8342&gt;`_)
- Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 &lt;https://github.com/pypa/pip/issues/8454&gt;`_)
- Headers provided by wheels in .data directories are now correctly installed
  into the user-provided locations, such as ``--prefix``, instead of the virtual
  environment pip is running in. (`8521 &lt;https://github.com/pypa/pip/issues/8521&gt;`_)

Vendored Libraries
------------------

- Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3.
- Upgrade appdirs to 1.4.4
- Upgrade certifi to 2020.6.20
- Upgrade distlib to 0.3.1
- Upgrade html5lib to 1.1
- Upgrade idna to 2.10
- Upgrade packaging to 20.4
- Upgrade requests to 2.24.0
- Upgrade six to 1.15.0
- Upgrade toml to 0.10.1
- Upgrade urllib3 to 1.25.9

Improved Documentation
----------------------

- Add ``--no-input`` option to pip docs (`7688 &lt;https://github.com/pypa/pip/issues/7688&gt;`_)
- List of options supported in requirements file are extracted from source of truth,
  instead of being maintained manually. (`7908 &lt;https://github.com/pypa/pip/issues/7908&gt;`_)
- Fix pip config docstring so that the subcommands render correctly in the docs (`8072 &lt;https://github.com/pypa/pip/issues/8072&gt;`_)
- replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 &lt;https://github.com/pypa/pip/issues/8353&gt;`_)
- Fix example for defining multiple values for options which support them (`8373 &lt;https://github.com/pypa/pip/issues/8373&gt;`_)
- Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 &lt;https://github.com/pypa/pip/issues/8459&gt;`_)
- Add feature flags to docs (`8512 &lt;https://github.com/pypa/pip/issues/8512&gt;`_)
- Document how to install package extras from git branch and source distributions. (`8576 &lt;https://github.com/pypa/pip/issues/8576&gt;`_)
   ```
   
  
  
   ### 20.3b1
   ```
   ===================

Deprecations and Removals
-------------------------

- ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version.
  To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Deprecate support for Python 3.5 (`8181 &lt;https://github.com/pypa/pip/issues/8181&gt;`_)
- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)
- Deprecate ``--find-links`` option in ``pip freeze`` (`9069 &lt;https://github.com/pypa/pip/issues/9069&gt;`_)

Features
--------

- Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Allow multiple values for --abi and --platform. (`6121 &lt;https://github.com/pypa/pip/issues/6121&gt;`_)
- Add option ``--format`` to subcommand ``list`` of ``pip  cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 &lt;https://github.com/pypa/pip/issues/8355&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Make the ``setup.py install`` deprecation warning less noisy. We warn only
  when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as
  situations where both fails are most probably irrelevant to this deprecation. (`8752 &lt;https://github.com/pypa/pip/issues/8752&gt;`_)
- Check the download directory for existing wheels to possibly avoid
  fetching metadata when the ``fast-deps`` feature is used with
  ``pip wheel`` and ``pip download``. (`8804 &lt;https://github.com/pypa/pip/issues/8804&gt;`_)
- When installing a git URL that refers to a commit that is not available locally
  after git clone, attempt to fetch it from the remote. (`8815 &lt;https://github.com/pypa/pip/issues/8815&gt;`_)
- Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 &lt;https://github.com/pypa/pip/issues/8892&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)
- Support Python 3.9. (`8971 &lt;https://github.com/pypa/pip/issues/8971&gt;`_)
- Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 &lt;https://github.com/pypa/pip/issues/8975&gt;`_)
- Switch to the new dependency resolver by default. (`9019 &lt;https://github.com/pypa/pip/issues/9019&gt;`_)
- Remove the ``--build-dir`` option, as per the deprecation. (`9049 &lt;https://github.com/pypa/pip/issues/9049&gt;`_)

Bug Fixes
---------

- Propagate ``--extra-index-url`` from requirements file properly to session auth,
  so that keyring auth will work as expected. (`8103 &lt;https://github.com/pypa/pip/issues/8103&gt;`_)
- Allow specifying verbosity and quiet level via configuration files
  and environment variables. Previously these options were treated as
  boolean values when read from there while through CLI the level can be
  specified. (`8578 &lt;https://github.com/pypa/pip/issues/8578&gt;`_)
- Only converts Windows path to unicode on Python 2 to avoid regressions when a
  POSIX environment does not configure the file system encoding correctly. (`8658 &lt;https://github.com/pypa/pip/issues/8658&gt;`_)
- List downloaded distributions before exiting ``pip download``
  when using the new resolver to make the behavior the same as
  that on the legacy resolver. (`8696 &lt;https://github.com/pypa/pip/issues/8696&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- Avoid polluting the destination directory by resolution artifacts
  when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 &lt;https://github.com/pypa/pip/issues/8827&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)
- Tweak the output during dependency resolution in the new resolver. (`8861 &lt;https://github.com/pypa/pip/issues/8861&gt;`_)
- Correctly search for installed distributions in new resolver logic in order
  to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 &lt;https://github.com/pypa/pip/issues/8963&gt;`_)
- Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file
  with editable=True. Render it as a non-editable ``file://`` URL until modern
  editable installs are standardized and supported. (`8996 &lt;https://github.com/pypa/pip/issues/8996&gt;`_)

Vendored Libraries
------------------

- Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed.
  It is mandatory for ``pip debug`` command.

Improved Documentation
----------------------

- Add documentation for &#39;.netrc&#39; support. (`7231 &lt;https://github.com/pypa/pip/issues/7231&gt;`_)
- Add OS tabs for OS-specific commands. (`7311 &lt;https://github.com/pypa/pip/issues/7311&gt;`_)
- Add note and example on keyring support for index basic-auth (`8636 &lt;https://github.com/pypa/pip/issues/8636&gt;`_)
- Added initial UX feedback widgets to docs. (`8783 &lt;https://github.com/pypa/pip/issues/8783&gt;`_, `8848 &lt;https://github.com/pypa/pip/issues/8848&gt;`_)
- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
- Update user docs to reflect new resolver as default in 20.3. (`9044 &lt;https://github.com/pypa/pip/issues/9044&gt;`_)
- Improve migration guide to reflect changes in new resolver behavior. (`9056 &lt;https://github.com/pypa/pip/issues/9056&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
bors bot added a commit to duckinator/emanate that referenced this issue Dec 4, 2020
194: Update pytest-pylint to 0.18.0 r=duckinator a=pyup-bot


This PR updates [pytest-pylint](https://pypi.org/project/pytest-pylint) from **0.17.0** to **0.18.0**.



*The bot wasn't able to find a changelog for this release. [Got an idea?](https://github.com/pyupio/changelogs/issues/new)*

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pytest-pylint
  - Changelog: https://pyup.io/changelogs/pytest-pylint/
  - Repo: https://github.com/carsongee/pytest-pylint
</details>



197: Update pip to 20.3.1 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.2.4** to **20.3.1**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.3.1
   ```
   ===================

Deprecations and Removals
-------------------------

- The --build-dir option has been restored as a no-op, to soften the transition
  for tools that still used it. (`9193 &lt;https://github.com/pypa/pip/issues/9193&gt;`_)
   ```
   
  
  
   ### 20.3
   ```
   - Introduce a new ResolutionImpossible error, raised when pip encounters un-satisfiable dependency conflicts (`8546 &lt;https://github.com/pypa/pip/issues/8546&gt;`_, `8377 &lt;https://github.com/pypa/pip/issues/8377&gt;`_)
- Add a subcommand ``debug`` to ``pip config`` to list available configuration sources and the key-value pairs defined in them. (`6741 &lt;https://github.com/pypa/pip/issues/6741&gt;`_)
- Warn if index pages have unexpected content-type (`6754 &lt;https://github.com/pypa/pip/issues/6754&gt;`_)
- Allow specifying ``--prefer-binary`` option in a requirements file (`7693 &lt;https://github.com/pypa/pip/issues/7693&gt;`_)
- Generate PEP 376 REQUESTED metadata for user supplied requirements installed
  by pip. (`7811 &lt;https://github.com/pypa/pip/issues/7811&gt;`_)
- Warn if package url is a vcs or an archive url with invalid scheme (`8128 &lt;https://github.com/pypa/pip/issues/8128&gt;`_)
- Parallelize network operations in ``pip list``. (`8504 &lt;https://github.com/pypa/pip/issues/8504&gt;`_)
- Allow the new resolver to obtain dependency information through wheels
  lazily downloaded using HTTP range requests.  To enable this feature,
  invoke ``pip`` with ``--use-feature=fast-deps``. (`8588 &lt;https://github.com/pypa/pip/issues/8588&gt;`_)
- Support ``--use-feature`` in requirements files (`8601 &lt;https://github.com/pypa/pip/issues/8601&gt;`_)

Bug Fixes
---------

- Use canonical package names while looking up already installed packages. (`5021 &lt;https://github.com/pypa/pip/issues/5021&gt;`_)
- Fix normalizing path on Windows when installing package on another logical disk. (`7625 &lt;https://github.com/pypa/pip/issues/7625&gt;`_)
- The VCS commands run by pip as subprocesses don&#39;t merge stdout and stderr anymore, improving the output parsing by subsequent commands. (`7968 &lt;https://github.com/pypa/pip/issues/7968&gt;`_)
- Correctly treat non-ASCII entry point declarations in wheels so they can be
  installed on Windows. (`8342 &lt;https://github.com/pypa/pip/issues/8342&gt;`_)
- Update author email in config and tests to reflect decommissioning of pypa-dev list. (`8454 &lt;https://github.com/pypa/pip/issues/8454&gt;`_)
- Headers provided by wheels in .data directories are now correctly installed
  into the user-provided locations, such as ``--prefix``, instead of the virtual
  environment pip is running in. (`8521 &lt;https://github.com/pypa/pip/issues/8521&gt;`_)

Vendored Libraries
------------------

- Vendored htmlib5 no longer imports deprecated xml.etree.cElementTree on Python 3.
- Upgrade appdirs to 1.4.4
- Upgrade certifi to 2020.6.20
- Upgrade distlib to 0.3.1
- Upgrade html5lib to 1.1
- Upgrade idna to 2.10
- Upgrade packaging to 20.4
- Upgrade requests to 2.24.0
- Upgrade six to 1.15.0
- Upgrade toml to 0.10.1
- Upgrade urllib3 to 1.25.9

Improved Documentation
----------------------

- Add ``--no-input`` option to pip docs (`7688 &lt;https://github.com/pypa/pip/issues/7688&gt;`_)
- List of options supported in requirements file are extracted from source of truth,
  instead of being maintained manually. (`7908 &lt;https://github.com/pypa/pip/issues/7908&gt;`_)
- Fix pip config docstring so that the subcommands render correctly in the docs (`8072 &lt;https://github.com/pypa/pip/issues/8072&gt;`_)
- replace links to the old pypa-dev mailing list with https://mail.python.org/mailman3/lists/distutils-sig.python.org/ (`8353 &lt;https://github.com/pypa/pip/issues/8353&gt;`_)
- Fix example for defining multiple values for options which support them (`8373 &lt;https://github.com/pypa/pip/issues/8373&gt;`_)
- Add documentation for the ResolutionImpossible error that helps the user fix dependency conflicts (`8459 &lt;https://github.com/pypa/pip/issues/8459&gt;`_)
- Add feature flags to docs (`8512 &lt;https://github.com/pypa/pip/issues/8512&gt;`_)
- Document how to install package extras from git branch and source distributions. (`8576 &lt;https://github.com/pypa/pip/issues/8576&gt;`_)
   ```
   
  
  
   ### 20.3b1
   ```
   ===================

Deprecations and Removals
-------------------------

- ``pip freeze`` will stop filtering the ``pip``, ``setuptools``, ``distribute`` and ``wheel`` packages from ``pip freeze`` output in a future version.
  To keep the previous behavior, users should use the new ``--exclude`` option. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Deprecate support for Python 3.5 (`8181 &lt;https://github.com/pypa/pip/issues/8181&gt;`_)
- Document that certain removals can be fast tracked. (`8417 &lt;https://github.com/pypa/pip/issues/8417&gt;`_)
- Document that Python versions are generally supported until PyPI usage falls below 5%. (`8927 &lt;https://github.com/pypa/pip/issues/8927&gt;`_)
- Deprecate ``--find-links`` option in ``pip freeze`` (`9069 &lt;https://github.com/pypa/pip/issues/9069&gt;`_)

Features
--------

- Add ``--exclude`` option to ``pip freeze`` and ``pip list`` commands to explicitly exclude packages from the output. (`4256 &lt;https://github.com/pypa/pip/issues/4256&gt;`_)
- Allow multiple values for --abi and --platform. (`6121 &lt;https://github.com/pypa/pip/issues/6121&gt;`_)
- Add option ``--format`` to subcommand ``list`` of ``pip  cache``, with ``abspath`` choice to output the full path of a wheel file. (`8355 &lt;https://github.com/pypa/pip/issues/8355&gt;`_)
- Improve error message friendliness when an environment has packages with
  corrupted metadata. (`8676 &lt;https://github.com/pypa/pip/issues/8676&gt;`_)
- Make the ``setup.py install`` deprecation warning less noisy. We warn only
  when ``setup.py install`` succeeded and ``setup.py bdist_wheel`` failed, as
  situations where both fails are most probably irrelevant to this deprecation. (`8752 &lt;https://github.com/pypa/pip/issues/8752&gt;`_)
- Check the download directory for existing wheels to possibly avoid
  fetching metadata when the ``fast-deps`` feature is used with
  ``pip wheel`` and ``pip download``. (`8804 &lt;https://github.com/pypa/pip/issues/8804&gt;`_)
- When installing a git URL that refers to a commit that is not available locally
  after git clone, attempt to fetch it from the remote. (`8815 &lt;https://github.com/pypa/pip/issues/8815&gt;`_)
- Include http subdirectory in ``pip cache info`` and ``pip cache purge`` commands. (`8892 &lt;https://github.com/pypa/pip/issues/8892&gt;`_)
- Cache package listings on index packages so they are guarenteed to stay stable
  during a pip command session. This also improves performance when a index page
  is accessed multiple times during the command session. (`8905 &lt;https://github.com/pypa/pip/issues/8905&gt;`_)
- New resolver: Tweak resolution logic to improve user experience when
  user-supplied requirements conflict. (`8924 &lt;https://github.com/pypa/pip/issues/8924&gt;`_)
- Support Python 3.9. (`8971 &lt;https://github.com/pypa/pip/issues/8971&gt;`_)
- Log an informational message when backtracking takes multiple rounds on a specific package. (`8975 &lt;https://github.com/pypa/pip/issues/8975&gt;`_)
- Switch to the new dependency resolver by default. (`9019 &lt;https://github.com/pypa/pip/issues/9019&gt;`_)
- Remove the ``--build-dir`` option, as per the deprecation. (`9049 &lt;https://github.com/pypa/pip/issues/9049&gt;`_)

Bug Fixes
---------

- Propagate ``--extra-index-url`` from requirements file properly to session auth,
  so that keyring auth will work as expected. (`8103 &lt;https://github.com/pypa/pip/issues/8103&gt;`_)
- Allow specifying verbosity and quiet level via configuration files
  and environment variables. Previously these options were treated as
  boolean values when read from there while through CLI the level can be
  specified. (`8578 &lt;https://github.com/pypa/pip/issues/8578&gt;`_)
- Only converts Windows path to unicode on Python 2 to avoid regressions when a
  POSIX environment does not configure the file system encoding correctly. (`8658 &lt;https://github.com/pypa/pip/issues/8658&gt;`_)
- List downloaded distributions before exiting ``pip download``
  when using the new resolver to make the behavior the same as
  that on the legacy resolver. (`8696 &lt;https://github.com/pypa/pip/issues/8696&gt;`_)
- New resolver: Pick up hash declarations in constraints files and use them to
  filter available distributions. (`8792 &lt;https://github.com/pypa/pip/issues/8792&gt;`_)
- Avoid polluting the destination directory by resolution artifacts
  when the new resolver is used for ``pip download`` or ``pip wheel``. (`8827 &lt;https://github.com/pypa/pip/issues/8827&gt;`_)
- New resolver: If a package appears multiple times in user specification with
  different ``--hash`` options, only hashes that present in all specifications
  should be allowed. (`8839 &lt;https://github.com/pypa/pip/issues/8839&gt;`_)
- Tweak the output during dependency resolution in the new resolver. (`8861 &lt;https://github.com/pypa/pip/issues/8861&gt;`_)
- Correctly search for installed distributions in new resolver logic in order
  to not miss packages (virtualenv packages from system-wide-packages for example) (`8963 &lt;https://github.com/pypa/pip/issues/8963&gt;`_)
- Do not fail in pip freeze when encountering a ``direct_url.json`` metadata file
  with editable=True. Render it as a non-editable ``file://`` URL until modern
  editable installs are standardized and supported. (`8996 &lt;https://github.com/pypa/pip/issues/8996&gt;`_)

Vendored Libraries
------------------

- Fix devendoring instructions to explicitly state that ``vendor.txt`` should not be removed.
  It is mandatory for ``pip debug`` command.

Improved Documentation
----------------------

- Add documentation for &#39;.netrc&#39; support. (`7231 &lt;https://github.com/pypa/pip/issues/7231&gt;`_)
- Add OS tabs for OS-specific commands. (`7311 &lt;https://github.com/pypa/pip/issues/7311&gt;`_)
- Add note and example on keyring support for index basic-auth (`8636 &lt;https://github.com/pypa/pip/issues/8636&gt;`_)
- Added initial UX feedback widgets to docs. (`8783 &lt;https://github.com/pypa/pip/issues/8783&gt;`_, `8848 &lt;https://github.com/pypa/pip/issues/8848&gt;`_)
- Add ux documentation (`8807 &lt;https://github.com/pypa/pip/issues/8807&gt;`_)
- Update user docs to reflect new resolver as default in 20.3. (`9044 &lt;https://github.com/pypa/pip/issues/9044&gt;`_)
- Improve migration guide to reflect changes in new resolver behavior. (`9056 &lt;https://github.com/pypa/pip/issues/9056&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
Co-authored-by: Ellen Marie Dash <me@duckie.co>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 10, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
!release blocker Hold a release until this is resolved
Projects
No open projects
Development

No branches or pull requests

3 participants