pip-sync fails to run on pip >= 21.3

[SethMMorton]

It seems that pip 21.3 has removed get_installed_distributions from _internal.utils.misc. pip-sync is importing this, and that is causing a catastrophic error on startup.

Environment Versions

1. Linux
2. Python: 3.8.6
3. pip version: 21.3
4. pip-tools version: 6.3.1

Steps to replicate

1. Create a virtual environment and activate it
2. pip install -U pip
3. pip install pip-tools
4. pip-sync -h

Expected result

Help

Actual result

Traceback (most recent call last):
  File "/home/smmorton/playground/demo/bin/pip-sync", line 5, in <module>
    from piptools.scripts.sync import cli
  File "/home/smmorton/playground/demo/lib/python3.8/site-packages/piptools/scripts/sync.py", line 12, in <module>
    from pip._internal.utils.misc import get_installed_distributions
ImportError: cannot import name 'get_installed_distributions' from 'pip._internal.utils.misc' (/home/smmorton/playground/demo/lib/python3.8/site-packages/pip/_internal/utils/misc.py)

[SethMMorton]

I have isolated this to be caused by a change introduced in pip version 21.3. pip-sync runs fine with pip version 21.2.4.

[SethMMorton]

The problem was introduced in this commit: pypa/pip@d051a00#diff-058e40cb3a9ea705f655937e48f3a053f5dc7c500b7f1b2aae76e9bd673faf64

[groodt]

There is a Draft PR with a proposed fix #1501

It was known that this issue was coming, but there isn't a mechanism to coordinate releases with pip and pip-tools, perhaps if pip-tools were to become part of PyPA this could have been avoided.

[martinlyra]

Got this issue just recently on a fresh virtualenv install. So my workaround so far until the fix is out is to do this

pip install pip==21.2

or if you fancy...

python -m pip install pip==21.2

Probably the most confusing issue I had so far.

[atugushev]

This is released as part of pip-tools 6.4.0. Sorry for the inconvenience folks. I've missed the pip's release somehow.

[pradyunsg]

@atugushev Is there someone I should mention when the release issue is created, to give y'all a heads up?

[atugushev]

@pradyunsg I'd appreciate it if someone cc-ed me on the release issue. That would be enough for now. Thanks for chiming in 👍🏻

[fancydancing]

This is released as part of pip-tools 6.4.0. Sorry for the inconvenience folks. I've missed the pip's release somehow.

Hi Albert! We're running into same issues in our CI/CD. I have read the thread and seems 6.4.0 should work fine with pip 21.3? But I still get an error. Below is the traceback, you can see from it that we have pip 21.3 and pip-sync 6.4.0 installed. I would appreciate if you give some advice - maybe something is still wrong here?

Requirement already up-to-date: pip in ./.venv/lib/python3.7/site-packages (21.3)
$ pip install -U pip-tools
Collecting pip-tools
  Downloading pip_tools-6.4.0-py3-none-any.whl (47 kB)
Requirement already satisfied, skipping upgrade: pep517 in ./.venv/lib/python3.7/site-packages (from pip-tools) (0.10.0)
Requirement already satisfied, skipping upgrade: setuptools in ./.venv/lib/python3.7/site-packages (from pip-tools) (49.6.0)
Requirement already satisfied, skipping upgrade: pip>=21.2 in ./.venv/lib/python3.7/site-packages (from pip-tools) (21.3)
Requirement already satisfied, skipping upgrade: wheel in ./.venv/lib/python3.7/site-packages (from pip-tools) (0.35.1)
Requirement already satisfied, skipping upgrade: click>=7 in ./.venv/lib/python3.7/site-packages (from pip-tools) (7.0)
Requirement already satisfied, skipping upgrade: zipp; python_version < "3.8" in ./.venv/lib/python3.7/site-packages (from pep517->pip-tools) (3.2.0)
Requirement already satisfied, skipping upgrade: importlib_metadata; python_version < "3.8" in ./.venv/lib/python3.7/site-packages (from pep517->pip-tools) (2.0.0)
Requirement already satisfied, skipping upgrade: toml in ./.venv/lib/python3.7/site-packages (from pep517->pip-tools) (0.10.2)
Installing collected packages: pip-tools
  Attempting uninstall: pip-tools
    Found existing installation: pip-tools 5.3.1
    Uninstalling pip-tools-5.3.1:
      Successfully uninstalled pip-tools-5.3.1
Successfully installed pip-tools-6.4.0
$ pip-sync -q dev_requirements.txt
Traceback (most recent call last):
  File "/builds/btcm/veelo/.venv/bin/pip-sync", line 5, in <module>
    from piptools.scripts.sync import cli
  File "/builds/btcm/veelo/.venv/lib/python3.7/site-packages/piptools/scripts/sync.py", line 12, in <module>
    from pip._internal.metadata import get_environment
ModuleNotFoundError: No module named 'pip._internal.metadata'

[atugushev]

Hello @fancydancing,

Could you please provide the following info?

$ python -m pip --version
$ python -m piptools --version

[fancydancing]

@atugushev

$ python -m pip --version
pip 20.2.2 from /builds/btcm/veelo/.venv/lib/python3.7/site-packages/pip (python 3.7)

For second line I get the same error as before:

$ python -m piptools --version
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/builds/btcm/veelo/.venv/lib/python3.7/site-packages/piptools/__main__.py", line 3, in <module>
    from piptools.scripts import compile, sync
  File "/builds/btcm/veelo/.venv/lib/python3.7/site-packages/piptools/scripts/sync.py", line 12, in <module>
    from pip._internal.metadata import get_environment
ModuleNotFoundError: No module named 'pip._internal.metadata'

I also ran pip show pip-tools, the result is:

$ pip show pip-tools
Name: pip-tools
Version: 6.4.0
Summary: pip-tools keeps your pinned dependencies fresh.
Home-page: https://github.com/jazzband/pip-tools/
Author: Vincent Driessen
Author-email: me@nvie.com
License: BSD 3 Clause
Location: /builds/btcm/veelo/.venv/lib/python3.7/site-packages
Requires: setuptools, wheel, pip, pep517, click
Required-by: 

So the pip is 20.2.2, not 21.3, although when trying to upgrade it for venv, I got the message from my previous comment - pip is up to date and is 21.3.

[atugushev]

@fancydancing could you open a separate issue? Let's track it there, this issue is closed.

[fancydancing]

@atugushev sure, will do, thanks!