21.3: Breaking editable installation with No module named pip error

[Panaetius]

Description

Our CI is failing due to version 21.3 release, in the pip wheel subprocess call during installation, with seemingly random packages being the culprit.

E.g.

  subprocess.CalledProcessError: Command '['/opt/hostedtoolcache/Python/3.7.12/x64/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpiq0bznxg', '--quiet', 'cryptography<3.5,>=3.4.1']' returned non-zero exit status 1.

Full Log attached below

The specific dependency it fails for seems completely random, in 5 subsequent runs it's been pathspec, cryptography, tabulate, rdflib and pyasn1.

I can reproduce the issue locally as well with Arch Linux, 21.3 fails, 21.2.4 works.
This only happens in editable mode (using -e flag), without it everything install fine.

Expected behavior

Renku Package to be installed without problem

pip version

21.3

Python version

3.7.8

OS

Arch Linux/Github Actions(Ubuntu)

How to Reproduce

1. clone https://github.com/SwissDataScienceCenter/renku-python
2. upgrade to pip 21.3
3. do pip install -e .[all]

Output

$ python -m pip install -e .[all]
/home/user/.pyenv/versions/renku-python/lib/python3.7/site-packages/secretstorage/dhcrypto.py:16: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
/home/user/.pyenv/versions/renku-python/lib/python3.7/site-packages/secretstorage/util.py:25: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead
  from cryptography.utils import int_from_bytes
Obtaining file:///home/user/DEV/ETH/renku-python
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build editable: started
  Getting requirements to build editable: finished with status 'done'
  Preparing editable metadata (pyproject.toml): started
  Preparing editable metadata (pyproject.toml): finished with status 'done'
  Build backend does not support editables, falling back to setup.py egg_info.
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /home/user/.pyenv/versions/renku-python/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/user/DEV/ETH/renku-python/setup.py'"'"'; __file__='"'"'/home/user/DEV/ETH/renku-python/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-0jfl5x91
       cwd: /home/user/DEV/ETH/renku-python/
  Complete output (31 lines):
  /home/user/.pyenv/versions/renku-python/bin/python: No module named pip
  Traceback (most recent call last):
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 75, in fetch_build_egg
      subprocess.check_call(cmd)
    File "/home/user/.pyenv/versions/3.7.8/lib/python3.7/subprocess.py", line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/home/user/.pyenv/versions/renku-python/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpppr5o_gk', '--quiet', 'zc.relation<1.2,>=1.1']' returned non-zero exit status 1.
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/home/user/DEV/ETH/renku-python/setup.py", line 266, in <module>
      "Development Status :: 4 - Beta",
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 152, in setup
      _install_setup_requires(attrs)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/__init__.py", line 147, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 809, in fetch_build_eggs
      replace_conflicting=True,
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 768, in resolve
      replace_conflicting=replace_conflicting
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1051, in best_match
      return self.obtain(req, installer)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1063, in obtain
      return installer(requirement)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/dist.py", line 877, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/tmp/pip-build-env-gh6wm4a4/overlay/lib/python3.7/site-packages/setuptools/installer.py", line 77, in fetch_build_egg
      raise DistutilsError(str(e)) from e
  distutils.errors.DistutilsError: Command '['/home/user/.pyenv/versions/renku-python/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpppr5o_gk', '--quiet', 'zc.relation<1.2,>=1.1']' returned non-zero exit status 1.
  ----------------------------------------
WARNING: Discarding file:///home/user/DEV/ETH/renku-python. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[teije01]

Experiencing the same problems, also

/opt/conda/envs/vdsdb/bin/python3.7: No module named pip

Same with other python versions. Downgrading pip resolved our problem

[sbidoul]

Ok, this seems to be related to the PEP 660 implementation.

Since this project has a pyproject.toml without build-system table, it seems that pip<21.3 did first prepare the metadata using the setuptools.build_meta:__legacy__ build backend, then called setup.py develop, so doing a mix of PEP 517 and legacy installation.

Now pip first tries a PEP 660 editable metadata preparation and, since the build backend does not support PEP 660, it falls back to a legacy metadata preparation (setup.py egg_info), which should in theory be equivalent but, for some reason I don't fully understand, does not work for your project.

[sbidoul]

@Panaetius can you test if #10577 fixes the issue ?

You can install it with pip install -U "pip @ git+https://github.com/pypa/pip@refs/pull/10577/head"

[sbidoul]

Note I tentatively added this to a 21.3.1 bugfix milestone, since it is a regression, but this does not mean doing an actual bugfix release has been decided yet.

[Panaetius]

It gets a lot further in the install process but then unfortunately has the same error when trying to install things:

https://gist.github.com/Panaetius/5d9b29b3694b310e567bc803fb9878a8

Thank you for looking into this!

[sbidoul]

@Panaetius Could you retry in a fresh environment ? Also removing any .eggs and .egg-info directories that might have been created by previous attempts.

[ceelian]

Having a similar problem on our CI System since the new release. Our CI does a fresh setup of the environment in each build. So IMO it looks like the "freshness" of the environment isn't the cause. Using the release versions pip-21.3 setuptools-58.2.0.

Our workaround for now: pinning versions to pip==21.2.4 setuptools==57.4.0

[sbidoul]

@ceelian my comment about retrying in a fresh environment was specifically for @Panaetius who is testing the linked PR.
If you'd like to test #10577 too, this is welcome.

[ceelian]

@sbidoul sorry, was too fast with typing and too slow with reading. Thx for the support!

[Panaetius]

https://gist.github.com/Panaetius/2fb17b122c88a9fc75d7ad650a1b8807 in a new virtualenv

[sbidoul]

@Panaetius is this with the master branch of https://github.com/SwissDataScienceCenter/renku-python (so I can try to reproduce myself) ?

[Panaetius]

Yes with current master, I did a fresh clone to test it.

[layday]

If I understand correctly, setuptools is calling pip to install setup_requires from egg_info and pip itself is calling setuptools from an "overlay" (i.e. an isolated build environment) where pip itself is not available. This is why egg_info didn't work in the first place and why it fails now when setuptools is calling egg_info prior to building the package.

[ceelian]

Tested @sbidoul's patch in our CI, same error as @Panaetius.

[sbidoul]

Alright, I I'm slowly nailing it...

A minimal reproducer for this is an empty pyproject.toml and a setup.py with any setup_requires.

I missed the fact that setup.py develop is running in an isolated build (!) environment if there is a pyproject.toml 🤷. I thought editable were totally independent of PEP 518/517 things until PEP 660, and this false assumption partly misled my PEP 660 implementation...

According to the changelog, this dates back to #4999. This presumably done so pip installs setup_requires requirements and not setuptools (which back then was not using pip) ?

Since the default implementation of get_requires_for_build_editable returns [], pip does not install the setup_requires, and setuptools attemps to do it by itself... in the isolated environment which does not have pip.

I'm pushing a hack commit to #10577 to let my thoughts settle down but I think I'll have to re-do metadata preparation to detect the missing get_requires_for_build_editable hook and fall back to regular metadata preparation much earlier.

[sbidoul]

In the meantime, a possible workaround is to use pip install -e . --no-use-pep517 when you have a setup.py that has a setup_requires and also a pyproject.toml.

[swhmirror]

On Sat, Oct 16, 2021, at 05:03, Tzu-ping Chung wrote: @swhmirror <https://github.com/swhmirror> Please avoid mentioning issues repeatedly from another repository, these messages are spamming the issue.

Hi, We've had to do two different changes to two different repositories to work around this issue in two different systems, so two Software Heritage team members have mentioned this issue once in two different commits, which happened to both be pushed to GitHub by our @swhmirror bot. Does this really qualify as spamming? If so we'll try to avoid mentioning any pip bugs in our commit messages in the future. Thanks for maintaining pip, Nicolas

[pradyunsg]

If so we'll try to avoid mentioning any pip bugs in our commit messages in the future.

That would indeed help! GitHub shows any-and-every place that mentions an issue, in the issue view which results in collapsing useful comments. :)

[layday]

I guess you could create a tracking issue on your bug tracker and link to the pip issue there to reduce flooding - people shouldn't be discouraged from adding context to their commits because of how GH does references.