Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Virtualenv with --system-site-packages breaks pip's build isolation #6264

Open
pganssle opened this issue Feb 13, 2019 · 29 comments

Comments

@pganssle
Copy link
Member

@pganssle pganssle commented Feb 13, 2019

It seems that if you create a virtualenv with --system-site-packages, the system packages (but not the user packages) will be on the PYTHONPATH in the PEP 517 isolated build environment that pip creates (it does not effect python -m pep517.build), and it seems they will be on there with higher precedence than the requirements installed in the build environment. It only affects pip >= 19.0.0.

The most common way I've seen this cause problems is with the new setuptools.build_meta:__legacy__ backend. Since the most recent version of pip requires a recent version of setuptools, if you have an older version of setuptools installed on the system, pip install will fail due to the missing setuptools.build_meta:__legacy__ backend. It is possible to reproduce this by crafting a deliberately bad package and installing it, then creating a wheel for a newer version of the package (which allowed me to test that this failure was actually introduced with pip==19.0.0), but for the MWE we can stick with setuptools.

To reproduce, create a package like this:

cd /tmp
mkdir demo
cd demo
mkdir badpkg

touch badpkg/pyproject.toml
echo 'from setuptools import setup; setup(name="foo", version="0.1.0")' \
    > badpkg/setup.py

Then install an older version of setuptools on your system (can't be --user or in the virtualenv), pip install 'setuptools < 40.8.0' (I did this in a pyenv environment). If your system already has an older version of setuptools on it, you're already good.

Next create a virtualenv with --system-site-packages and activate it:

virtualenv venv --system-site-packages
source venv/bin/activate

Finally try to install badpkg (pip wheel or pip install -t tmp also works to demonstrate the problem):

pip install ./badpkg

You should get a traceback the ends like this:

AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

At first I thought this was because the affected packages had too loose bounds on build-system.requires (e.g. requires=["setuptools"]) and that pip was failing to install a more recent version in the isolated environment, but this bug still occurs even if you specify "setuptools>=40.8.0", so I believe it's not a problem at install-time, it's a problem with the path resolution at build time.

At the moment it's unclear if this is a problem with pip or virtualenv, but since it does not affect python -m pep517.build, I'm reporting it here. It could be a combination of both.

CC: @gaborbernat

@pganssle

This comment has been minimized.

Copy link
Member Author

@pganssle pganssle commented Feb 13, 2019

Per my Bloomberg colleague @lkollar, it seems that the problem is that pip creates a custom sitecustomize.py file in order to inject itself into the build environment.

Looks to me like it's just blacklisting the system packages with distutils.get_python_lib, which doesn't include the system path added by virtualenv. Seems like pep517.envbuild.BuildEnvironment doesn't have this problem - maybe switch over to using that? Or was there a reason to do it this way?

@boegel

This comment has been minimized.

Copy link

@boegel boegel commented Feb 26, 2019

I'm also seeing this problem, when installing cryptography 2.5 using Python 3.7.2, even though I'm using the latest setuptools (40.8.0).

What I don't understand is why I'm seeing this even though I'm using pip install --no-build-isolation .... How is that possible if the problem is related to pip's build isolation feature?

@boegel

This comment has been minimized.

Copy link

@boegel boegel commented Feb 26, 2019

I should also mention that I'm not using virtualenv at all (although maybe somehow the cryptography installation procedure does that somehow?).

@pganssle

This comment has been minimized.

Copy link
Member Author

@pganssle pganssle commented Feb 26, 2019

@boegel Can you create a minimal-working example that reproduces this, and include your pip versions? I've never known anyone to be able to trigger this without virtualenv --system-site-packages.

@boegel

This comment has been minimized.

Copy link

@boegel boegel commented Feb 26, 2019

@pganssle Does this suffice?

Note that this is a self-built Python 3.7.2 from source, in a custom location, not sure if that's relevant to the problem...

$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug

$ curl -OL https://files.pythonhosted.org/packages/69/ed/5e97b7f54237a9e4e6291b6e52173372b7fa45ca730d36ea90b790c0059a/cryptography-2.5.tar.gz

$ tar xfz cryptography-2.5.tar.gz

$ cd cryptography-2.5

$ pip install --prefix=$PWD --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/user/pip_debug/cryptography-2.5
    Preparing wheel metadata ... error
    Complete output from command /prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e:
    Traceback (most recent call last):
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/Python/3.7.2-GCCcore-8.2.0/bin/python /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpkwc5sc4e" failed with error code 1 in /tmp/pip-req-build-q9knj58j

This is on CentOS 7.6.1810, with only python 2.x installed on the system. We do have an old setuptools in the OS as well, but I think that one is irrelevant since that's only for Python 2.

$ rpm -qa | grep ^python[0-9]*-[0-9]
python-2.7.5-76.el7.x86_64
$ rpm -qa | grep setuptools
python-setuptools-0.9.8-7.el7.noarch
@boegel

This comment has been minimized.

Copy link

@boegel boegel commented Feb 26, 2019

I should also mention that --no-use-pep517 fixes the problem for me.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Feb 26, 2019

pip 19.0.3 from /prefix/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip-19.0.3-py3.7.egg/pip (python 3.7)

Looks like your pip (and your setuptools) have been installed as eggs, presumably using easy_install. This is obsolete, can you try with a copy of pip installed using pip?

@boegel

This comment has been minimized.

Copy link

@boegel boegel commented Mar 7, 2019

@pfmoore Same problem when pip and setuptools were installed with pip, when using latest cryptography from PyPI. Note that there's no other Python 3 installation on the system (and hence no other setuptools either).

$ which python
/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python
$ python -V
Python 3.7.2

$ pip -V
pip 19.0.3 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip (python 3.7)

$ easy_install --version
setuptools 40.8.0 from /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages (Python 3.7)

$ mkdir -p /tmp/$USER/pip_debug
$ cd /tmp/$USER/pip_debug
$ curl -OL https://files.pythonhosted.org/packages/07/ca/bc827c5e55918ad223d59d299fff92f3563476c3b00d0a9157d9c0217449/cryptography-2.6.1.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  480k  100  480k    0     0   755k      0 --:--:-- --:--:-- --:--:--  754k

$ tar xfz cryptography-2.6.1.tar.gz
$ cd cryptography-2.6.1

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  .
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
    Preparing wheel metadata ... error
    Complete output from command /prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_:
    Traceback (most recent call last):
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
        main()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 62, in prepare_metadata_for_build_wheel
        backend = _build_backend()
      File "/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py", line 39, in _build_backend
        obj = getattr(obj, path_part)
    AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

    ----------------------------------------
Command "/prefix/software/Python/3.7.2-GCCcore-8.2.0/bin/python3.7 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/pip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /tmp/tmpb28ucw3_" failed with error code 1 in /tmp/pip-req-build-s_w63g1g

Works fine when using --no-use-pep517:

$ pip install --prefix=/tmp/$USER --no-deps  --ignore-installed  --no-build-isolation  . --no-use-pep517
Processing /tmp/myuser/pip_debug/cryptography-2.6.1
Building wheels for collected packages: cryptography
  Building wheel for cryptography (setup.py) ...
...
@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Mar 7, 2019

module 'setuptools.build_meta' has no attribute 'legacy'

That indicates you're not picking up setuptools 40.8.0 - see here

Try running

import setuptools.build_meta
print(setuptools.build_meta.__legacy__)

You should get something like <setuptools.build_meta._BuildMetaLegacyBackend object at 0x000000000401B2E8>. If you're not, then there's something wrong with your system setuptools. If you are, your build environment is not picking up the same packages as your system Python - in spite of your assertion that there's only the one setuptools installed on your system :-(

Also, the fact that you're using --prefix may be relevant here - I don't know how that interacts (if at all) with build environments. Can you reproduce this issue without --prefix?

@gaborbernat

This comment has been minimized.

Copy link

@gaborbernat gaborbernat commented Mar 7, 2019

@pfmoore if virtualenv inherits the system site package (and that has a setuptoools before 40.8.0) the pip install of the setuptools build dependency is ignored (as pip does not check the version of what is installed unless -U is passed, and says any setuptools satisfies the install requirement); therefore once pip tries to get the build backend the above failure is thrown. I don't think there needs to be anything wrong with system python to fall into this issue. pip should make sure to never-ever create an isolated build environment that inherits from the system site package, which is not the case at the moment.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Mar 7, 2019

if virtualenv inherits the system site package

Presumably when --no-build-isolation is specified? An isolated build ignores the existing installation.

(and that has a setuptoools before 40.8.0)

Well, the point of --no-build-isolation is that the user is agreeing to set up the correct build environment, so that seems like it's a simple case of user error.

However, in this case @boegel is claiming that he has setuptools 40.8.0 installed, and no other setuptools is present. If that's the case, and his setuptools doesn't contain build_meta.__legacy__, then that setuptools installation is somehow broken, because 40.8.0 does contain that backend.

My suspicion is that @boegel is actually mistaken, and there's another copy of setuptools lingering around somewhere, which is getting picked up and is confusing things. The other possibility is that there's a bug somewhere - but if that is the case, then we're currently struggling to reduce the example to something that can easily reproduce/demonstrate that bug.

pip should make sure to never-ever create an isolated build environment that inherits from the system site package

Well, yes, that's sort of the definition of "isolated" :-) But I'm not sure how relevant that is in this case, where --no-build-isolation is set.

which is not the case at the moment

That's what @pganssle is saying in the original post in this thread, yes. But I'm responding to @boegel, who is claiming that he can reproduce the problem without a virtualenv, using --no-build-isolation.

If @boegel's problem turns out to be real, then I don't think it's related to this one (there's too many differences in the steps to reproduce it). So it should probably be a separate issue. But I'd be interested to understand why he thinks it is the same issue...

@boegel

This comment has been minimized.

Copy link

@boegel boegel commented Mar 7, 2019

Dug a little further, it looks it's indeed a case of a broken setuptools installation, in some sense...

First, I should clarify my statement w.r.t. not having another setuptools installed. I should clarify that I mean that there's no other setuptools installed for Python 3.x. There is one for Python 2 (which is available in the OS), I'm assuming that one will never be picked up by Python 3.

That said, I'm pretty sure the right setuptools is being picked up:

$ python -c "import setuptools; print(setuptools.__file__); print(setuptools.version.__version__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/__init__.py
40.8.0

Yet:

$ python -c "import setuptools.build_meta; print(setuptools.build_meta.__file__); print(setuptools.build_meta.__legacy__)"
/prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools/build_meta.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'

So I do have setuptools 40.8.0, but no __legacy__ in setuptools.build_meta?!

This sheds some light on it:

ls -ld /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools*
drwxr-xr-x 6 myuser myuser 4096 Mar  7 18:48 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:56 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.6.2.dist-info
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:57 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools-40.8.0-py3.7.egg-info
drwxr-xr-x 3 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm
drwxr-xr-x 2 myuser myuser 4096 Mar  7 11:58 /prefix/software/Python/3.7.2-GCCcore-8.2.0/lib/python3.7/site-packages/setuptools_scm-3.2.0.dist-info

The setuptools-40.6.2.dist-info looks very suspicious, it seems like I somehow ended up with a frankenbuild of setuptools-40.6.2 (which doesn't have __legacy__ in setuptools.build_meta yet) and setuptools-40.8.0?! Ugh...

I got redirect to this issue via pypa/setuptools#1694, which shows the exact same error I'm running into, so I naively assumed it's the same problem.

It seems in my case it's indeed because of a corrupted setuptools though, my apologies for the noise...

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Mar 7, 2019

No problem, glad we got to the bottom of it!

@Lawouach

This comment has been minimized.

Copy link

@Lawouach Lawouach commented Mar 8, 2019

Hey,

I'm running into this issue as well and I'm thoroughly confused. As a mere user, it looks totally broken with no clear path of understanding how I'm doing anything wrong.

@Lawouach

This comment has been minimized.

Copy link

@Lawouach Lawouach commented Mar 8, 2019

Well, having said that. I ended up removing the system-wide setuptools package and now local pip doesn't complain.

@tgs

This comment has been minimized.

Copy link
Contributor

@tgs tgs commented Mar 14, 2019

I'm encountering the same issue, I think. My organization defaults virtualenvs to --system-site-packages, and the globally installed setuptools is 40.4.3. I cloned the pip git repo, and then:

In a system-site-packages virtualenv, pip install ./pip/ gives the AttributeError: module 'setuptools.build_meta' has no attribute '__legacy__'.

In a virtualenv with no-site-packages, pip install ./pip/ works correctly.

I think I agree with @gaborbernat that it would be better for "build isolation" to include ignoring --system-site-packages. Or maybe there's at least some way to give a better error message?

@webknjaz

This comment has been minimized.

Copy link
Member

@webknjaz webknjaz commented Apr 2, 2019

FTR I'm hitting this with tox + system-site-packages with setuptools>=40.8.0 in pyproject.toml under Python 2.7.15. I was actually hitting pypa/setuptools#1136 originally (which I knew was fixed a while back) and only after some time of playing around with removing pyproject.toml and then just playing with --no-build-isolation I've found this one...

Couldn't figure it out for a while because it was only reproducible in Travis CI which drops you into a virtualenv from the beginning automatically. Had to request debug access to their VMs to nail it down...

@webknjaz

This comment has been minimized.

Copy link
Member

@webknjaz webknjaz commented Apr 4, 2019

Okay, now I'm hitting this in readthedocs/readthedocs.org#5572 somehow.

@webknjaz

This comment has been minimized.

Copy link
Member

@webknjaz webknjaz commented Apr 4, 2019

@webknjaz

This comment has been minimized.

Copy link
Member

@webknjaz webknjaz commented Apr 4, 2019

@pganssle it looks like it's time to remove --system-site-packages from the title. I've already hit two instances of the bug with --no-site-packages.

@pganssle

This comment has been minimized.

Copy link
Member Author

@pganssle pganssle commented Apr 4, 2019

@webknjaz How does it manifest with --no-site-packages? That may even be a distinct bug.

@webknjaz

This comment has been minimized.

Copy link
Member

@webknjaz webknjaz commented Apr 4, 2019

@pganssle it looks the same to me: setuptools fails to use a newer feature because it's not the version of setuptools I specified in pyproject.toml

@webknjaz

This comment has been minimized.

Copy link
Member

@webknjaz webknjaz commented Apr 4, 2019

@webknjaz

This comment has been minimized.

Copy link
Member

@webknjaz webknjaz commented Apr 4, 2019

Yea, it's another missing feature hit but the root of the issue is still the same.

@pganssle

This comment has been minimized.

Copy link
Member Author

@pganssle pganssle commented Apr 4, 2019

I just meant we already know the cause of the problem with --system-site-packages , can you verify if it's the same mechanism here?

@webknjaz

This comment has been minimized.

Copy link
Member

@webknjaz webknjaz commented Apr 4, 2019

@pganssle can you point me to the comment with verification plan, please? I've scanned through and it's not obvious which one to follow...

@pganssle

This comment has been minimized.

Copy link
Member Author

@pganssle pganssle commented Apr 4, 2019

@webknjaz Just this: #6264 (comment)

TBH I remember verifying this by setting a breakpoint or printing something, but I guess at the time it seemed obvious what to do once you knew the cause 😅.

@webknjaz

This comment has been minimized.

Copy link
Member

@webknjaz webknjaz commented Apr 5, 2019

I'd need more precise instructions because debugging this with all subprocesses and stuff is PITA and unfortunately I don't have time for this right now... Maybe later.

humitos added a commit to readthedocs/readthedocs-docker-images that referenced this issue Apr 8, 2019
The issue related to pypa/pip is happening because pip somehow fails to properly isolate build env in PEP 517 mode.

Related to readthedocs/readthedocs.org#5572
Related to pypa/pip#6264
webknjaz added a commit to webknjaz/molecule that referenced this issue Apr 16, 2019
webknjaz added a commit to ansible/molecule that referenced this issue Apr 17, 2019
* 📝 Document having user-level setuptools updated

* 📝 Bump versions in the Test PyPI demo

* 📝 Update the recommendations about bleeding-edge

* 📝 Reword the tox mention in the install doc

Co-authored-by: Luke Murphy <lukewm@riseup.net>

* 📝 Add a link to the pip bug

Ref: pypa/pip#6264
@rkaplan

This comment has been minimized.

Copy link

@rkaplan rkaplan commented Jul 15, 2019

Hitting this issue when installing scikit-learn inside a --system-site-packages virtualenv that uses Python 3.6.8 on Ubuntu 18.04. My setuptools version is 41.0.1.

webknjaz added a commit to evitalis/ansible that referenced this issue Jul 26, 2019
facebook-github-bot added a commit to facebookincubator/magma that referenced this issue Nov 6, 2019
Summary:
There's a bug for virtualenv installations using pip when pip version is `>=19.0.0` AND setuptools system package version is less than `40.8.0` (setuptools on VM we have is `33.1.1`) AND using `--system-site-packages` flag for virtualenv installation. (similarly described on: pypa/pip#6264).

`lte` gateway python build passes but on `orc8r` module when trying to install egg link it fails with:
```
Installing egg link for orc8r
/home/vagrant/build/python/bin/pip3 install -q -U --cache-dir ~/.pipcache -e .[dev]
  WARNING: orc8r 0.0.0 does not provide the extra 'dev'
...
ImportError: No module named 'setuptools.build_meta'
```

Update: psutil recently added 'setuptools.build_meta' dependency on pyproject.toml file, this update fires the bug described above. giampaolo/psutil#1565

Reviewed By: andreilee

Differential Revision: D18305887

fbshipit-source-id: 33858079dcc5eefba93c1ee3de9bae7cc7d9d147
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
10 participants
You can’t perform that action at this time.