New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

assertion when using --no-cache-dir in 19.0 #6158

Closed
snstanton opened this Issue Jan 22, 2019 · 55 comments

Comments

Projects
None yet
@snstanton
Copy link

snstanton commented Jan 22, 2019

Environment

  • pip version: 19.0
  • Python version: 3.6.7
  • OS: Linux 50de819ca3ba 4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018 x86_64 GNU/Linux

Running in a dockerfile.

Description

The following command works with pip 18.1 and fails with 19.0.

pip3 install --no-cache-dir --upgrade -r requirements.txt

With 19.0, it fails with the following exception:

Exception:
Traceback (most recent call last):
  File "/Users/scotts/.virtualenvs/python3/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/Users/scotts/.virtualenvs/python3/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/Users/scotts/.virtualenvs/python3/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

Removing the --no-cache-dir flag causes the install to succeed.
requirements.txt

@Mohitsharma44

This comment has been minimized.

Copy link

Mohitsharma44 commented Jan 22, 2019

Same thing happening with:
Python v3.6.8
pip version 18.1
on
Ubuntu:latest image.

@snstanton what base image are you using? I'm seeing a similar issue on pip v18.1 as well

@gst

This comment has been minimized.

Copy link

gst commented Jan 22, 2019

I've got the exact same issue.
on my side it seems it doesn't matter which package/distribution I try to install

@dan-blanchard

This comment has been minimized.

Copy link

dan-blanchard commented Jan 22, 2019

I'm seeing this even without --no-cache-dir set. It happens for all packages I try to install, even if they're already installed.

  • pip version: 19.0
  • Python version: 3.6.0
  • OS: Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-91-generic x86_64)

I should note that in my case I'm seeing it when running pip with a combination of sudo -H and bash -l -c.

$ sudo -H bash -l -c  "/data/virtualenvs/events_beta/bin/pip install hypothesis"
Looking in indexes: https://pypi.org/simple, http://pypi.lan.cogtree.com/cogtree/simple/
Collecting hypothesis
  Downloading http://pypi.lan.cogtree.com/cogtree/simple/hypothesis/hypothesis-4.1.0-py3-none-any.whl (238kB)
    100% |████████████████████████████████| 245kB 120.5MB/s
Requirement already satisfied: attrs>=16.0.0 in /data/virtualenvs/events_beta/lib/python3.6/site-packages (from hypothesis) (18.2.0)
Exception:
Traceback (most recent call last):
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/data/virtualenvs/events_beta/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

Running the same commands without -l on my bash -c, or without bash -l -c involved at all, it all works fine.

$ sudo -H bash -c  "/data/virtualenvs/events_beta/bin/pip install hypothesis"
Collecting hypothesis
  Downloading https://files.pythonhosted.org/packages/89/7b/d6206dcde963139daa03a1d85b0c3428cb3ebf2ae8de3244b14a63e22680/hypothesis-4.1.0.tar.gz (180kB)
    100% |████████████████████████████████| 184kB 33.7MB/s
Requirement already satisfied: attrs>=16.0.0 in /data/virtualenvs/events_beta/lib/python3.6/site-packages (from hypothesis) (18.2.0)
Building wheels for collected packages: hypothesis
  Building wheel for hypothesis (setup.py) ... done
  Stored in directory: /root/.cache/pip/wheels/10/12/eb/4ab734432e8466d545c8501f531458845b45e8c4427d5367f9
Successfully built hypothesis
Installing collected packages: hypothesis
Successfully installed hypothesis-4.1.0

Fascinatingly, if I run the same command without sudo or bash involved at all, it still fails with the same error, so it seems like it's some weird permissions issue maybe.

@tgs

This comment has been minimized.

Copy link
Contributor

tgs commented Jan 22, 2019

Another workaround for some situations

For people who hit this bug because virtualenv is automatically installing the latest version of pip, you can work around it by giving virtualenv the --no-download option, or setting VIRTUALENV_NO_DOWNLOAD=1.

But be aware that this may give you a very old version of pip, depending on the last time you upgraded virtualenv.

This also works with tox: VIRTUALENV_NO_DOWNLOAD=1 tox.

@gst

This comment has been minimized.

Copy link

gst commented Jan 22, 2019

for what it's worth : it also fails with same error if package is already installed:

gregory.starck@canon:~/tmp$ ./venv/bin/pip install --no-cache-dir six ; echo $?
Looking in indexes: http://pypi:3141/root/ax/+simple/
Requirement already satisfied: six in ./venv/lib/python3.6/site-packages (1.12.0)
Exception:
Traceback (most recent call last):
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/home/gregory.starck/tmp/venv/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError
2
gregory.starck@canon:~/tmp$
@cherls

This comment has been minimized.

Copy link

cherls commented Jan 22, 2019

Ran into the same issue. Ended up pinning the pip version as a fix for now.

pip install --upgrade pip==18.1

@D0han

This comment has been minimized.

Copy link

D0han commented Jan 22, 2019

Issue is with failing assert, so setting env PYTHONOPTIMIZE=1 (or with parameter -O) makes this error go away.
Just tested it.
This workaround works because python optimizes code removing all asserts as one of the things.
Do not go for =2 (or -OO), as this removes docstrings and other tracebacks will appear - some code wants to operate on them.

@vinnyfuria

This comment has been minimized.

Copy link

vinnyfuria commented Jan 22, 2019

It looks like someone knew this might end up being an issue (source):

        # TODO: This check fails if --no-cache-dir is set. And yet we
        #       might be able to build into the ephemeral cache, surely?
        building_is_possible = self._wheel_dir or (
            autobuilding and self.wheel_cache.cache_dir
        )
        assert building_is_possible
@burkestar

This comment has been minimized.

Copy link

burkestar commented Jan 22, 2019

#5884 looks like this is a related change that could have caused this?

@burkestar

This comment has been minimized.

Copy link

burkestar commented Jan 22, 2019

Seems like the pip maintainers should rollback the recent 19 release to address this breaking change?
19.0 release notes: https://github.com/pypa/pip/blob/master/NEWS.rst#190-2019-01-22

UPDATE: not trying to cast aspersions here, was just suggesting as one way to quickly unblock people affected by this seeing as the release had just happened. Rolling forward with a hotfix works too. I appreciate the hard work of the community that supports this mission critical tooling, and agree with the sentiments below about postmortems to learn from mistakes and prevent future issues. Meanwhile, we are doing the same internally, which means a liberal amount of hardpinning pip versions in all the places :)

@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Jan 22, 2019

The PR adding the TODO comment also has this comment in reply: https://github.com/pypa/pip/pull/5743/files#r215832743

Based on that comment and also the commenter above saying that passing PYTHONOPTIMIZE=1 makes the error go away, it seems like simply removing the assertion might be the correct fix (independent of the question of rolling back).

@tgs

This comment has been minimized.

Copy link
Contributor

tgs commented Jan 22, 2019

Yeah, when I delete that assert, packages do install fine with --no-cache-dir. In that case, it says it's Running setup.py install instead of Building wheel for sdist packages.

@matthewfeickert

This comment has been minimized.

Copy link

matthewfeickert commented Jan 22, 2019

This is also happening to my projects. I can reproduce this in Docker images built FROM ubuntu:bionic and FROM centos:centos7 where I am installing Python 3 from source (here is a Gist that demonstrates a failing example for both of those Docker images in the event that it might be helpful). For the requirements.txt in the example in the Gist and

$ pip3 install --upgrade pip setuptools wheel
Requirement already up-to-date: pip in /usr/lib/python3.6/site-packages (19.0)
Requirement already up-to-date: setuptools in /usr/lib/python3.6/site-packages (40.6.3)
Requirement already up-to-date: wheel in /usr/lib/python3.6/site-packages (0.32.3)

then

$ pip3 install --upgrade --no-cache-dir -r requirements.txt

fails with

Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/usr/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/usr/lib/python3.6/site-packages/pip/_internal/wheel.py", line 848, in build
    assert building_is_possible
AssertionError

but

$ pip3 install --upgrade -r requirements.txt

works fine as expected.

@asottile

This comment has been minimized.

Copy link
Contributor

asottile commented Jan 22, 2019

I'm particularly hitting this with tox + docker + ENV PIP_NO_CACHE_DIR=off

My workaround is to use tox-virtualenv-no-download plugin to prevent pip from auto-updating

@coderanger

This comment has been minimized.

Copy link

coderanger commented Jan 23, 2019

We also have --no-cache-dir in our installs inside Docker to keep the images small. Our workaround has been --cache-dir=/pipcache and then rm -rf /pipcache in the same RUN step so it never ends up in the image.

marstr added a commit to Azure/azure-cli that referenced this issue Jan 23, 2019

@zehauser

This comment has been minimized.

Copy link

zehauser commented Jan 23, 2019

Software development is hard and bugs like this are always going to happen. Certainly nobody should blame the pip maintainers or contributors for this incident.

However, I would suggest that this bug merits some sort of post-mortem analysis on the part of the pip team, due to the number of (missed) opportunities for this bug to have been caught before slipping into a general release. For example:

  • automated testing of core functionality like --no-cache-dir
  • pre-commit, pre-merge, or pre-release checks that flag (or prohibit) TODOs
  • a (human) pre-merge review of all unresolved review comments in the PR (Github autominimizes most review comment threads when their associated code has been changed, and as of recently lets you manually mark review comment threads as resolved)
  • changes in the release process — why not release a beta first and then wait several weeks before a general release?
  • etc

A postmortem could result in a number of helpful betterments to ensure that software that is as core to the Python project as pip doesn't ship with bugs of this magnitude in future.

@pradyunsg pradyunsg added the type: bug label Jan 23, 2019

@erinxocon

This comment has been minimized.

Copy link

erinxocon commented Jan 23, 2019

I can replicate this bug. Removing --no-cache-dir fixes it. Since I don't want it in my docker image I'm using the solution @coderanger proposed. Cheers 🌈🍰🌈

@matthewfeickert

This comment has been minimized.

Copy link

matthewfeickert commented Jan 23, 2019

@MrAtheist You have a small typo/missing a decimal. The patch release is 19.0.1 but you have 19.01 written.

@MrAtheist

This comment has been minimized.

Copy link

MrAtheist commented Jan 23, 2019

oops my mistake, but either way, the possible versions dont have 19.0.1 listed... ¯_(ツ)_/¯

@cmebarrow

This comment has been minimized.

Copy link

cmebarrow commented Jan 23, 2019

Like @dmulter I am finding the issue still unresolved. Extract from build output:

. venv/bin/activate;  python -m pip install --upgrade pip; python -m pip install ndg_httpsclient; python -m pip install . -i https://xxxx.yyyy.com/simple --upgrade --no-cache-dir flask
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: pip in ./venv/lib/python2.7/site-packages (19.0.1)
...
Requirement already satisfied, skipping upgrade: pycparser in ./venv/lib/python2.7/site-packages (from cffi>=1.1->bcrypt>=3.1.3->paramiko<3.0,>=1.10->Fabric==1.14.0->conference-gll-load-test===0.0.1-SNAPSHOT) (2.19)
Exception:
Traceback (most recent call last):
  File "/mnt/jenkins/workspace/venv/local/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 176, in main
    status = self.run(options, args)
  File "/mnt/jenkins/workspace/venv/local/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 346, in run
    session=session, autobuilding=True
  File "/mnt/jenkins/workspace/venv/local/lib/python2.7/site-packages/pip/_internal/wheel.py", line 886, in build
    assert have_directory_for_build
AssertionError
make: *** [install] Error 2
@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Jan 23, 2019

Earlier in the thread I had asked if including --no-use-pep517 along with --no-cache-dir makes things work for people, but I didn't see a reply. For people that are still experiencing the option, can you try that?

@dmulter

This comment has been minimized.

Copy link

dmulter commented Jan 23, 2019

Adding the --no-use-pep517 option fixed the problem for me. Hope that helps narrow things down.

@Estartu

This comment has been minimized.

Copy link

Estartu commented Jan 24, 2019

pip 19.0.1 working for me in a virtualenv. But inside Jenkins (Shining Panda) it still fails. Adding --no-use-pep517 fixes the problem

@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Jan 24, 2019

I'm reopening since some people are still experiencing the same issue.

@cjerdonek cjerdonek reopened this Jan 24, 2019

rjw57 added a commit to uisautomation/dockerimages that referenced this issue Jan 24, 2019

python: pin pip version to <19
pip version 19 has a bug[1] which is breaking our builds. Until it's
fixed, pin pip to a pre-19 version.

[1] pypa/pip#6158

pradyunsg referenced this issue Jan 24, 2019

Check wheel build locations only when ephem cache isn't used
When the ephemeral cache is used, the build can always occur. There is
no need to check for those.
@rifelpet

This comment has been minimized.

Copy link

rifelpet commented Jan 24, 2019

I can also confirm that --no-use-pep517 fixed the issue for me after upgrading to pip 19.0.1 did not.

@hrw

This comment was marked as off-topic.

Copy link

hrw commented Jan 24, 2019

But why all those projects have to adapt whenever pip gets new version?

@cjerdonek

This comment has been minimized.

Copy link
Member

cjerdonek commented Jan 24, 2019

At @pradyunsg's request, I've opened a new issue (#6197) specific to the AssertionError in the 19.0.1 release, as it's narrower in scope and will need new investigation. So I'm reclosing this issue.

@cjerdonek cjerdonek closed this Jan 24, 2019

@pcrespov pcrespov referenced this issue Jan 28, 2019

Merged

Is539/master build fails #540

5 of 5 tasks complete

HeavenVolkoff added a commit to HeavenVolkoff/foundation that referenced this issue Jan 28, 2019

Update setup.cfg
Dont use pip 19.0 due to [Issue 6158](pypa/pip#6158)
@marcstreeter

This comment has been minimized.

Copy link

marcstreeter commented Jan 29, 2019

Ran into the same issue. Ended up pinning the pip version as a fix for now.

pip install --upgrade pip==18.1

or your FROM python:3.6-alpine can change to FROM python:3.6.7-alpine

boogheta added a commit to medialab/hyphe that referenced this issue Jan 30, 2019

one more attempt to fix Docker builds
they are failing because pycparser has issues with --no-cache-dir
solution picked from pypa/pip#6158 (comment)

seblucas added a commit to seblucas/alpine-homeassistant that referenced this issue Jan 31, 2019

crazy-max added a commit to crazy-max/docker-firefox-syncserver that referenced this issue Feb 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment