Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip 10 is unable to install from non-editable VCS URLs #5251

Closed
RazerM opened this issue Apr 16, 2018 · 30 comments · Fixed by #5280
Closed

pip 10 is unable to install from non-editable VCS URLs #5251

RazerM opened this issue Apr 16, 2018 · 30 comments · Fixed by #5280
Assignees
Labels
auto-locked Outdated issues that have been locked by automation type: bug A confirmed bug or unintended behavior
Milestone

Comments

@RazerM
Copy link

RazerM commented Apr 16, 2018

  • Pip version: 10.0.0
  • Python version: 3.5.4
  • Operating system: Linux

Description:

I'm trying to install a package from Git.

What I've run:

$ pip3 install git+ssh://git@example.com/mypackage.git#egg=mypackage
Building wheels for collected packages: mypackage
Exception:
Traceback (most recent call last):
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/commands/install.py", line 305, in run
    session=session, autobuilding=True
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 773, in build
    python_tag=python_tag,
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 633, in _build_one
    python_tag=python_tag)
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 637, in _build_one_inside_env
    if self.__build_one(req, temp_dir.path, python_tag=python_tag):
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 663, in __build_one
    base_args = self._base_setup_args(req)
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/wheel.py", line 659, in _base_setup_args
    SETUPTOOLS_SHIM % req.setup_py
  File "/home/p/local/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 416, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for mypackage from git+ssh://git@example.com/mypackage.git#egg=mypackage in /home/p/local/lib/python3.5/site-packages

If I uninstall the package and then install, it works. But it continues to fail when trying install again.

@bruceduhamel
Copy link

bruceduhamel commented Apr 16, 2018

I saw this same error on Python 2 and found that using pip 9 worked as expected. I wonder if this is related to the deprecation and removal of the egg switch? Removing that also seemed to solve the issue.

@mmulich
Copy link

mmulich commented Apr 16, 2018

I'm experiencing this issue on Travis-CI with Python 3.5.2 and pip==10.0.0. I do not experience it with Python 3.6 (local) or 2.7 (both local and on travis).

Traceback (most recent call last):
...
  File "/home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages/pip/_internal/req/req_install.py", line 415, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for cnx-query-grammar from git+https://github.com/Connexions/cnx-query-grammar.git#egg=cnx-query-grammar in /home/travis/virtualenv/python3.5.2/lib/python3.5/site-packages (from -r requirements/./test.txt (line 9))

@makmanalp
Copy link

Exact same issue, on Ubuntu 14.04.2 LTS, pip 10.0.0 installed with get-pip.py on python 3.4.0 specifically on my git installs.

@makmanalp
Copy link

Also using the #egg= parameter. I found this thread: #1749

@makmanalp
Copy link

Oddly removing the whole virtualenv and reinstalling (including the get-pip bit) worked.

@makmanalp
Copy link

... but it failed again on the second deploy on top of an existing virtualenv (which should have been a no-op since all the packages are installed already but I guess it redownloads the git packages each time).

@mmulich
Copy link

mmulich commented Apr 16, 2018

Initial installation works, but attempts to reinstall are the problem space. So far I'm seeing a difference in the InstallRequirement's build_env value, where the initial install produced a NoOpBuildEnvironment object and a reinstall has a BuildEnvironment.

@JordanSlaman
Copy link

Also experiencing this issue with pip 10.0.0
This breaks our ops/build workflows. 😢

@mmulich
Copy link

mmulich commented Apr 16, 2018

In case others are also debugging this. I've narrowed down the two paths, first install and reinstall, to the difference in the results coming out of the resolver's _get_abstract_dist_for. Prior to that call both paths have <InstallRequirement>.source_dir == None. After this call, only the first install path has a source_dir set to something other than None. I'm guessing this is because it needs to go off and fetch the source during the first-install, but not on the reinstall.

... I could be off in the weeds though. It is my first time looking at this code. 😕

@makmanalp
Copy link

My workaround: pip install pip==9.0.3

@mmulich
Copy link

mmulich commented Apr 16, 2018

O... so the requirements file I'm working on isn't using -e prefixes for vcs installs. Adding -e to the beginning of the line does make this work. And in @RazerM 's case: pip3 install -e git+ssh://git@example.com/mypackage.git#egg=mypackage

So part of this issue is user error??? The other part of it is that we should get back something to the effect of Requirement already satisfied: ... and instead we get a Traceback.

@makmanalp
Copy link

If it helps, mine also doesn't use -e, though I thought VCS package URLs needn't be editable.

@RazerM
Copy link
Author

RazerM commented Apr 17, 2018

Correct, they needn't be:

VCS projects can be installed in editable mode (using the --editable option) or not.

  • For editable installs, the clone location by default is "<venv path>/src/SomeProject" in virtual environments, and "<cwd>/src/SomeProject" for global installs. The --src option can be used to modify this location.
  • For non-editable installs, the project is built locally in a temp dir and then normally. Note that if a satisfactory version of the package is already installed, the VCS source will not overwrite it without an --upgrade flag. VCS requirements pin the package version (specified in the setup.py file) of the target commit, not necessarily the commit itself.
  • The pip freeze subcommand will record the VCS requirement specifier (referencing a specific commit) if and only if the install is done using the editable option.

@techalchemy
Copy link
Member

FYI we are running into this in pipenv (our testing infrastructure caught this), here's what I can say on the topic:

  • This only impacts non-editable VCS installs (i.e. not passing -e
  • We explicitly pass a PIP_SRC_DIR during testing and it seems to make no difference
  • All of our tests run in isolated environments

As far as the distinction between editable and non-editable dependencies, from a resolution standpoint we can resolve dependency graphs for editable dependencies but can't for non-editable dependencies. Still-- we test to ensure that they can be installed, which now fails.

As a temporary fix we are moving our pip install calls to python -m <vendored pip9> install for compatibility.

@pradyunsg pradyunsg changed the title No source dir for <package> pip 10 is unable to install from non-editable VCS URLs Apr 17, 2018
@pradyunsg pradyunsg added the type: bug A confirmed bug or unintended behavior label Apr 17, 2018
@pradyunsg pradyunsg self-assigned this Apr 17, 2018
@pradyunsg
Copy link
Member

I am unable to reproduce this. Could someone provide steps to reproduce? If it's with in a docker container, that'd be awesome.

@eely22
Copy link

eely22 commented Apr 17, 2018

Just run this twice with pip 10 in a clean venv, it will fail on the second try:

pip install git+https://github.com/requests/requests.git#egg=requests

@di
Copy link
Sponsor Member

di commented Apr 17, 2018

I can't reproduce:

/tmp $ python -m venv env
/tmp $ source env/bin/activate
(env) /tmp $ pip install -U pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl (1.3MB)
    100% |████████████████████████████████| 1.3MB 854kB/s
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-10.0.0
(env) /tmp $ pip install git+https://github.com/requests/requests.git#egg=requests
Collecting requests from git+https://github.com/requests/requests.git#egg=requests
  Cloning https://github.com/requests/requests.git to /private/var/folders/px/dlfyqxyx3mx0t_yx3dvbb9nc0000gn/T/pip-install-32abmlma/requests
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 6.9MB/s
Collecting idna<2.7,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 9.3MB/s
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 10.9MB/s
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 10.3MB/s
Installing collected packages: chardet, idna, urllib3, certifi, requests
  Running setup.py install for requests ... done
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
(env) /tmp $ pip install git+https://github.com/requests/requests.git#egg=requests
Requirement already satisfied: requests from git+https://github.com/requests/requests.git#egg=requests in ./env/lib/python3.6/site-packages (2.18.4)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./env/lib/python3.6/site-packages (from requests) (3.0.4)
Requirement already satisfied: idna<2.7,>=2.5 in ./env/lib/python3.6/site-packages (from requests) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in ./env/lib/python3.6/site-packages (from requests) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.6/site-packages (from requests) (2018.4.16)

@techalchemy
Copy link
Member

@di @pradyunsg first spotted: https://travis-ci.org/pypa/pipenv/jobs/367322542#L715

log:

 ◰³ tempenv-3321664538cc  ~/g/pypa-pipenv   tmp $…  pip install --no-cache-dir pip
Requirement already satisfied: pip in /home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages (10.0.0)
 ◰³ tempenv-3321664538cc  ~/g/pypa-pipenv   tmp $…  pip install git+https://github.com/benjaminp/six.git#egg=six
Collecting six from git+https://github.com/benjaminp/six.git#egg=six
  Cloning https://github.com/benjaminp/six.git to /tmp/pip-install-2ff500pr/six
Building wheels for collected packages: six
  Running setup.py bdist_wheel for six ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-ebgi9ij6/wheels/a9/36/0a/8173089e25b3c1df51c0bb21d21ec8dc5d7ee9d0ec9b6b51e1
Successfully built six
Installing collected packages: six
Successfully installed six-1.11.0
 ◰³ tempenv-3321664538cc  ~/g/pypa-pipenv   tmp $…  pip install git+https://github.com/benjaminp/six.git#egg=six
Requirement already satisfied: six from git+https://github.com/benjaminp/six.git#egg=six in /home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages (1.11.0)
Building wheels for collected packages: six
Exception:
Traceback (most recent call last):
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 305, in run
    session=session, autobuilding=True
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 773, in build
    python_tag=python_tag,
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 633, in _build_one
    python_tag=python_tag)
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 637, in _build_one_inside_env
    if self.__build_one(req, temp_dir.path, python_tag=python_tag):
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 663, in __build_one
    base_args = self._base_setup_args(req)
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/wheel.py", line 659, in _base_setup_args
    SETUPTOOLS_SHIM % req.setup_py
  File "/home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 415, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for six from git+https://github.com/benjaminp/six.git#egg=six in /home/hawk/.virtualenvs/tempenv-3321664538cc/lib/python3.6/site-packages

@techalchemy
Copy link
Member

And a Dockerfile which reproduces:

FROM python:3.6-alpine3.7

ENV PYTHONUNBUFFERED=1
ENV LC_ALL=en_US.UTF-8

RUN apk update \
    && apk add git

WORKDIR /app/


RUN set -ex \
    && pip install --upgrade pip \
    && pip install git+https://github.com/benjaminp/six.git#egg=six \
    && pip install git+https://github.com/benjaminp/six.git#egg=six

CMD ["/bin/bash"]

@eely22
Copy link

eely22 commented Apr 17, 2018

We can reproduce on our build machines, try this:

docker run -it circleci/python:2.7.13
cd ~
virtualenv env
. env/bin/activate
pip install -U pip
pip install git+https://github.com/requests/requests.git#egg=requests
pip install git+https://github.com/requests/requests.git#egg=requests

Here's my output:

ericely$ docker run -it circleci/python:2.7.13
$ cd ~
$ virtualenv env
New python executable in /home/circleci/env/bin/python
Installing setuptools, pip, wheel...
done.
$ $ . env/bin/activate
(env) $ pip install -U pip
Requirement already up-to-date: pip in ./env/lib/python2.7/site-packages (10.0.0)
(env) $ pip list
Package    Version
---------- -------
pip        10.0.0 
setuptools 39.0.1 
wheel      0.31.0 
(env) $ pip install git+https://github.com/requests/requests.git#egg=requests
Collecting requests from git+https://github.com/requests/requests.git#egg=requests
  Cloning https://github.com/requests/requests.git to /tmp/pip-install-I8n5LX/requests
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 3.9MB/s 
Collecting idna<2.7,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 7.9MB/s 
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 8.0MB/s 
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 4.3MB/s 
Building wheels for collected packages: requests
  Running setup.py bdist_wheel for requests ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-F7Eu9c/wheels/8d/84/fa/1ded919cc5b46b899830a8f8d18817f59abde627aa07362b23
Successfully built requests
Installing collected packages: chardet, idna, urllib3, certifi, requests
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
(env) $ 
(env) $ pip install git+https://github.com/requests/requests.git#egg=requests
Requirement already satisfied: requests from git+https://github.com/requests/requests.git#egg=requests in ./env/lib/python2.7/site-packages (2.18.4)
Requirement already satisfied: idna<2.7,>=2.5 in ./env/lib/python2.7/site-packages (from requests) (2.6)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in ./env/lib/python2.7/site-packages (from requests) (1.22)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python2.7/site-packages (from requests) (2018.4.16)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./env/lib/python2.7/site-packages (from requests) (3.0.4)
Building wheels for collected packages: requests
Exception:
Traceback (most recent call last):
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/basecommand.py", line 228, in main
    status = self.run(options, args)
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 305, in run
    session=session, autobuilding=True
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 773, in build
    python_tag=python_tag,
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 633, in _build_one
    python_tag=python_tag)
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 637, in _build_one_inside_env
    if self.__build_one(req, temp_dir.path, python_tag=python_tag):
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 663, in __build_one
    base_args = self._base_setup_args(req)
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/wheel.py", line 659, in _base_setup_args
    SETUPTOOLS_SHIM % req.setup_py
  File "/home/circleci/env/lib/python2.7/site-packages/pip/_internal/req/req_install.py", line 415, in setup_py
    assert self.source_dir, "No source dir for %s" % self
AssertionError: No source dir for requests from git+https://github.com/requests/requests.git#egg=requests in ./env/lib/python2.7/site-packages
(env) $ 

@pradyunsg pradyunsg added the !release blocker Hold a release until this is resolved label Apr 17, 2018
@pradyunsg pradyunsg added this to the 10.0.1 milestone Apr 17, 2018
@pradyunsg
Copy link
Member

@pfmoore This is an important core use-case which is broken. I'm looking into a fix right now.

@bbakersmith
Copy link

Also experiencing this issue

@pfmoore
Copy link
Member

pfmoore commented Apr 17, 2018

@pradyunsg Cool, keep me posted - if there's anything I can help with let me know.

@pradyunsg
Copy link
Member

If someone could take #5280 for a spin and check if that fixes this issue, that would be awesome! :)

@bbakersmith
Copy link

@pradyunsg I didn't do extensive testing, but #5280 resolves the issue for our case.

@techalchemy
Copy link
Member

Thanks for the quick turnaround @pradyunsg — perhaps someone else can whip up a test case

Haven’t tested yet but can in the morning. What is the planned release timeline for pip 10.0.1? If we can avoid hacking around this in our point release I’d prefer to stick to our normal subprocess invocation of pip install but we have some bugfixes in the pipeline so I’m trying to assess whether to hold ours back for this— guidance appreciated!

/cc @pfmoore

@pradyunsg
Copy link
Member

What is the planned release timeline for pip 10.0.1?

It'll happen this week, in all likelihood. :)

@pfmoore
Copy link
Member

pfmoore commented Apr 18, 2018

Correct - basically once all the issues/PRs marked as release blockers are resolved, I'll be doing a final pass of the issues and then releasing 10.0.1.

@RazerM
Copy link
Author

RazerM commented Apr 18, 2018

#5280 works for me too

@lock
Copy link

lock bot commented May 29, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label May 29, 2019
@lock lock bot locked as resolved and limited conversation to collaborators May 29, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation type: bug A confirmed bug or unintended behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.