Installing from VCS fails (egg mismatch / "No such file or directory")

[K377U]

• Pip version: 10.0.0
• Python version: Python 2.7.12
• Operating system: Ubuntu 16.04

Description:

I'm doing pip install for VCS

pip 9.0.3

$ virtualenv env
New python executable in .../env/bin/python
Installing setuptools, pip, wheel...done.

$ . env/bin/activate

$ pip install 'pip==9.0.3' --upgrade
Collecting pip==9.0.3
  Using cached https://files.pythonhosted.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 10.0.0
    Uninstalling pip-10.0.0:
      Successfully uninstalled pip-10.0.0
Successfully installed pip-9.0.3
You are using pip version 9.0.3, however version 10.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$ pip install git+https://github.com/requests/requests.git#egg=requests-dev
Collecting requests-dev from git+https://github.com/requests/requests.git#egg=requests-dev
  Cloning https://github.com/requests/requests.git to /tmp/pip-build-MKrJMy/requests-dev
  Running setup.py (path:/tmp/pip-build-MKrJMy/requests-dev/setup.py) egg_info for package requests-dev produced metadata for project name requests. Fix your #egg=requests-dev fragments.
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 1.9MB/s 
Collecting idna<2.7,>=2.5 (from requests)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 2.6MB/s 
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)
    100% |████████████████████████████████| 133kB 1.7MB/s 
Collecting certifi>=2017.4.17 (from requests)
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl (150kB)
    100% |████████████████████████████████| 153kB 2.1MB/s 
Installing collected packages: chardet, idna, urllib3, certifi, requests
  Running setup.py install for requests ... done
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
You are using pip version 9.0.3, however version 10.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
certifi (2018.4.16)
chardet (3.0.4)
idna (2.6)
pip (9.0.3)
requests (2.18.4)
setuptools (39.0.1)
urllib3 (1.22)
wheel (0.31.0)
You are using pip version 9.0.3, however version 10.0.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Works as expected.

pip 10.0.0

$ virtualenv env
New python executable in .../env/bin/python
Installing setuptools, pip, wheel...done.

$ . env/bin/activate

$ pip install pip --upgrade
Requirement already up-to-date: pip in ./env/lib/python2.7/site-packages (10.0.0)

$ pip list
Package    Version
---------- -------
pip        10.0.0 
setuptools 39.0.1 
wheel      0.31.0 
$ pip install git+https://github.com/requests/requests.git#egg=requests-dev
Collecting requests-dev from git+https://github.com/requests/requests.git#egg=requests-dev
  Cloning https://github.com/requests/requests.git to /tmp/pip-install-9Mc3ed/requests-dev
  Running setup.py (path:/tmp/pip-install-9Mc3ed/requests-dev/setup.py) egg_info for package requests-dev produced metadata for project name requests. Fix your #egg=requests-dev fragments.
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl
Building wheels for collected packages: requests, requests
  Running setup.py bdist_wheel for requests ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-vRGLBg/wheels/8d/84/fa/1ded919cc5b46b899830a8f8d18817f59abde627aa07362b23
  Running setup.py bdist_wheel for requests ... error
  Complete output from command .../env/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-9Mc3ed/requests/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-2znDz7 --python-tag cp27:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-9Mc3ed/requests/setup.py'
  
  ----------------------------------------
  Failed building wheel for requests
  Running setup.py clean for requests
  Complete output from command .../env/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-9Mc3ed/requests/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-9Mc3ed/requests/setup.py'
  
  ----------------------------------------
  Failed cleaning build dir for requests
Successfully built requests
Failed to build requests
Installing collected packages: chardet, idna, urllib3, certifi, requests
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22

$ pip list
Package    Version  
---------- ---------
certifi    2018.4.16
chardet    3.0.4    
idna       2.6      
pip        10.0.0   
requests   2.18.4   
setuptools 39.0.1   
urllib3    1.22     
wheel      0.31.0 

Manages to install but shows errrors.

I did some debug and found out the problem is that pip is duplicated in requirementset.

<RequirementSet object; 2 requirement(s): requests, requests>

Narrowed down the problem to:
Resolver.resolve()
Resolver._resolve_one()

From there I managed to pinpoint possible bug in RequirementSet.has_requirement(). It does not find the
duplicate.

This could also be a problem where the requirementset has requests-dev in the dictionary. In older version of pip this requests-dev was used.

[pradyunsg]

This is a duplicate of #5251. This should be fixed with pip 10.0.1. If it's not, feel free to comment there.

[K377U]

I tried with the current master and still fails. Assuming(looks like it was) that the actual fix is merged already.

$ virtualenv env
New python executable in .../env/bin/python
Installing setuptools, pip, wheel...done.

$ . env/bin/activate

$ pip install git+https://github.com/pypa/pip.git#egg=pip-dev --upgrade
Collecting pip-dev from git+https://github.com/pypa/pip.git#egg=pip-dev
  Cloning https://github.com/pypa/pip.git to /tmp/pip-install-dJslXN/pip-dev
  Installing build dependencies ... done
  Running setup.py (path:/tmp/pip-install-dJslXN/pip-dev/setup.py) egg_info for package pip-dev produced metadata for project name pip. Fix your #egg=pip-dev fragments.
Building wheels for collected packages: pip, pip
  Running setup.py bdist_wheel for pip ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-m6I4nU/wheels/26/59/17/819f95cd36eb6f74513dfdcdceea3d4e27a102a2a41d241c57
  Running setup.py bdist_wheel for pip ... error
  Complete output from command .../env/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-dJslXN/pip/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-Q8H4Wg --python-tag cp27:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-dJslXN/pip/setup.py'
  
  ----------------------------------------
  Failed building wheel for pip
  Running setup.py clean for pip
  Complete output from command .../env/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-dJslXN/pip/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-dJslXN/pip/setup.py'
  
  ----------------------------------------
  Failed cleaning build dir for pip
Successfully built pip
Failed to build pip
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-10.0.1

$ pip list
Package    Version
---------- -------
pip        10.0.1 
setuptools 39.0.1 
wheel      0.31.0 

$ pip install git+https://github.com/requests/requests.git#egg=requests-dev
Collecting requests-dev from git+https://github.com/requests/requests.git#egg=requests-dev
  Cloning https://github.com/requests/requests.git to /tmp/pip-install-VkI3dM/requests-dev
  Running setup.py (path:/tmp/pip-install-VkI3dM/requests-dev/setup.py) egg_info for package requests-dev produced metadata for project name requests. Fix your #egg=requests-dev fragments.
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.7,>=2.5 (from requests)
  Using cached https://files.pythonhosted.org/packages/27/cc/6dd9a3869f15c2edfab863b992838277279ce92663d334df9ecf5106f5c6/idna-2.6-py2.py3-none-any.whl
Collecting urllib3<1.23,>=1.21.1 (from requests)
  Using cached https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests)
  Using cached https://files.pythonhosted.org/packages/7c/e6/92ad559b7192d846975fc916b65f667c7b8c3a32bea7372340bfe9a15fa5/certifi-2018.4.16-py2.py3-none-any.whl
Building wheels for collected packages: requests, requests
  Running setup.py bdist_wheel for requests ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-QAiEkP/wheels/8d/84/fa/1ded919cc5b46b899830a8f8d18817f59abde627aa07362b23
  Running setup.py bdist_wheel for requests ... error
  Complete output from command .../env/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-VkI3dM/requests/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-6xK1ut --python-tag cp27:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-VkI3dM/requests/setup.py'
  
  ----------------------------------------
  Failed building wheel for requests
  Running setup.py clean for requests
  Complete output from command .../env/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-VkI3dM/requests/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  IOError: [Errno 2] No such file or directory: '/tmp/pip-install-VkI3dM/requests/setup.py'
  
  ----------------------------------------
  Failed cleaning build dir for requests
Successfully built requests
Failed to build requests
Installing collected packages: chardet, idna, urllib3, certifi, requests
Successfully installed certifi-2018.4.16 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22

$ pip list
Package    Version  
---------- ---------
certifi    2018.4.16
chardet    3.0.4    
idna       2.6      
pip        10.0.1   
requests   2.18.4   
setuptools 39.0.1   
urllib3    1.22     
wheel      0.31.0 

[pradyunsg]

Don't include the -dev suffix on the #egg fragment.

pip install git+https://github.com/requests/requests.git#egg=requests

[pradyunsg]

That said, I think pip is supposed to be printing an error message/warning here.

[K377U]

Actually my real problem is a little different. I just managed to reproduce the problem with simple example.

This worked before in 9.0.3 as you can see from the example above. Is this feature removed now?

We are using dependency_links in setup.py. This might be in our own code so I will check it. But before we could not get dependency links work without the version number in the egg.

[K377U]

I can confirm that 10.0.1 made everything work without the -versionumber part. And that is no longer a problem for us. Thank you!

This is still relevant question and good to know:

This worked before in 9.0.3 as you can see from the example above. Is this feature removed now?

[pradyunsg]

Is this feature removed now?

Off the top of my head, I don't think it was ever a feature -- I guess this is an "undocumented thing that works and people depend on it".

[pradyunsg]

TBH, the current behavior is the correct one IMO.

[cjerdonek]

I believe this is a duplicate of #5271.