Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pip sometimes prohibits 2 .dist-info directories in wheels #7487

Closed
chrahunt opened this issue Dec 15, 2019 · 5 comments · Fixed by #7494
Closed

pip sometimes prohibits 2 .dist-info directories in wheels #7487

chrahunt opened this issue Dec 15, 2019 · 5 comments · Fixed by #7494
Labels
auto-locked Outdated issues that have been locked by automation state: awaiting PR Feature discussed, PR is needed type: feature request Request for a new feature

Comments

@chrahunt
Copy link
Member

Environment

  • pip version: 19.3.1
  • Python version: 3.8.0
  • OS: Linux

Description

Currently, pip rejects wheel files containing multiple .dist-info directories, but only if they start with the name of the package being installed.

Expected behavior

pip should prohibit more than 1 top-level .dist-info directory, without regard to the name of the directories (besides ending with .dist-info).

From PEP 427:

A wheel is a ZIP-format archive with a specially formatted file name and the .whl extension. It contains a single distribution

And from PEP 376:

One .dist-info directory per installed distribution

How to Reproduce

Save t.sh below, make it executable, and execute it.

t.sh
#!/bin/sh
cd "$(mktemp -d)"
python -m venv env
. env/bin/activate
pip install --upgrade pip wheel

echo "= Versions ====================================================================="
python -V
pip -V
wheel version

echo "= Setup ========================================================================"
echo "from setuptools import setup; setup(name='hello')" > setup.py

echo "= (1) Base case ================================================================"
echo "= (1) Build ===================================================================="
python setup.py bdist_wheel

echo "= (1) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

echo "= (2) 2 .dist-info dirs (expected) ============================================="
echo "= (2) Build ===================================================================="
python setup.py bdist_wheel

echo "= (2) Add second .dist-info (shared prefix) ===================================="
mkdir hello-example-0.0.0.dist-info
zip dist/hello-0.0.0-py3-none-any.whl hello-example-0.0.0.dist-info

echo "= (2) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

echo "= (3) 2 .dist-info dirs (NOT expected) ========================================="
echo "= (3) Build ===================================================================="
python setup.py bdist_wheel

echo "= (3) Add second .dist-info (no shared prefix) ================================="
mkdir goodbye-example-0.0.0.dist-info
zip dist/hello-0.0.0-py3-none-any.whl goodbye-example-0.0.0.dist-info

echo "= (3) Install =================================================================="
pip install --ignore-installed dist/hello-0.0.0-py3-none-any.whl

Output

Output
Collecting pip
  Using cached https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Installing collected packages: pip, wheel
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3.1 wheel-0.33.6
= Versions =====================================================================
Python 3.8.0
pip 19.3.1 from /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip (python 3.8)
wheel 0.33.6
= Setup ========================================================================
= (1) Base case ================================================================
= (1) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
creating hello.egg-info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
writing manifest file 'hello.egg-info/SOURCES.txt'
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (1) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
Successfully installed hello-0.0.0
= (2) 2 .dist-info dirs (expected) =============================================
= (2) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (2) Add second .dist-info (shared prefix) ====================================
  adding: hello-example-0.0.0.dist-info/ (stored 0%)
= (2) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 153, in _main
    status = self.run(options, args)
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 446, in run
    installed = install_given_reqs(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/__init__.py", line 58, in install_given_reqs
    requirement.install(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 858, in install
    self.move_wheel_files(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/req/req_install.py", line 487, in move_wheel_files
    wheel.move_wheel_files(
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/wheel.py", line 461, in move_wheel_files
    clobber(source, lib_dir, True)
  File "/tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/pip/_internal/wheel.py", line 408, in clobber
    assert not info_dir, ('Multiple .dist-info directories: ' +
AssertionError: Multiple .dist-info directories: /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/hello-example-0.0.0.dist-info, /tmp/user/1000/tmp.bXrdYptdl9/env/lib/python3.8/site-packages/hello-0.0.0.dist-info
= (3) 2 .dist-info dirs (NOT expected) =========================================
= (3) Build ====================================================================
running bdist_wheel
running build
installing to build/bdist.linux-x86_64/wheel
running install
running install_egg_info
running egg_info
writing hello.egg-info/PKG-INFO
writing dependency_links to hello.egg-info/dependency_links.txt
writing top-level names to hello.egg-info/top_level.txt
reading manifest file 'hello.egg-info/SOURCES.txt'
writing manifest file 'hello.egg-info/SOURCES.txt'
Copying hello.egg-info to build/bdist.linux-x86_64/wheel/hello-0.0.0-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/hello-0.0.0.dist-info/WHEEL
creating 'dist/hello-0.0.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'hello-0.0.0.dist-info/METADATA'
adding 'hello-0.0.0.dist-info/WHEEL'
adding 'hello-0.0.0.dist-info/top_level.txt'
adding 'hello-0.0.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
= (3) Add second .dist-info (no shared prefix) =================================
  adding: goodbye-example-0.0.0.dist-info/ (stored 0%)
= (3) Install ==================================================================
Processing ./dist/hello-0.0.0-py3-none-any.whl
Installing collected packages: hello
Successfully installed hello-0.0.0

The unexpected case is (3) Install - the installation should not go through since there are multiple .dist-info directories.

@triage-new-issues triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Dec 15, 2019
@uranusjr
Copy link
Member

Does pip allow multiple dist-info dirs (in general) to handle installing from flat directory? (It’s the case IIRC.) I kind of feel the responsibility to produce a valid wheel should fall on to the wheel builder (so build backend).

pip can be free to interpret and invalid wheel and do what’s most comfortable (e.g. choose one by whatever logic, as the current behaviour) since the input already breaks the contract. It is also a valid behaviour to error out as well, but I wouldn’t say it’s necessary.

@pradyunsg
Copy link
Member

I think erroring out results in the best end-user experience here -- you gave pip something invalid, so it did the wrong thing vs you gave pip something invalid and it refused to use it.

@pradyunsg pradyunsg added state: awaiting PR Feature discussed, PR is needed type: feature request Request for a new feature and removed S: needs triage Issues/PRs that need to be triaged labels Dec 15, 2019
@chrahunt
Copy link
Member Author

Does pip allow multiple dist-info dirs (in general) to handle installing from flat directory?

Good question. The current behavior is here - we essentially guess. I think this use case is different from installing a wheel because the wheel format has a spec and we control everything about the directory it is unpacked into.

I kind of feel the responsibility to produce a valid wheel should fall on to the wheel builder (so build backend).

Agreed.

pip can be free to interpret and invalid wheel and do what’s most comfortable

I am trying to refactor some of our wheel installation logic. The purpose of calling out the expected behavior here is to make it easier to make things more comfortable. :)

Our current validation approach has lead to situations like this one, but extracting it while preserving the exact same behavior would result in code that is harder to follow and doesn't really map to the spec. I expect taking the stricter approach will lead to fewer "print better errors" issues, since we can print a nice error upfront, at least in this case.

@uranusjr
Copy link
Member

I’m totally on board if the behaviour change would improve the code quality.

@chrahunt
Copy link
Member Author

chrahunt commented Dec 16, 2019

I checked all wheels associated with the most recent release of all projects on PyPI and it looks like only intel-tensorflow would be affected by this change. I couldn't find any good contact information for that project after a few minutes of looking, since it just copies the existing tensorflow metadata. Maybe https://github.com/Intel-tensorflow/tensorflow is the repo?

In any case a possible backwards-compatible workaround:

  1. upload old wheels with a new build number if needed
  2. put non-primary .dist-info into {distribution}-{version}.data/purelib/

@lock lock bot added the auto-locked Outdated issues that have been locked by automation label Jan 24, 2020
@lock lock bot locked as resolved and limited conversation to collaborators Jan 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
auto-locked Outdated issues that have been locked by automation state: awaiting PR Feature discussed, PR is needed type: feature request Request for a new feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants