Installing several libs to the same target can result in an exception

[mal]

Steps to produce:

mkdir test; cd test
pip install -t local bottle
pip install -t local pytest
pip install -t local pytest_cov

Result:

Successfully installed pytest-cov pytest cov-core py coverage
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/home/mal/.pyenv/versions/3.3.2/lib/python3.3/site-packages/pip-1.4.1-py3.3.egg/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/home/mal/.pyenv/versions/3.3.2/lib/python3.3/site-packages/pip-1.4.1-py3.3.egg/pip/commands/install.py", line 268, in run
    os.path.join(options.target_dir, item)
  File "/home/mal/.pyenv/versions/3.3.2/lib/python3.3/shutil.py", line 520, in move
    raise Error("Destination path '%s' already exists" % real_dst)
shutil.Error: Destination path '/home/mal/local/__pycache__/__pycache__' already exists

Wrapping this line in a try..except solves this problem for me, not sure how viable that is as a real world fix.

Guessing this occurs because each tries to move the __pycache__ folder. The first two succeed (sort of; the second __pycache__ ends up inside the first), the third fails. Another solution might be to exempt __pycache__ from being copied here?

[qwcode]

Another solution might be to exempt pycache from being copied here?

I agree, this copy needs to filter out pycache.
marking as a bug.

[mal]

That approach appears to work on python3 (tested with pip 1.4.1 on 3.3.2), however the way this manifests in python2 (pip 1.5.2 on 2.7.3) is a little different:

$ python2 -m pip install -t python_modules coverage
... installs fine
$ python2 -m pip install -t python_modules coverage
... appears to install fine
$ python2 -m pip install -t python_modules coverage
Exception:
Traceback (most recent call last):
  File "/home/mal/.pyenv/versions/etl/lib/python2.7/site-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/home/mal/.pyenv/versions/etl/lib/python2.7/site-packages/pip/commands/install.py", line 307, in run
    os.path.join(options.target_dir, item)
  File "/home/mal/.pyenv/versions/2.7.3/lib/python2.7/shutil.py", line 289, in move
    raise Error, "Destination path '%s' already exists" % real_dst
Error: Destination path '/home/mal/code/test/python_modules/coverage/coverage' already exists

'.../python_modules/coverage/coverage' already exists

[mal]

Two possible solutions would be to either;

• Remove an existing destination before moving the new file tree
• Skip the copy if the file tree already exists
• Something more complicated

I'm happy to implement either of the first two, however I suspect someone with greater knowledge of pip's code may have a better approach. If that's the case and they are happy to explain it, I'd be happy to work on it.

[coffenbacher]

I'm hit with this one - pressing deadline at the moment so I'm just going to work around, but I'll try to come back and submit a pull request soon. I'm assuming no one else has tried anything since Feb?

[mal]

/cc @qwcode

As of #2122 (v6.0) this issue changed, but still exists:

$ python --version; pip --version
Python 3.4.1
pip 6.0.8 from /home/mal/.pyenv/versions/3.4.1/lib/python3.4/site-packages (python 3.4)
$ pip install -t local bottle
...
Successfully installed bottle

$ pip install -t local pytest
...
Successfully installed py pytest
Target directory /home/mal/code/personal/pip/test/local/__pycache__ already exists. Specify --upgrade to force replacement.

$ python --version; pip --version
Python 2.7.3
pip 6.0.8 from /usr/local/lib/python2.7/dist-packages (python 2.7)
$ pip install -t local bottle
...
Successfully installed bottle

$ pip install -t local pytest
...
Successfully installed py-1.4.13 pytest-2.3.4

$ pip install -t local pytest_cov
...
Successfully installed cov-core coverage py-1.4.13 pytest-2.3.4 pytest-cov
Target directory /home/mal/code/personal/pip/test/local/pytest-2.6.4-py2.7.egg-info already exists. Specify --upgrade to force replacement.
Target directory /home/mal/code/personal/pip/test/local/pytest.py already exists. Specify --upgrade to force replacement.
Target directory /home/mal/code/personal/pip/test/local/py-1.4.26-py2.7.egg-info already exists. Specify --upgrade to force replacement.
Target directory /home/mal/code/personal/pip/test/local/_pytest already exists. Specify --upgrade to force replacement.
Target directory /home/mal/code/personal/pip/test/local/py already exists. Specify --upgrade to force replacement.
Target directory /home/mal/code/personal/pip/test/local/pytest.pyc already exists. Specify --upgrade to force replacement.

It can now be mitigated more easily by simply always using the --upgrade flag however this is far from desired and risks breaking the dependency tree. I'm guessing the "proper" solution to this involves some earlier part of pip looking in the target directory to establish the current state of it and the contained dependencies?

As previously stated, I'm happy to spend a bit of time on this, but would appreciate some guidance as to what desired behaviour would be, and where that logic should live in advance of committing time and effort to a solution that may ultimately be unacceptable in the context of the broader community.

[dstufft]

Closing this. It currently works fine, but you do get some warnings due to already existing files and not wanting to upgrade those files without a --upgrade flag. However the install itself appears to successfully complete.