--target option causes issues when upgrading or re-installing

[theacodes]

If you install using the "--target" option:

pip install --target lib requests

And then repeat the command a second time, it doesn't say "requirement satisfied" it installs it again as a subdirectory of the existing installation (lib/requests/requests and lib/requests-2.2.0.dist-info/requests-2.2.0.dist-info

Repeat the command a third time and it appears the nesting stops but it chokes and still doesn't report "requirement satisfied":

Downloading/unpacking requests
  Downloading requests-2.2.0-py2.py3-none-any.whl (623kB): 623kB downloaded
Installing collected packages: requests
Successfully installed requests
Cleaning up...
Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 122, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 307, in run
    os.path.join(options.target_dir, item)
  File "/usr/lib/python2.7/shutil.py", line 291, in move
    raise Error, "Destination path '%s' already exists" % real_dst
Error: Destination path '/tmp/piptest/lib/requests/requests' already exists

Storing debug log for failure in /tmp/tmpcBDoql

[dmartin-gh]

Has anyone had a chance to look at this since January? I just ran across this issue with the newest pip release.

[webmaven]

@dstufft, there seems to be a cluster of issues relating to this one. Has anyone looked at it?

[theacodes]

Hey guys, this issue has been open for some time. Is there anything I can do to help this along?

[webmaven]

@brosner, @dholth, @dstufft, @jezdez, @pfmoore, @pnasrat, @r1chardj0n3s

This is a longstanding issue that is really annoying.

[pfmoore]

@jonparrott Apart from the obvious "patches welcome", not much...

The nesting behaviour is really odd. On the other hand, the fact that pip doesn't recognise that requests is installed is understandable (after all, it isn't installed in the sense that import requests will fail) - I can see that the behaviour is not what you'd want, but it's at least understandable (unlike the nesting!)

Personally, I only ever use --target to dump a copy of a distribution somewhere (for vendoring, for example) and I never want to "manage" it - I just blow away the directory and re-download if needed. Even the dist-info directories are unwanted clutter for that usage.

To be honest, I could argue that --target is just broken, and should be removed. Maybe we should replace it with a command, "pip unpack" that just unpacked the distribution into the target and made no pretense that it was "installed"? That might better match the reality.

[webmaven]

@pfmoore thanks for the response!

--target is currently rather important for one environment in particular, Google App Engine (eg. https://github.com/GoogleCloudPlatform/appengine-python-flask-skeleton and https://github.com/GoogleCloudPlatform/appengine-python-bottle-skeleton

An 'unpack' command, that still retrieved dependencies, updated in place, etc. could be a workable replacement, but you might want to talk to the folks in that community for feedback.

The 'blow away and re-download' approach is what @jonparrott's tool Gaelic does as a workaround, BTW.

[theacodes]

@pfmoore The vendoring use case is exactly what the app engine community is trying to do, as mentioned by @webmaven. The problem stems that even though --target works great for the initial vendoring, running it again just breaks everything. I've even written a tool to do exactly what you described- nuke the target directory and re-"install" everything.

Personally I think a command like "unpack" would perfectly suit the app engine community. I'm unfamiliar with pip internals and the particulars behind package installation, but I'm willing to dive in and take a first stab at implementing it. What are your thoughts, @webmaven?

Of course, if Google would add proper virtualenv support to app engine we'd be fine where we are. :(

[pfmoore]

@webmaven as you say, it's probably the AppEngine community that need to discuss alternative approaches. If we can fix upgrades with --target we will, but I'm not promising anything (it feels like a hard problem).

[webmaven]

OK, here is the bare minimum of functionality that I think a notional pip unpack would need. For all operations, unpack needs to get dependencies by default in the exact same way install does:

pip unpack <packagename> --target lib/

pip unpack -r requirements.txt --target lib/

A current pain point not yet addressed by pip: deriving a list of installed packages and their versions from a target folder. Because it isn't installing anything, you need an alternate method of creating expanded requirements files that include the unpacked dependencies. Either of the following would work: pip unpack -r requirements.txt --freeze > requirements.txt or pip freeze --target lib/ > requirements.txt Frankly, in the absence of an unpack command, the latter spelling would still be a good feature for pip, for the same reasons.

@jonparrott, can you think of anything else that is required to replace pip install --target for GAE's use case?