refactor --no-install/--no-download/--download (and use mkdtemp build dirs)

[qwcode]

pip install --download -> pip download
pip install --no-install -> pip unpack
pip install --no-download -> get rid of it (see comments from @g2p below)

along with this, move to mkdtemp build dirs (instead of static ones) in virtualenvs and global installs.
I think the --no-install/--no-download "workflow" is what fueled the static build dirs pip has, which have caused so many problems.

[g2p]

pip install --no-download isn't very useful or predictable compared to either

pip install unpack-root/*/
pip install download-root/*.*

The flag requires a well-known build dir which doesn't seem terribly safe (symlink attacks?) and gives an opportunity for multiple processes by the same user to step on each other's toes. --no-download pkgname also still requires network access (pip will hit PyPI or similar), whereas pip install download-root/* at least addresses the offline use case.

IMHO it would be better to deprecate this flag, and not make any new references to it or to the current "stuff accumulates in a static build dir" behaviour. Separating the other two flags into their own commands would be pretty good, I approve (and pip unpack should take a directory argument like download does).

[qwcode]

@g2p, I just updated the description regarding dropping --no-download.
when a pull for this comes up for review, we could try to surface the original motivation for the --no-install/--no-download combo and make sure we're not losnig a valuable use case.

[dstufft]

What is the work flow that is supported by pip download, pip unpack?

[qwcode]

in the description I mentioned "workflow" in reference to the existing --no-install / --no-download.
I'm guessing they were for local edits to packages, and then installing from those local edits. not sure?

as for pip download it's just better conceptually than pip install --download.

and pip unpack is better conceptually than pip install --no-install. As for why pip should help people do this kind of unpacking, I'm not sure, except that pip always has.

[dstufft]

Well that's what I'm trying to explore, why should we have this feature. Primarily because I want to see if we should even have this, and if we should what the best form for this feature to exist as. As it is we're just kind of guessing and unless we identify an actual use case that we can decide on if we want to keep supporting that use case or not I think we should just remove it.

Perhaps we should ping the pip mailing list and distutils-sig and see if anyone speaks up in support of this workflow?

[qwcode]

after the refactor, we'll have the following:

• pip download: it helps fill up --find-link dirs for fast local installs.
• pip unpack: yes, this is debateable
• make sure pip install path/* works. valuable I think.

with these pieces, IF anyone was using the old --no-install / --no-download "workflow", then they can do it with these pieces, without us having to make a big decision on whether that workflow has merit or not.

but yea, we can ask about the old workflow on the list, or also ask @ianb directly. it came from him I'm guessing.

[pfmoore]

For what it's worth, I use pip install --download (what will be pip download) a lot. But I think of it in terms of "get the sdist" - either to have a local copy, or to then unpack it to make some local modifications before building. Two consequences of this:

• I'd be surprised if pip download downloaded a wheel - I don't think that's possible, but I wanted to be explicit, I am always looking for an sdist when I use the command.
• pip download /path/to/local/dir doesn't work as I expect (it copies the unpacked local dir). I use this reasonably frequently, and it always confuses me.

I'd actually argue for a pip sdist command, working like pip wheel rather than the simple pip download. It seems to me that it would have more useful semantics.

I don't see any real need for pip unpack - I'd just use pip sdist and unpack it myself without thinking. Remembering that pip unpack existed would be more effort than just doing that :-)

[qwcode]

then unpack it to make some local modifications before building.

just curious. what modifications?
I'm guessing making modifications was the original motivation for pip install --no-install, which would theoretically be replaced by pip unpack. (and if not replaced, just removed)

I don't see any real need for pip unpack

if there is a substantial reason for people to be making local edits prior to install, then pip unpack just makes it easier to do that across a number of requirements

[qwcode]

pip download /path/to/local/dir

this should just fail immediately IMO, with "you can't download a local directory"

[pfmoore]

just curious. what modifications?

Usually, it's been to set universal=True in setup.cfg to get universal wheels, things like that.

Other things have been fiddling with custom script generation to try to use entry points (and hence pip's script support) instead, that's more the sort of thing I'd normally use a local clone for, but in at least one case (docutils) they use Subversion and frankly, life's too short to bother with svn :-)

[quodlibetor]

FWIW I use pip install --no-install when I want to get all dependencies, recursively, for a package. Generally this is to make sure that all the dependencies exist within some local repository, and if they don't, then build them -- sometimes in an alternate format, like RPM.

[quodlibetor]

I could switch to unpacking the tar.gz/zip manually -- I obviously don't actually need the unpack functionality to be built in.

[ncoghlan]

Just a heads up that CPython currently relies on --no-install in the venv test suite: I use it to make sure that venv & ensurepip are ignoring the related pip configuration settings as expected. (That is, when ensurepip originally wasn't ignoring the global settings, it would fail to install pip and test_venv would fail, so the test case demonstrates that is no longer happening)

If --no-install goes away entirely, we'll need to redesign those tests to do something else that would break the environment if the settings were being processed.

[zroadhouse-rmn]

I use the "pip install --no-install --download" command at the tail end of my build process to help gather the sdists for all dependencies and include them in my deployment artifact. This enables me to ensure that the code I'm deploying is the same as the code I've built and tested.

download_dependencies ()
{
  echo "*** download_requirements ***"
  pip freeze | grep -vE "(wsm.|pygrametl|pyhive|pep8|pyflakes|pylint|nose|teamcity-|unittest2|coverage)" > build/dependencies.txt
  pip install -i $INDEX_URL --no-install --use-mirrors --ignore-installed --download=build/$DIST_NAME/packages -r build/dependencies.txt
  echo "*** download_requirements done. ***"
}

Then at deployment time, I use the following commands to extract my deployment artifact, create the virtualenv, and ensure that only the included packages are installed:

app_install_build()
{
    if [ ! -d "$APP_BUILD" ]
    then
        echo "Installing $APP_BUILD"

        # Un-pack the tgz
        tar zxf "$APP_BUILD.tgz"

        # Create the virtual environment
        virtualenv -p "$APP_PYTHON" "$APP_BUILD"
        source "$APP_BUILD/bin/activate"

        # Install the packages
        if [ -f "$APP_BUILD/requirements.txt" ]
        then
            # New style packaging using source distributions that is installable via pip
            pip install --find-links="file://$APP_HOME/$APP_BUILD/packages/" --no-index -r "$APP_BUILD/requirements.txt"
...

I'm looking at upgrading to a newer version of pip to pick up wheel support. How is this workflow handled by the new commands?

[Ivoz]

@zroadhouse-wsm Everything still works for pip 1.5, which allows you to build (with the wheel package) and install wheels. This refactoring will be for 1.6 or later, in the future.

[derekstavis]

@qwcode yeah, I can confirm the --build still works in pip 1.5.6

[derekstavis]

but when combined with --target gives a fatal error on already existing files in build dir that wasn't correctly cleaned.

[qwcode]

in hindsight, I think it was a mistake to deprecate --build and --no-clean. see #2062 about reversing this.

[qwcode]

@derekstavis there's some fixes to --target in develop that @pfmoore worked on. possibly fixed. not sure.

[derekstavis]

I have worked out --build and --no-clean, by setting TMPFS and PIP_DOWNLOAD_CACHE env variables, things that are less likely to change in the future. I don't think it's a mistake to deprecate as long as the alternatives are well documented.

[derekstavis]

@qwcode I will try to build develop branch and test my scripts again to see if my problems got fixed.

[qwcode]

fyi, --build and --no-clean are no longer deprecated (see PR #2064). The warnings will be updated in the next pip release. I've updated the description of this issue to exclude mention of these 2 options.

[msabramo]

What is the status of this? I came to this because of:

$ pip install --download=. --no-install foo
DEPRECATION: --no-install and --no-download are deprecated.
See https://github.com/pypa/pip/issues/906.

and now I'm just confused. :-(

$ pip download
ERROR: unknown command "download"

[qwcode]

they're marked with RemovedInPip7Warning, i.e. destined to be removed in pip v7
i.e. this can/should be worked on now...

[dstufft]

I'm going to go ahead and remove this now. I think the only open thing is moving pip install --download to pip download. I've captured this in #2643

[rbtcollins]

@msabramo pip install --download --no-install has been broken for ages - it was behaving identically to pip install --download. So you can just use pip install --download in all probability.

[asonnenschein]

This command in Python 3.4:

pip install --download psycopg2

Gives me this message instead of downloading psycopg2:

You must give at least one requirement to install (see "pip help install")

Any ideas?

[pfmoore]

--download takes a directory name, so that's what it's treating psycopg2 as. It's then reached the end of the arguments and hasn't seen any requirements to install. Hence the error.

[meejah]

FWIW, I was using --no-download after doing a peep install of dependencies to ensure that a final pip install --no-download -e . didn't go and download anything further (that won't get verified, of course, because no peep). Now I'm giving pip a bogus proxy instead ...

[ssbarnea]

Is there a way to emulate a --dry mode with current version of pip?