Skip to content

refactor --no-install/--no-download/--download (and use mkdtemp build dirs) #906

Closed
qwcode opened this Issue Apr 20, 2013 · 62 comments
@qwcode
Python Packaging Authority member
qwcode commented Apr 20, 2013

pip install --download -> pip download
pip install --no-install -> pip unpack
pip install --no-download -> get rid of it (see comments from @g2p below)

along with this, move to mkdtemp build dirs (instead of static ones) in virtualenvs and global installs.
I think the --no-install/--no-download "workflow" is what fueled the static build dirs pip has, which have caused so many problems.

@g2p
g2p commented Apr 21, 2013

pip install --no-download isn't very useful or predictable compared to either

pip install unpack-root/*/
pip install download-root/*.*

The flag requires a well-known build dir which doesn't seem terribly safe (symlink attacks?) and gives an opportunity for multiple processes by the same user to step on each other's toes. --no-download pkgname also still requires network access (pip will hit PyPI or similar), whereas pip install download-root/* at least addresses the offline use case.

IMHO it would be better to deprecate this flag, and not make any new references to it or to the current "stuff accumulates in a static build dir" behaviour. Separating the other two flags into their own commands would be pretty good, I approve (and pip unpack should take a directory argument like download does).

@qwcode
Python Packaging Authority member
qwcode commented Apr 27, 2013

@g2p, I just updated the description regarding dropping --no-download.
when a pull for this comes up for review, we could try to surface the original motivation for the --no-install/--no-download combo and make sure we're not losnig a valuable use case.

@dstufft
Python Packaging Authority member
dstufft commented Nov 25, 2013

What is the work flow that is supported by pip download, pip unpack?

@qwcode
Python Packaging Authority member
qwcode commented Nov 25, 2013

in the description I mentioned "workflow" in reference to the existing --no-install / --no-download.
I'm guessing they were for local edits to packages, and then installing from those local edits. not sure?

as for pip download it's just better conceptually than pip install --download.

and pip unpack is better conceptually than pip install --no-install. As for why pip should help people do this kind of unpacking, I'm not sure, except that pip always has.

@dstufft
Python Packaging Authority member
dstufft commented Nov 25, 2013

Well that's what I'm trying to explore, why should we have this feature. Primarily because I want to see if we should even have this, and if we should what the best form for this feature to exist as. As it is we're just kind of guessing and unless we identify an actual use case that we can decide on if we want to keep supporting that use case or not I think we should just remove it.

Perhaps we should ping the pip mailing list and distutils-sig and see if anyone speaks up in support of this workflow?

@qwcode
Python Packaging Authority member
qwcode commented Nov 25, 2013

after the refactor, we'll have the following:

  • pip download: it helps fill up --find-link dirs for fast local installs.
  • pip unpack: yes, this is debateable
  • make sure pip install path/* works. valuable I think.

with these pieces, IF anyone was using the old --no-install / --no-download "workflow", then they can do it with these pieces, without us having to make a big decision on whether that workflow has merit or not.

but yea, we can ask about the old workflow on the list, or also ask @ianb directly. it came from him I'm guessing.

@pfmoore
Python Packaging Authority member
pfmoore commented Nov 25, 2013

For what it's worth, I use pip install --download (what will be pip download) a lot. But I think of it in terms of "get the sdist" - either to have a local copy, or to then unpack it to make some local modifications before building. Two consequences of this:

  • I'd be surprised if pip download downloaded a wheel - I don't think that's possible, but I wanted to be explicit, I am always looking for an sdist when I use the command.
  • pip download /path/to/local/dir doesn't work as I expect (it copies the unpacked local dir). I use this reasonably frequently, and it always confuses me.

I'd actually argue for a pip sdist command, working like pip wheel rather than the simple pip download. It seems to me that it would have more useful semantics.

I don't see any real need for pip unpack - I'd just use pip sdist and unpack it myself without thinking. Remembering that pip unpack existed would be more effort than just doing that :-)

@qwcode
Python Packaging Authority member
qwcode commented Nov 25, 2013

then unpack it to make some local modifications before building.

just curious. what modifications?
I'm guessing making modifications was the original motivation for pip install --no-install, which would theoretically be replaced by pip unpack. (and if not replaced, just removed)

I don't see any real need for pip unpack

if there is a substantial reason for people to be making local edits prior to install, then pip unpack just makes it easier to do that across a number of requirements

@qwcode
Python Packaging Authority member
qwcode commented Nov 25, 2013

pip download /path/to/local/dir

this should just fail immediately IMO, with "you can't download a local directory"

@pfmoore
Python Packaging Authority member
pfmoore commented Nov 25, 2013

just curious. what modifications?

Usually, it's been to set universal=True in setup.cfg to get universal wheels, things like that.

Other things have been fiddling with custom script generation to try to use entry points (and hence pip's script support) instead, that's more the sort of thing I'd normally use a local clone for, but in at least one case (docutils) they use Subversion and frankly, life's too short to bother with svn :-)

@quodlibetor

FWIW I use pip install --no-install when I want to get all dependencies, recursively, for a package. Generally this is to make sure that all the dependencies exist within some local repository, and if they don't, then build them -- sometimes in an alternate format, like RPM.

@quodlibetor

I could switch to unpacking the tar.gz/zip manually -- I obviously don't actually need the unpack functionality to be built in.

@ncoghlan
ncoghlan commented Mar 3, 2014

Just a heads up that CPython currently relies on --no-install in the venv test suite: I use it to make sure that venv & ensurepip are ignoring the related pip configuration settings as expected. (That is, when ensurepip originally wasn't ignoring the global settings, it would fail to install pip and test_venv would fail, so the test case demonstrates that is no longer happening)

If --no-install goes away entirely, we'll need to redesign those tests to do something else that would break the environment if the settings were being processed.

@zroadhouse-wsm

I use the "pip install --no-install --download" command at the tail end of my build process to help gather the sdists for all dependencies and include them in my deployment artifact. This enables me to ensure that the code I'm deploying is the same as the code I've built and tested.

download_dependencies ()
{
  echo "*** download_requirements ***"
  pip freeze | grep -vE "(wsm.|pygrametl|pyhive|pep8|pyflakes|pylint|nose|teamcity-|unittest2|coverage)" > build/dependencies.txt
  pip install -i $INDEX_URL --no-install --use-mirrors --ignore-installed --download=build/$DIST_NAME/packages -r build/dependencies.txt
  echo "*** download_requirements done. ***"
}

Then at deployment time, I use the following commands to extract my deployment artifact, create the virtualenv, and ensure that only the included packages are installed:

app_install_build()
{
    if [ ! -d "$APP_BUILD" ]
    then
        echo "Installing $APP_BUILD"

        # Un-pack the tgz
        tar zxf "$APP_BUILD.tgz"

        # Create the virtual environment
        virtualenv -p "$APP_PYTHON" "$APP_BUILD"
        source "$APP_BUILD/bin/activate"

        # Install the packages
        if [ -f "$APP_BUILD/requirements.txt" ]
        then
            # New style packaging using source distributions that is installable via pip
            pip install --find-links="file://$APP_HOME/$APP_BUILD/packages/" --no-index -r "$APP_BUILD/requirements.txt"
...

I'm looking at upgrading to a newer version of pip to pick up wheel support. How is this workflow handled by the new commands?

@Ivoz
Python Packaging Authority member
Ivoz commented Mar 7, 2014

@zroadhouse-wsm Everything still works for pip 1.5, which allows you to build (with the wheel package) and install wheels. This refactoring will be for 1.6 or later, in the future.

@zroadhouse-wsm

@Ivoz - thanks! Just want to make sure that the basic workflow of being able to download sdists via pip is preserved. We will be utilizing wheels as a mechanism to speed up deploys by caching the binary builds off packages on the target servers. I.e. our deployment artifacts produced by the build system are architecture neutral (sdists) and our deployments are fast (wheel).

@qwcode
Python Packaging Authority member
qwcode commented Mar 7, 2014

"pip install --no-install --download" command at the tail end of my build process
to help gather the sdists

btw, --no-install is not necessary here. --download does not install, and ignores what's already installed.

@fy0 fy0 referenced this issue in mozillazg/baidu-pcs-python-sdk Mar 11, 2014
Closed

python3 无法安装 #7

@Ivoz
Python Packaging Authority member
Ivoz commented Mar 11, 2014

Btw, thinking out loud this could be a great avenue to do some refactoring? e.g, introduce a new codepath for pip download (and perhaps, pip unpack) rather than hooking into the old one, that is far easier to understand.

@qwcode
Python Packaging Authority member
qwcode commented Mar 11, 2014

to be clear, since the deprecations appeared for 1.5, I think 1.7 is the timeframe for starting on this work IMO, but yes, @Ivoz , refactoring could occur : )

@dstufft
Python Packaging Authority member
dstufft commented Mar 11, 2014

Well we should get pip download into 1.6 I think. So people have it there to be able to migrate to before the old way is gone.

@Ivoz
Python Packaging Authority member
Ivoz commented Mar 11, 2014

@qwcode it was my impression that in 1.6 pip download could coexist with pip install --download? And you'd only remove the latter in a later minor.

@qwcode
Python Packaging Authority member
qwcode commented Mar 12, 2014

ok, sure, adding pip download, could happen now. I was thinking about the rest.

rather than hooking into the old one, that is far easier to understand.

just realize that in whatever refactor you might attempt,. both install and download do really share a lot of the same logic. both have to download archives, unpack, and walk dependencies.

@Ivoz
Python Packaging Authority member
Ivoz commented Mar 12, 2014

Oh, definitely. So IMHO this would be a good waypoint into making progress on refactoring install as well.

@qwcode
Python Packaging Authority member
qwcode commented Mar 12, 2014

btw, if you haven't seen it already, see #1526

@auuron
auuron commented Mar 17, 2014

What should i use instead of pip install --build?

@qwcode
Python Packaging Authority member
qwcode commented Mar 17, 2014

@auuron although none of this has been implemented yet, the idea is that pip would start using random tmp dirs for building, i.e. no fixed location that a user could set with --build.

@auuron
auuron commented Mar 17, 2014

@qwcode good idea, because now i'm doing that by myself using --build...

So, i understand that --build will work until this will be implemented and i can keep using it? (its very important for me to keep every single build in a diffrend temporary directory)
Abother question is that i keep getting warnings about DEPRECATION, is it possible to turn it off? (or all warnings)

@qwcode
Python Packaging Authority member
qwcode commented Mar 17, 2014

@auuron yes, it will keep working. sorry, no good way atm to turn off deprecation warnings, other than using -qq, i.e. doubly quiet which will only log errors. http://www.pip-installer.org/en/latest/reference/pip.html#console-logging

@gmist gmist referenced this issue in gae-init/gae-init Mar 22, 2014
Merged

Install and manage python packages from pip #116

@Nabellaleen

I don't understand the --no-clean deprecation. It make me do a download + unpack instead of simply use install, because if a large package install fail (for example, a dependancy is missing and cannot be downloaded automatically, making me do a pip install dep1), I have to re-download it and re-unpack it.

@Ivoz
Python Packaging Authority member
Ivoz commented Apr 4, 2014

@Nabellaleen you may wish to make use of --download-cache to cache installs.

@qwcode
Python Packaging Authority member
qwcode commented Apr 4, 2014
@clayg
clayg commented May 2, 2014

debugging a build failure on a new version of a dependency and the author of the libary needed the build logs - I very much needed --no-clean and it worked on pip 1.5.4 with a warning - please do not remove --no-clean (or document the alternative for this use case?)

Thanks!

@qwcode
Python Packaging Authority member
qwcode commented May 2, 2014

fair point @clayg , I marked it for deprecation along with the others because it was tied into the static build dir code, which will be going away, but I can see keeping it for the dynamic tmp build dirs. an alternative use case could be that one has to pip unpack the troublesome dependency and build/install it manually to do troubleshooting. I'll note in the description that the fate of --no-clean is TBD.

@dstufft
Python Packaging Authority member
dstufft commented May 2, 2014

I think keeping --no-clean makes sense, and maybe even --build.

@qwcode
Python Packaging Authority member
qwcode commented May 2, 2014

I'm a little concerned about pip leaving tmp dirs because someone is using --no-clean (separate from --build)

@dstufft
Python Packaging Authority member
dstufft commented May 2, 2014

I don't think there's anything to be concerned about, that's exactly what the end user is asking for by using that option.

@qwcode
Python Packaging Authority member
qwcode commented May 2, 2014

It feels odd to me for a --no-clean flag to leave random tmp dirs. one thought is to have --build imply no cleaning, and just keep the --build flag for those wanting to troubleshoot or self-manage a build area.

@dstufft
Python Packaging Authority member
dstufft commented May 2, 2014

Eh, I don't see any reason to refuse to do --no-clean with a random tmp dir. It's not something I would probably do, but if someone wants to I don't see any reason. What's the downside?

@clayg
clayg commented May 2, 2014

well what is --build supposed to do? I ran install with in a virtualenv with --build ./build and there's nothing in ./build after install? --build ./build --no-clean seemed to get what I needed, but I'm still left needing --no-clean. pip 1.5.4 doesn't have an unpack command? But if there was a command that just downloads and inflates without actually attempting the install the way pip would have during install (and leaving artifacts left over for debugging) - it's probably not quite what this particular use case would be going after (does pip always just python setup.py install?).

I needed something like "hey package maintainer, on my funky distro no one uses anymore pip install xyz failed with this crazy gcc i don't even" 'ok, np, run pip install --magic-flag xyz and give me everything in ./magic-place-where-all-the-build-stuff-is'.

--no-clean seemed to fit the bill in this one case, pip even warned me when I tried to do the install again and it encountered a left over build directory!

I'll probably bow out at this point, as I don't think I can do much to help you guys figure out "the right thing" beyond advocating for the use case - which i'm guessing you grok pretty clearly. Good luck!

@CARocha
CARocha commented May 16, 2014

Hi guys i try install GDAL but i need compiled the source i try this tuto http://linfiniti.com/2013/02/installing-python-gdal-into-a-python-virtualenv-in-osx/ but in my venv local with pip 1.5.5 not working anything this command not working pip install --no-download GDAL
this log
DEPRECATION: --no-install, --no-download, --build, and --no-clean are deprecated. See #906.
Cleaning up...
Could not install requirement GDAL because source folder /Users/n/proyectos/django/personal/venvmapas/build/GDAL does not exist (perhaps --no-download was used without first running an equivalent install with --no-install?)
Storing debug log for failure in /Users/n/.pip/pip.log

i try with this
$pip --download gdal
Usage:

pip [options]
no such option: --download

nothing happen :( which is the best way for install GDAL in my virtualenv??
Cheers

@blueyed
blueyed commented May 26, 2014

My use case for --build is that I want to prevent pip install to fail because of an existing build dir ("pip can't proceed with requirement 'foo' due to a pre-existing build directory. … Please delete it and try again.").

Therefore I set it to a specific directory and remove this in the deployment script.

Without --build I could remove the automatic build directory, assuming that there's still a root somewhere?!

Another use case for --build is using '/tmp' or some other location backed by tmpfs (RAM filesystem).

@andycasey

My use case for --build is using /tmp because some old FORTRAN codes break down if the folder name is too long, which it is for the default site packages folder. Having 'random tmp directories' that are short (I think ~40 chars was the limit for the FORTRAN code in question) in length would resolve my situation.

@remh remh added a commit to DataDog/omnibus-software that referenced this issue Jul 22, 2014
@remh remh --build parameter is deprecated and should be removed ea885ad
@derekstavis

If --build no longer works and tmpfs is not sufficiently large to build a package what should I do?

@g2p
g2p commented Sep 24, 2014

Set TMPDIR to a directory with free space.

@derekstavis

@g2p Thanks! It would be nice to reference this in deprecation message :)

@qwcode
Python Packaging Authority member
qwcode commented Sep 25, 2014

@derekstavis --build still works. (#2060 raised some doubts, but it still works in 1.5.6 based on my testing)

you do need to use -no-clean if you want the build contents to remain , and not be cleaned every install.

@qwcode
Python Packaging Authority member
qwcode commented Sep 25, 2014

@derekstavis the only exception to it working that I know of is #804

@derekstavis

@qwcode yeah, I can confirm the --build still works in pip 1.5.6

@derekstavis

but when combined with --target gives a fatal error on already existing files in build dir that wasn't correctly cleaned.

@qwcode
Python Packaging Authority member
qwcode commented Sep 25, 2014

in hindsight, I think it was a mistake to deprecate --build and --no-clean. see #2062 about reversing this.

@qwcode
Python Packaging Authority member
qwcode commented Sep 25, 2014

@derekstavis there's some fixes to --target in develop that @pfmoore worked on. possibly fixed. not sure.

@derekstavis

I have worked out --build and --no-clean, by setting TMPFS and PIP_DOWNLOAD_CACHE env variables, things that are less likely to change in the future. I don't think it's a mistake to deprecate as long as the alternatives are well documented.

@derekstavis

@qwcode I will try to build develop branch and test my scripts again to see if my problems got fixed.

@qwcode
Python Packaging Authority member
qwcode commented Sep 26, 2014

fyi, --build and --no-clean are no longer deprecated (see PR #2064). The warnings will be updated in the next pip release. I've updated the description of this issue to exclude mention of these 2 options.

@msabramo

What is the status of this? I came to this because of:

$ pip install --download=. --no-install foo
DEPRECATION: --no-install and --no-download are deprecated.
See https://github.com/pypa/pip/issues/906.

and now I'm just confused. :-(

$ pip download
ERROR: unknown command "download"
@qwcode
Python Packaging Authority member
qwcode commented Mar 24, 2015

they're marked with RemovedInPip7Warning, i.e. destined to be removed in pip v7
i.e. this can/should be worked on now...

@dstufft
Python Packaging Authority member
dstufft commented Apr 7, 2015

I'm going to go ahead and remove this now. I think the only open thing is moving pip install --download to pip download. I've captured this in #2643

@dstufft dstufft closed this Apr 7, 2015
@rbtcollins

@msabramo pip install --download --no-install has been broken for ages - it was behaving identically to pip install --download. So you can just use pip install --download in all probability.

@asonnenschein

This command in Python 3.4:

pip install --download psycopg2

Gives me this message instead of downloading psycopg2:

You must give at least one requirement to install (see "pip help install")

Any ideas?

@pfmoore
Python Packaging Authority member
pfmoore commented Apr 10, 2015

--download takes a directory name, so that's what it's treating psycopg2 as. It's then reached the end of the arguments and hasn't seen any requirements to install. Hence the error.

@meejah
meejah commented Jun 18, 2015

FWIW, I was using --no-download after doing a peep install of dependencies to ensure that a final pip install --no-download -e . didn't go and download anything further (that won't get verified, of course, because no peep). Now I'm giving pip a bogus proxy instead ...

@ssbarnea
ssbarnea commented Apr 1, 2016

Is there a way to emulate a --dry mode with current version of pip?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.