New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor --no-install/--no-download/--download (and use mkdtemp build dirs) #906

Closed
qwcode opened this Issue Apr 20, 2013 · 62 comments

Comments

Projects
None yet
@qwcode
Contributor

qwcode commented Apr 20, 2013

pip install --download -> pip download
pip install --no-install -> pip unpack
pip install --no-download -> get rid of it (see comments from @g2p below)

along with this, move to mkdtemp build dirs (instead of static ones) in virtualenvs and global installs.
I think the --no-install/--no-download "workflow" is what fueled the static build dirs pip has, which have caused so many problems.

@g2p

This comment has been minimized.

Show comment
Hide comment
@g2p

g2p Apr 21, 2013

Contributor

pip install --no-download isn't very useful or predictable compared to either

pip install unpack-root/*/
pip install download-root/*.*

The flag requires a well-known build dir which doesn't seem terribly safe (symlink attacks?) and gives an opportunity for multiple processes by the same user to step on each other's toes. --no-download pkgname also still requires network access (pip will hit PyPI or similar), whereas pip install download-root/* at least addresses the offline use case.

IMHO it would be better to deprecate this flag, and not make any new references to it or to the current "stuff accumulates in a static build dir" behaviour. Separating the other two flags into their own commands would be pretty good, I approve (and pip unpack should take a directory argument like download does).

Contributor

g2p commented Apr 21, 2013

pip install --no-download isn't very useful or predictable compared to either

pip install unpack-root/*/
pip install download-root/*.*

The flag requires a well-known build dir which doesn't seem terribly safe (symlink attacks?) and gives an opportunity for multiple processes by the same user to step on each other's toes. --no-download pkgname also still requires network access (pip will hit PyPI or similar), whereas pip install download-root/* at least addresses the offline use case.

IMHO it would be better to deprecate this flag, and not make any new references to it or to the current "stuff accumulates in a static build dir" behaviour. Separating the other two flags into their own commands would be pretty good, I approve (and pip unpack should take a directory argument like download does).

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Apr 27, 2013

Contributor

@g2p, I just updated the description regarding dropping --no-download.
when a pull for this comes up for review, we could try to surface the original motivation for the --no-install/--no-download combo and make sure we're not losnig a valuable use case.

Contributor

qwcode commented Apr 27, 2013

@g2p, I just updated the description regarding dropping --no-download.
when a pull for this comes up for review, we could try to surface the original motivation for the --no-install/--no-download combo and make sure we're not losnig a valuable use case.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Nov 25, 2013

Member

What is the work flow that is supported by pip download, pip unpack?

Member

dstufft commented Nov 25, 2013

What is the work flow that is supported by pip download, pip unpack?

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Nov 25, 2013

Contributor

in the description I mentioned "workflow" in reference to the existing --no-install / --no-download.
I'm guessing they were for local edits to packages, and then installing from those local edits. not sure?

as for pip download it's just better conceptually than pip install --download.

and pip unpack is better conceptually than pip install --no-install. As for why pip should help people do this kind of unpacking, I'm not sure, except that pip always has.

Contributor

qwcode commented Nov 25, 2013

in the description I mentioned "workflow" in reference to the existing --no-install / --no-download.
I'm guessing they were for local edits to packages, and then installing from those local edits. not sure?

as for pip download it's just better conceptually than pip install --download.

and pip unpack is better conceptually than pip install --no-install. As for why pip should help people do this kind of unpacking, I'm not sure, except that pip always has.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Nov 25, 2013

Member

Well that's what I'm trying to explore, why should we have this feature. Primarily because I want to see if we should even have this, and if we should what the best form for this feature to exist as. As it is we're just kind of guessing and unless we identify an actual use case that we can decide on if we want to keep supporting that use case or not I think we should just remove it.

Perhaps we should ping the pip mailing list and distutils-sig and see if anyone speaks up in support of this workflow?

Member

dstufft commented Nov 25, 2013

Well that's what I'm trying to explore, why should we have this feature. Primarily because I want to see if we should even have this, and if we should what the best form for this feature to exist as. As it is we're just kind of guessing and unless we identify an actual use case that we can decide on if we want to keep supporting that use case or not I think we should just remove it.

Perhaps we should ping the pip mailing list and distutils-sig and see if anyone speaks up in support of this workflow?

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Nov 25, 2013

Contributor

after the refactor, we'll have the following:

  • pip download: it helps fill up --find-link dirs for fast local installs.
  • pip unpack: yes, this is debateable
  • make sure pip install path/* works. valuable I think.

with these pieces, IF anyone was using the old --no-install / --no-download "workflow", then they can do it with these pieces, without us having to make a big decision on whether that workflow has merit or not.

but yea, we can ask about the old workflow on the list, or also ask @ianb directly. it came from him I'm guessing.

Contributor

qwcode commented Nov 25, 2013

after the refactor, we'll have the following:

  • pip download: it helps fill up --find-link dirs for fast local installs.
  • pip unpack: yes, this is debateable
  • make sure pip install path/* works. valuable I think.

with these pieces, IF anyone was using the old --no-install / --no-download "workflow", then they can do it with these pieces, without us having to make a big decision on whether that workflow has merit or not.

but yea, we can ask about the old workflow on the list, or also ask @ianb directly. it came from him I'm guessing.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Nov 25, 2013

Member

For what it's worth, I use pip install --download (what will be pip download) a lot. But I think of it in terms of "get the sdist" - either to have a local copy, or to then unpack it to make some local modifications before building. Two consequences of this:

  • I'd be surprised if pip download downloaded a wheel - I don't think that's possible, but I wanted to be explicit, I am always looking for an sdist when I use the command.
  • pip download /path/to/local/dir doesn't work as I expect (it copies the unpacked local dir). I use this reasonably frequently, and it always confuses me.

I'd actually argue for a pip sdist command, working like pip wheel rather than the simple pip download. It seems to me that it would have more useful semantics.

I don't see any real need for pip unpack - I'd just use pip sdist and unpack it myself without thinking. Remembering that pip unpack existed would be more effort than just doing that :-)

Member

pfmoore commented Nov 25, 2013

For what it's worth, I use pip install --download (what will be pip download) a lot. But I think of it in terms of "get the sdist" - either to have a local copy, or to then unpack it to make some local modifications before building. Two consequences of this:

  • I'd be surprised if pip download downloaded a wheel - I don't think that's possible, but I wanted to be explicit, I am always looking for an sdist when I use the command.
  • pip download /path/to/local/dir doesn't work as I expect (it copies the unpacked local dir). I use this reasonably frequently, and it always confuses me.

I'd actually argue for a pip sdist command, working like pip wheel rather than the simple pip download. It seems to me that it would have more useful semantics.

I don't see any real need for pip unpack - I'd just use pip sdist and unpack it myself without thinking. Remembering that pip unpack existed would be more effort than just doing that :-)

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Nov 25, 2013

Contributor

then unpack it to make some local modifications before building.

just curious. what modifications?
I'm guessing making modifications was the original motivation for pip install --no-install, which would theoretically be replaced by pip unpack. (and if not replaced, just removed)

I don't see any real need for pip unpack

if there is a substantial reason for people to be making local edits prior to install, then pip unpack just makes it easier to do that across a number of requirements

Contributor

qwcode commented Nov 25, 2013

then unpack it to make some local modifications before building.

just curious. what modifications?
I'm guessing making modifications was the original motivation for pip install --no-install, which would theoretically be replaced by pip unpack. (and if not replaced, just removed)

I don't see any real need for pip unpack

if there is a substantial reason for people to be making local edits prior to install, then pip unpack just makes it easier to do that across a number of requirements

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Nov 25, 2013

Contributor

pip download /path/to/local/dir

this should just fail immediately IMO, with "you can't download a local directory"

Contributor

qwcode commented Nov 25, 2013

pip download /path/to/local/dir

this should just fail immediately IMO, with "you can't download a local directory"

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Nov 25, 2013

Member

just curious. what modifications?

Usually, it's been to set universal=True in setup.cfg to get universal wheels, things like that.

Other things have been fiddling with custom script generation to try to use entry points (and hence pip's script support) instead, that's more the sort of thing I'd normally use a local clone for, but in at least one case (docutils) they use Subversion and frankly, life's too short to bother with svn :-)

Member

pfmoore commented Nov 25, 2013

just curious. what modifications?

Usually, it's been to set universal=True in setup.cfg to get universal wheels, things like that.

Other things have been fiddling with custom script generation to try to use entry points (and hence pip's script support) instead, that's more the sort of thing I'd normally use a local clone for, but in at least one case (docutils) they use Subversion and frankly, life's too short to bother with svn :-)

@quodlibetor

This comment has been minimized.

Show comment
Hide comment
@quodlibetor

quodlibetor Nov 25, 2013

FWIW I use pip install --no-install when I want to get all dependencies, recursively, for a package. Generally this is to make sure that all the dependencies exist within some local repository, and if they don't, then build them -- sometimes in an alternate format, like RPM.

quodlibetor commented Nov 25, 2013

FWIW I use pip install --no-install when I want to get all dependencies, recursively, for a package. Generally this is to make sure that all the dependencies exist within some local repository, and if they don't, then build them -- sometimes in an alternate format, like RPM.

@quodlibetor

This comment has been minimized.

Show comment
Hide comment
@quodlibetor

quodlibetor Nov 25, 2013

I could switch to unpacking the tar.gz/zip manually -- I obviously don't actually need the unpack functionality to be built in.

quodlibetor commented Nov 25, 2013

I could switch to unpacking the tar.gz/zip manually -- I obviously don't actually need the unpack functionality to be built in.

@ncoghlan

This comment has been minimized.

Show comment
Hide comment
@ncoghlan

ncoghlan Mar 3, 2014

Member

Just a heads up that CPython currently relies on --no-install in the venv test suite: I use it to make sure that venv & ensurepip are ignoring the related pip configuration settings as expected. (That is, when ensurepip originally wasn't ignoring the global settings, it would fail to install pip and test_venv would fail, so the test case demonstrates that is no longer happening)

If --no-install goes away entirely, we'll need to redesign those tests to do something else that would break the environment if the settings were being processed.

Member

ncoghlan commented Mar 3, 2014

Just a heads up that CPython currently relies on --no-install in the venv test suite: I use it to make sure that venv & ensurepip are ignoring the related pip configuration settings as expected. (That is, when ensurepip originally wasn't ignoring the global settings, it would fail to install pip and test_venv would fail, so the test case demonstrates that is no longer happening)

If --no-install goes away entirely, we'll need to redesign those tests to do something else that would break the environment if the settings were being processed.

@zroadhouse-rmn

This comment has been minimized.

Show comment
Hide comment
@zroadhouse-rmn

zroadhouse-rmn Mar 7, 2014

I use the "pip install --no-install --download" command at the tail end of my build process to help gather the sdists for all dependencies and include them in my deployment artifact. This enables me to ensure that the code I'm deploying is the same as the code I've built and tested.

download_dependencies ()
{
  echo "*** download_requirements ***"
  pip freeze | grep -vE "(wsm.|pygrametl|pyhive|pep8|pyflakes|pylint|nose|teamcity-|unittest2|coverage)" > build/dependencies.txt
  pip install -i $INDEX_URL --no-install --use-mirrors --ignore-installed --download=build/$DIST_NAME/packages -r build/dependencies.txt
  echo "*** download_requirements done. ***"
}

Then at deployment time, I use the following commands to extract my deployment artifact, create the virtualenv, and ensure that only the included packages are installed:

app_install_build()
{
    if [ ! -d "$APP_BUILD" ]
    then
        echo "Installing $APP_BUILD"

        # Un-pack the tgz
        tar zxf "$APP_BUILD.tgz"

        # Create the virtual environment
        virtualenv -p "$APP_PYTHON" "$APP_BUILD"
        source "$APP_BUILD/bin/activate"

        # Install the packages
        if [ -f "$APP_BUILD/requirements.txt" ]
        then
            # New style packaging using source distributions that is installable via pip
            pip install --find-links="file://$APP_HOME/$APP_BUILD/packages/" --no-index -r "$APP_BUILD/requirements.txt"
...

I'm looking at upgrading to a newer version of pip to pick up wheel support. How is this workflow handled by the new commands?

zroadhouse-rmn commented Mar 7, 2014

I use the "pip install --no-install --download" command at the tail end of my build process to help gather the sdists for all dependencies and include them in my deployment artifact. This enables me to ensure that the code I'm deploying is the same as the code I've built and tested.

download_dependencies ()
{
  echo "*** download_requirements ***"
  pip freeze | grep -vE "(wsm.|pygrametl|pyhive|pep8|pyflakes|pylint|nose|teamcity-|unittest2|coverage)" > build/dependencies.txt
  pip install -i $INDEX_URL --no-install --use-mirrors --ignore-installed --download=build/$DIST_NAME/packages -r build/dependencies.txt
  echo "*** download_requirements done. ***"
}

Then at deployment time, I use the following commands to extract my deployment artifact, create the virtualenv, and ensure that only the included packages are installed:

app_install_build()
{
    if [ ! -d "$APP_BUILD" ]
    then
        echo "Installing $APP_BUILD"

        # Un-pack the tgz
        tar zxf "$APP_BUILD.tgz"

        # Create the virtual environment
        virtualenv -p "$APP_PYTHON" "$APP_BUILD"
        source "$APP_BUILD/bin/activate"

        # Install the packages
        if [ -f "$APP_BUILD/requirements.txt" ]
        then
            # New style packaging using source distributions that is installable via pip
            pip install --find-links="file://$APP_HOME/$APP_BUILD/packages/" --no-index -r "$APP_BUILD/requirements.txt"
...

I'm looking at upgrading to a newer version of pip to pick up wheel support. How is this workflow handled by the new commands?

@Ivoz

This comment has been minimized.

Show comment
Hide comment
@Ivoz

Ivoz Mar 7, 2014

Member

@zroadhouse-wsm Everything still works for pip 1.5, which allows you to build (with the wheel package) and install wheels. This refactoring will be for 1.6 or later, in the future.

Member

Ivoz commented Mar 7, 2014

@zroadhouse-wsm Everything still works for pip 1.5, which allows you to build (with the wheel package) and install wheels. This refactoring will be for 1.6 or later, in the future.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Sep 25, 2014

Contributor

@derekstavis the only exception to it working that I know of is #804

Contributor

qwcode commented Sep 25, 2014

@derekstavis the only exception to it working that I know of is #804

@derekstavis

This comment has been minimized.

Show comment
Hide comment
@derekstavis

derekstavis Sep 25, 2014

@qwcode yeah, I can confirm the --build still works in pip 1.5.6

derekstavis commented Sep 25, 2014

@qwcode yeah, I can confirm the --build still works in pip 1.5.6

@derekstavis

This comment has been minimized.

Show comment
Hide comment
@derekstavis

derekstavis Sep 25, 2014

but when combined with --target gives a fatal error on already existing files in build dir that wasn't correctly cleaned.

derekstavis commented Sep 25, 2014

but when combined with --target gives a fatal error on already existing files in build dir that wasn't correctly cleaned.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Sep 25, 2014

Contributor

in hindsight, I think it was a mistake to deprecate --build and --no-clean. see #2062 about reversing this.

Contributor

qwcode commented Sep 25, 2014

in hindsight, I think it was a mistake to deprecate --build and --no-clean. see #2062 about reversing this.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Sep 25, 2014

Contributor

@derekstavis there's some fixes to --target in develop that @pfmoore worked on. possibly fixed. not sure.

Contributor

qwcode commented Sep 25, 2014

@derekstavis there's some fixes to --target in develop that @pfmoore worked on. possibly fixed. not sure.

@derekstavis

This comment has been minimized.

Show comment
Hide comment
@derekstavis

derekstavis Sep 25, 2014

I have worked out --build and --no-clean, by setting TMPFS and PIP_DOWNLOAD_CACHE env variables, things that are less likely to change in the future. I don't think it's a mistake to deprecate as long as the alternatives are well documented.

derekstavis commented Sep 25, 2014

I have worked out --build and --no-clean, by setting TMPFS and PIP_DOWNLOAD_CACHE env variables, things that are less likely to change in the future. I don't think it's a mistake to deprecate as long as the alternatives are well documented.

@derekstavis

This comment has been minimized.

Show comment
Hide comment
@derekstavis

derekstavis Sep 25, 2014

@qwcode I will try to build develop branch and test my scripts again to see if my problems got fixed.

derekstavis commented Sep 25, 2014

@qwcode I will try to build develop branch and test my scripts again to see if my problems got fixed.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Sep 26, 2014

Contributor

fyi, --build and --no-clean are no longer deprecated (see PR #2064). The warnings will be updated in the next pip release. I've updated the description of this issue to exclude mention of these 2 options.

Contributor

qwcode commented Sep 26, 2014

fyi, --build and --no-clean are no longer deprecated (see PR #2064). The warnings will be updated in the next pip release. I've updated the description of this issue to exclude mention of these 2 options.

@msabramo

This comment has been minimized.

Show comment
Hide comment
@msabramo

msabramo Mar 23, 2015

Contributor

What is the status of this? I came to this because of:

$ pip install --download=. --no-install foo
DEPRECATION: --no-install and --no-download are deprecated.
See https://github.com/pypa/pip/issues/906.

and now I'm just confused. :-(

$ pip download
ERROR: unknown command "download"
Contributor

msabramo commented Mar 23, 2015

What is the status of this? I came to this because of:

$ pip install --download=. --no-install foo
DEPRECATION: --no-install and --no-download are deprecated.
See https://github.com/pypa/pip/issues/906.

and now I'm just confused. :-(

$ pip download
ERROR: unknown command "download"
@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Mar 24, 2015

Contributor

they're marked with RemovedInPip7Warning, i.e. destined to be removed in pip v7
i.e. this can/should be worked on now...

Contributor

qwcode commented Mar 24, 2015

they're marked with RemovedInPip7Warning, i.e. destined to be removed in pip v7
i.e. this can/should be worked on now...

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Apr 7, 2015

Member

I'm going to go ahead and remove this now. I think the only open thing is moving pip install --download to pip download. I've captured this in #2643

Member

dstufft commented Apr 7, 2015

I'm going to go ahead and remove this now. I think the only open thing is moving pip install --download to pip download. I've captured this in #2643

@rbtcollins

This comment has been minimized.

Show comment
Hide comment
@rbtcollins

rbtcollins Apr 8, 2015

Contributor

@msabramo pip install --download --no-install has been broken for ages - it was behaving identically to pip install --download. So you can just use pip install --download in all probability.

Contributor

rbtcollins commented Apr 8, 2015

@msabramo pip install --download --no-install has been broken for ages - it was behaving identically to pip install --download. So you can just use pip install --download in all probability.

@asonnenschein

This comment has been minimized.

Show comment
Hide comment
@asonnenschein

asonnenschein Apr 10, 2015

This command in Python 3.4:

pip install --download psycopg2

Gives me this message instead of downloading psycopg2:

You must give at least one requirement to install (see "pip help install")

Any ideas?

asonnenschein commented Apr 10, 2015

This command in Python 3.4:

pip install --download psycopg2

Gives me this message instead of downloading psycopg2:

You must give at least one requirement to install (see "pip help install")

Any ideas?

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Apr 10, 2015

Member

--download takes a directory name, so that's what it's treating psycopg2 as. It's then reached the end of the arguments and hasn't seen any requirements to install. Hence the error.

Member

pfmoore commented Apr 10, 2015

--download takes a directory name, so that's what it's treating psycopg2 as. It's then reached the end of the arguments and hasn't seen any requirements to install. Hence the error.

@meejah

This comment has been minimized.

Show comment
Hide comment
@meejah

meejah Jun 18, 2015

FWIW, I was using --no-download after doing a peep install of dependencies to ensure that a final pip install --no-download -e . didn't go and download anything further (that won't get verified, of course, because no peep). Now I'm giving pip a bogus proxy instead ...

meejah commented Jun 18, 2015

FWIW, I was using --no-download after doing a peep install of dependencies to ensure that a final pip install --no-download -e . didn't go and download anything further (that won't get verified, of course, because no peep). Now I'm giving pip a bogus proxy instead ...

@ssbarnea

This comment has been minimized.

Show comment
Hide comment
@ssbarnea

ssbarnea Apr 1, 2016

Contributor

Is there a way to emulate a --dry mode with current version of pip?

Contributor

ssbarnea commented Apr 1, 2016

Is there a way to emulate a --dry mode with current version of pip?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment