New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support pip install --prefix #3252

Merged
merged 3 commits into from Nov 26, 2015

Conversation

Projects
None yet
6 participants
@domenkozar
Contributor

domenkozar commented Nov 17, 2015

Review on Reviewable

Currently pip supports only --root for installing packages relative to a folder (with the whole prefix, which is extracted from sys.prefix.

This PR adds support for --prefix to denote the prefix path relative to lib, bin, etc.

Example:

$ mkdir tmp
$ pip install --prefix /usr/local/ --root tmp/ Django
$ ls tmp/usr/local/
bin  lib

This is needed for Nix package manager to be able to install packages to separate folders. It is possible to specify prefix to install command with --install-option="--prefix=...", but that doesn't work for wheels at all so hence this PR.

@domenkozar domenkozar referenced this pull request Nov 17, 2015

Merged

buildPythonPackage: use wheels internally #11086

7 of 7 tasks complete
@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 18, 2015

Contributor

@dstufft @qwcode ready for review

Contributor

domenkozar commented Nov 18, 2015

@dstufft @qwcode ready for review

Show outdated Hide outdated pip/locations.py Outdated
@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 19, 2015

Contributor

@xavfernandez thanks, good point. Fixed.

Contributor

domenkozar commented Nov 19, 2015

@xavfernandez thanks, good point. Fixed.

Show outdated Hide outdated pip/locations.py Outdated
@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Nov 19, 2015

Contributor

Except for the CommandError/InstallationError nitpicking, this looks good to me :)

Contributor

xavfernandez commented Nov 19, 2015

Except for the CommandError/InstallationError nitpicking, this looks good to me :)

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Nov 22, 2015

Contributor

there's no test for sdist installs using --prefix

Contributor

qwcode commented Nov 22, 2015

there's no test for sdist installs using --prefix

domenkozar added some commits Nov 16, 2015

Support --prefix argument to install command.
There was no way to override prefix for an installation for wheels
previously. The default it whatever sys.prefix points to.
@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 22, 2015

Contributor

@qwcode @xavfernandez @dstufft :

  • moved flag detection into the installCommand
  • wrote a test for it
  • wrote a test for sdist install to a --prefix
  • mr. Travis approves
Contributor

domenkozar commented Nov 22, 2015

@qwcode @xavfernandez @dstufft :

  • moved flag detection into the installCommand
  • wrote a test for it
  • wrote a test for sdist install to a --prefix
  • mr. Travis approves
@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Nov 22, 2015

Contributor

lgtm :shipit:

the only reservation I can think of is that we might want to plan out a complete list of setuptools-agnostic install flags vs doing these one by one like this.

maybe @dstufft has a vision for that. not sure.

Contributor

qwcode commented Nov 22, 2015

lgtm :shipit:

the only reservation I can think of is that we might want to plan out a complete list of setuptools-agnostic install flags vs doing these one by one like this.

maybe @dstufft has a vision for that. not sure.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Nov 22, 2015

Member

I don't have a long term plan other than translating whatever useful flags into top level ones. I haven't had a chance to look at this PR because I'm moving soon and been packing and such but I'm +1 on the idea and I don't think we need to hold off on a useful flag to try and figure out all the other ones that might be useful.

Sent from my iPhone

On Nov 22, 2015, at 5:02 PM, Marcus Smith notifications@github.com wrote:

lgtm

the only reservation I can think of is that we might want to plan out a complete list of setuptools-agnostic install flags vs doing these one by one like this.

maybe @dstufft has a vision for that. not sure.


Reply to this email directly or view it on GitHub.

Member

dstufft commented Nov 22, 2015

I don't have a long term plan other than translating whatever useful flags into top level ones. I haven't had a chance to look at this PR because I'm moving soon and been packing and such but I'm +1 on the idea and I don't think we need to hold off on a useful flag to try and figure out all the other ones that might be useful.

Sent from my iPhone

On Nov 22, 2015, at 5:02 PM, Marcus Smith notifications@github.com wrote:

lgtm

the only reservation I can think of is that we might want to plan out a complete list of setuptools-agnostic install flags vs doing these one by one like this.

maybe @dstufft has a vision for that. not sure.


Reply to this email directly or view it on GitHub.

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 23, 2015

Contributor

👍

Contributor

domenkozar commented Nov 23, 2015

👍

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 23, 2015

Contributor

It seems I also need to fix editable mode, so hang on a bit so I also fix that (and test it).

Contributor

domenkozar commented Nov 23, 2015

It seems I also need to fix editable mode, so hang on a bit so I also fix that (and test it).

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 24, 2015

Contributor

Added --prefix support also to editable mode and added a test.

Contributor

domenkozar commented Nov 24, 2015

Added --prefix support also to editable mode and added a test.

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 24, 2015

Contributor

@qwcode @xavfernandez @dstufft ready for a review. Not sure if anyone else will find pip install -e --prefix useful, but I do.

Contributor

domenkozar commented Nov 24, 2015

@qwcode @xavfernandez @dstufft ready for a review. Not sure if anyone else will find pip install -e --prefix useful, but I do.

# make sure target path is in PYTHONPATH
pythonpath = script.scratch_path / site_packages
pythonpath.makedirs()
script.environ["PYTHONPATH"] = pythonpath

This comment has been minimized.

@xavfernandez

xavfernandez Nov 24, 2015

Contributor

why is this needed ?

@xavfernandez

xavfernandez Nov 24, 2015

Contributor

why is this needed ?

This comment has been minimized.

@domenkozar

domenkozar Nov 24, 2015

Contributor

otherwise setuptools will refuse to install at that location

@domenkozar

domenkozar Nov 24, 2015

Contributor

otherwise setuptools will refuse to install at that location

This comment has been minimized.

@xavfernandez

xavfernandez Nov 24, 2015

Contributor

I guess this means you should add extra docs or enhance the help message to make that clear for potential other users... or make sure pip automatically update the environment before invoking setuptools ?

@xavfernandez

xavfernandez Nov 24, 2015

Contributor

I guess this means you should add extra docs or enhance the help message to make that clear for potential other users... or make sure pip automatically update the environment before invoking setuptools ?

This comment has been minimized.

@domenkozar

domenkozar Nov 24, 2015

Contributor

pip can only modify environment for itself so that won't help the user.

I think current situation is good enough as setuptools prints out a long and verbose error if you try to install a package as editable to a location that is not in PYTHONPATH.

@domenkozar

domenkozar Nov 24, 2015

Contributor

pip can only modify environment for itself so that won't help the user.

I think current situation is good enough as setuptools prints out a long and verbose error if you try to install a package as editable to a location that is not in PYTHONPATH.

This comment has been minimized.

@xavfernandez

xavfernandez Nov 24, 2015

Contributor

Indeed, for the curious, you get something like:

    running develop
    Checking .pth file support in /home/xfernandez/tmp/lib/python2.7/site-packages
    /home/xfernandez/.virtualenvs/c13ba271efca52d0/bin/python -E -c pass
    TEST FAILED: /home/xfernandez/tmp/lib/python2.7/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH

    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /home/xfernandez/tmp/lib/python2.7/site-packages

    and your PYTHONPATH environment variable currently contains:

        ''

    Here are some of your options for correcting the problem:

    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files

    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)

    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:

      https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations

    Please make the appropriate changes for your system and try again.

    ----------------------------------------
Command "/home/xfernandez/.virtualenvs/c13ba271efca52d0/bin/python -c "import setuptools,
tokenize; __file__='/home/xfernandez/dev/blease/setup.py'; exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
--prefix=/home/xfernandez/tmp" failed with error code 1 in /home/xfernandez/dev/blease
@xavfernandez

xavfernandez Nov 24, 2015

Contributor

Indeed, for the curious, you get something like:

    running develop
    Checking .pth file support in /home/xfernandez/tmp/lib/python2.7/site-packages
    /home/xfernandez/.virtualenvs/c13ba271efca52d0/bin/python -E -c pass
    TEST FAILED: /home/xfernandez/tmp/lib/python2.7/site-packages does NOT support .pth files
    error: bad install directory or PYTHONPATH

    You are attempting to install a package to a directory that is not
    on PYTHONPATH and which Python does not read ".pth" files from.  The
    installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:

        /home/xfernandez/tmp/lib/python2.7/site-packages

    and your PYTHONPATH environment variable currently contains:

        ''

    Here are some of your options for correcting the problem:

    * You can choose a different installation directory, i.e., one that is
      on PYTHONPATH or supports .pth files

    * You can add the installation directory to the PYTHONPATH environment
      variable.  (It must then also be on PYTHONPATH whenever you run
      Python and want to use the package(s) you are installing.)

    * You can set up the installation directory to support ".pth" files by
      using one of the approaches described here:

      https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations

    Please make the appropriate changes for your system and try again.

    ----------------------------------------
Command "/home/xfernandez/.virtualenvs/c13ba271efca52d0/bin/python -c "import setuptools,
tokenize; __file__='/home/xfernandez/dev/blease/setup.py'; exec(compile(getattr(tokenize, 'open',
open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" develop --no-deps
--prefix=/home/xfernandez/tmp" failed with error code 1 in /home/xfernandez/dev/blease
@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Nov 24, 2015

Contributor

LGTM

Contributor

xavfernandez commented Nov 24, 2015

LGTM

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 26, 2015

Contributor

@qwcode @dstufft any feedback? :)

Contributor

domenkozar commented Nov 26, 2015

@qwcode @dstufft any feedback? :)

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Nov 26, 2015

Contributor

ok, merging. will push a changelog entry afterwards.

Contributor

qwcode commented Nov 26, 2015

ok, merging. will push a changelog entry afterwards.

qwcode added a commit that referenced this pull request Nov 26, 2015

@qwcode qwcode merged commit ca47dc9 into pypa:develop Nov 26, 2015

1 of 2 checks passed

code-review/reviewable Review in progress: 0 of 7 files reviewed, 4 unresolved discussions
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 26, 2015

Contributor

Thank you!

Contributor

domenkozar commented Nov 26, 2015

Thank you!

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Nov 27, 2015

How does this mix with --target?
Also I agree having both --root and --prefix seems odd (like you mention in #2677 (comment)). Would it make sense to have --prefix denote whole path so that specifying --root wouldn't be needed? But then what would be the difference to --target?

piotr-dobrogost commented Nov 27, 2015

How does this mix with --target?
Also I agree having both --root and --prefix seems odd (like you mention in #2677 (comment)). Would it make sense to have --prefix denote whole path so that specifying --root wouldn't be needed? But then what would be the difference to --target?

@domenkozar domenkozar deleted the domenkozar:install/prefix branch Nov 27, 2015

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 27, 2015

Contributor

@piotr-dobrogost

  File "/nix/store/i2yqin5ikgbph0yd2pvndq43pbss7dvj-python-2.7.10-bootstrapped-pip-7.1.2/lib/python2.7/site-packages/pip/locations.py", line 194, in distutils_scheme
    i.finalize_options()
  File "/nix/store/a45l8r0zqfr5jhqswqqcya28h59n63js-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg/setuptools/command/install.py", line 38, in finalize_options
    orig.install.finalize_options(self)
  File "/nix/store/rfcsnszaw2jlkqdgg65h3phss8xa0mlp-python-2.7.10/lib/python2.7/distutils/command/install.py", line 264, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both"
DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

These two flags should conflict and --prefix is superior because it will also affect data_files.

Contributor

domenkozar commented Nov 27, 2015

@piotr-dobrogost

  File "/nix/store/i2yqin5ikgbph0yd2pvndq43pbss7dvj-python-2.7.10-bootstrapped-pip-7.1.2/lib/python2.7/site-packages/pip/locations.py", line 194, in distutils_scheme
    i.finalize_options()
  File "/nix/store/a45l8r0zqfr5jhqswqqcya28h59n63js-python2.7-setuptools-18.2/lib/python2.7/site-packages/setuptools-18.2-py2.7.egg/setuptools/command/install.py", line 38, in finalize_options
    orig.install.finalize_options(self)
  File "/nix/store/rfcsnszaw2jlkqdgg65h3phss8xa0mlp-python-2.7.10/lib/python2.7/distutils/command/install.py", line 264, in finalize_options
    "must supply either home or prefix/exec-prefix -- not both"
DistutilsOptionError: must supply either home or prefix/exec-prefix -- not both

These two flags should conflict and --prefix is superior because it will also affect data_files.

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost Nov 27, 2015

@domenkozar
Sorry, but I don't understand your comment. We have --target, --root and --prefix; which ones out of these do you mean when you say these two flags (...)? When you say --prefix is superior – what do you think it's superior to?

piotr-dobrogost commented Nov 27, 2015

@domenkozar
Sorry, but I don't understand your comment. We have --target, --root and --prefix; which ones out of these do you mean when you say these two flags (...)? When you say --prefix is superior – what do you think it's superior to?

@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 27, 2015

Contributor

@piotr-dobrogost all these flags mean different things, but some of them collide.

  • --root tells pip to install relative to the specified folder (instead of / )
  • --prefix specifies the global installation path, be it usr/ or usr/local or opt/
  • --target is an alias to --home and pip somehow extracts pure-lib directory out of it. I don't really understand why all this complex logic
Contributor

domenkozar commented Nov 27, 2015

@piotr-dobrogost all these flags mean different things, but some of them collide.

  • --root tells pip to install relative to the specified folder (instead of / )
  • --prefix specifies the global installation path, be it usr/ or usr/local or opt/
  • --target is an alias to --home and pip somehow extracts pure-lib directory out of it. I don't really understand why all this complex logic
@domenkozar

This comment has been minimized.

Show comment
Hide comment
@domenkozar

domenkozar Nov 27, 2015

Contributor

pip install --help says -t, --target <dir> Install packages into <dir>. which is as abstract as it can get :)

Contributor

domenkozar commented Nov 27, 2015

pip install --help says -t, --target <dir> Install packages into <dir>. which is as abstract as it can get :)

@davidedelvento

This comment has been minimized.

Show comment
Hide comment
@davidedelvento

davidedelvento Dec 15, 2015

Any ETA for when this will be included in a release? It is sorely needed and it is making me wish I could use "make install" instead....

davidedelvento commented Dec 15, 2015

Any ETA for when this will be included in a release? It is sorely needed and it is making me wish I could use "make install" instead....

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Dec 16, 2015

Contributor

@davidedelvento This will be included in pip 8 that should be released in january 2016, I'd guess.

Contributor

xavfernandez commented Dec 16, 2015

@davidedelvento This will be included in pip 8 that should be released in january 2016, I'd guess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment