Determine what should be done about --(install|global)-option with Wheels #2677

Open
dstufft opened this Issue Apr 13, 2015 · 24 comments

Projects

None yet
@dstufft
Member
dstufft commented Apr 13, 2015

Currently pip has --install-option and --global-option to make it possible to pass flags to the setup.py invocation. However these currently get ignored completely when installing from Wheel. This previously wasn't a big deal because commonly these options are only used for compiled packages and those often don't have Wheels on the platforms you might need these options on. However now in pip 7.0 via #2618 we now attempt to automatically build wheels for all projects. This means that, assuming we can build a wheel, these options are just completely ignored.

So what do we do with these options? Do we make their use imply --no-use-wheel? Do we attempt to convert them? Get rid of them?

@dstufft dstufft added this to the 7.0 milestone Apr 13, 2015
@qwcode
Contributor
qwcode commented Apr 13, 2015

I had previously opened #1716 for this. I'll close that.

@qwcode
Contributor
qwcode commented Apr 13, 2015

the only extra tidbit from #1716 is just the awareness that install-options are recognized when placed in distutils config files.

@pfmoore
Member
pfmoore commented Apr 13, 2015

The main use I've made of --install-option (actually --global-option) is the obscene hack:

pip wheel --global-option "--with-libyaml" --global-option "build_ext" --global-option "-DYAML_DECLARE_STATIC" --global-option "-LC:\Work\Scratch\pyyaml\yaml-0.1.5\win32\vs2008\Output\Release\lib" --global-option "-IC:\Work\Scratch\pyyaml\yaml-0.1.5\include"  .\PyYAML-3.10.tar.gz

This lets me add build options to the extension build (by injecting an explicit build_ext step into the setup command). The reason for doing this is to avoid having to download and edit the source distribution just to add some settings into setup.cfg.

Maybe having a means to add section.key=value items into the source archive's setup.cfg when doing the build would be a more general solution, that is explicitly a "build step" option.

Something like --build-setting [build_ext]define=YAML_DECLARE_STATIC, maybe? Then there's no implication that it has an effect when installing from wheel, it's clearly a build-only option.

(Whether this would be viable depends on what uses other people have for --install-option and --global-option...)

@xavfernandez
Contributor

According to the recently merged #2537 , people seem to have a need for these options.

An implied --no-use-wheel when --install-option or --global-option are used seems easy enough.

@rbtcollins
Contributor

So the problem (and its not unique to the wheel autobuilding) is scoping those options.

We can and should use the options from requirements file binding to pass through to the wheels. But the top level --install-option is only defined as applying to the thing being installed, isn't it? Or is it global?

Anyhow, if someone can define 'right' here, I'll happily write the code for it.

Remember too that folk can always use 'pip wheel' and then -f wheelhouse to get exactly what they want.

@dstufft
Member
dstufft commented Apr 17, 2015

Scoping is certainly a problem, and I've thought before that maybe we should remove those options and instead we should do something like --build-option:myproject="--bar" or something that handles the scoping.

There's also the question of what --install-option and --global-option even means in the future (tm) where we're trying to move to a situation where we only ever install from wheels and the only thing we use a source distribution for is building a wheel. In that scenario there is nothing to pass a --install-option too (and ideally anything people are using it for we'd either explicitly say we don't support, or we'd get a better, generic option for) and --global-option doesn't really make much sense because if you're only invoking a source distribution to build a wheel there is no global, only build.

On the other hand, we might not want to define the API for --build-option util the metabuild system is done so we can implement it in terms of that (e.g. if the API for metabuild is CLI based, we'd want to pass --foo options, if it's Pythonic "call this function" based, we'd want some way to translate CLI opts into Python opts).

In the short term it might just make the most sense that using --install-option and/or --global-option disables the automatic wheel cache and then open up a new issue to deprecate and replace those with a --build-option.

@rbtcollins
Contributor

@dstufft and I discussed this on IRC, he says that the resolution is: "--install-option, --global-option, --build-option will all disable the consumption of wheels, and any autobuilding of wheels." And that follow on work can be done to assess what we really want/need longer term.

@qwcode
Contributor
qwcode commented Apr 21, 2015

the question of what --install-option and --global-option even means in the future (tm)
where we're trying to move to a situation where we only ever install from wheels

why doesn't --install-option make sense when installing from wheels?
regardless of the format, you might want to set different paths for the install.

feels like a loss for wheel installs to have less flexibility than sdists.
(see twang817/pip@e3338f7 as an example of a pip user hacking this themselves, and wanting us to add this support)

how would this get sorted out in the "longer term"?

@dstufft
Member
dstufft commented Apr 21, 2015

Because --install-option, --global-option, and --build-option is conceptually "pass this option to the setup.py execution", it's documented that way and I'm sure people are using it that way. When we're installing from a Wheel we have no setup.py to execute and thus nothing to pass options to. Further more as we move to the meta build system we (most likely) aren't even going to be guaranteed to have a setup.py that exists for even a source distribution.

Longer term I think that the way to solve this is something like that:

  • Figure out what people are using --install-option for that makes sense at all for wheels (one example I've seen is to be able to direct scripts to a different location) and expose a top level pip flag that supports that same use case. When installing from a source distribution we'll pass those options to setup.py and when installing from a wheel we'll take them into consideration for computing paths.
  • Figure out what people are using --global-option for, and if it still makes sense in the new world order. I don't know off the top of my head what people are using this for, so I can't speak to what this would look like.
  • Figure out the meta build system and what API is available for pip to call to pass options into the build system, once that is nailed down figure out a way that we can expose those build options to both pip wheel and pip install.
@dstufft
Member
dstufft commented Apr 21, 2015

In short, the basic problem with those options is that they are explicitly tied to executing setup.py and one of their use cases is for people being able to pass random arbitrary options to their setup.py invocations.

@qwcode
Contributor
qwcode commented Apr 21, 2015

these all seem like valid uses of install-options that could be applied to wheels.

 --home                               (Unix only) home directory to install
                                       under
  --install-base                       base installation directory (instead of
                                       --prefix or --home)
  --install-platbase                   base installation directory for
                                       platform-specific files (instead of --
                                       exec-prefix or --home)
  --root                               install everything relative to this
                                       alternate root directory
  --install-purelib                    installation directory for pure Python
                                       module distributions
  --install-platlib                    installation directory for non-pure
                                       module distributions
  --install-lib                        installation directory for all module
                                       distributions (overrides --install-
                                       purelib and --install-platlib)
  --install-headers                    installation directory for C/C++
                                       headers
  --install-scripts                    installation directory for Python
                                       scripts
  --install-data    

yes, these options are conceptually tied to setuptools, but don't have to be literally. we'd map them over to our distutils scheme utility.

for sure, it's weird to use setuptools options when there's no "setup.py" involved, but the reality is things are fractured right now, and maybe it's ok to break a conceptual line, to give people support for install flexibility in the mean time.

to be clear, I'm ok with the approach of ignoring them for wheels for now, but just making it clear, that we could do otherwise, if a lot of people wanted this.

@dstufft
Member
dstufft commented Apr 21, 2015

We already support --root, I'm not sure what --home does, and it seems logical to me to add the top level arguments there straight to pip install.

@qwcode
Contributor
qwcode commented Apr 21, 2015

yes, agreed, I'd like to see new top-level options that are setuptools-agnostic

@qwcode
Contributor
qwcode commented Apr 21, 2015

--home is for the "home scheme". it's similar to the user scheme, but free standing, and not on the python path by default

@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 21, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
506aeae
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 22, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
ee4a90e
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 22, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
7f9867a
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 22, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
ce40e97
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 22, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
288ccea
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 23, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
bb182ac
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 23, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
9212553
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 23, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
fc2c6b0
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 24, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
cf73acc
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 24, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
54dda99
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 24, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
f1c0de7
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 24, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
af084d9
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 24, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
6cfb998
@rbtcollins rbtcollins pushed a commit to rbtcollins/pip that referenced this issue Apr 24, 2015
Robert Collins Issue #2677: Disable wheels for setup.py options.
Using --install-options, --build-options, --global-options changes
the way that setup.py behaves, and isn't honoured by the wheel code.
The new wheel autobuilding code made this very obvious - disable
the use of wheels when these options are supplied.
582af6e
@dstufft
Member
dstufft commented May 9, 2015

Going to drop this off a release blocker because @rbtcollins fixed it so we don't autobuild wheels if these options are specified, so that removes the immediate need for this.

@dstufft dstufft removed the release-blocker label May 9, 2015
@dstufft dstufft removed this from the 7.0 milestone May 9, 2015
@asfaltboy

I may be off here, but I think it make sense to also not build wheels if a setup.cfg includes any of the install-options mentioned above. I just had the unfortunate "switched to wheels stuff broke" scenario with this.

@BrenBarn

Someone just asked a stackoverflow question about this: http://stackoverflow.com/questions/31979029/installing-a-whl-python-package-into-a-specific-directory-other-than-the-defaul

is the current state of affairs that it is not possible to install a wheel to a non-default location? That seems like a problem that should be remedied quickly.

@domenkozar
Contributor

I'd like to tell pip to install wheels at a certain prefix, which is currently not possible at all (there is --target but then there is no way to tell where to install scripts).

@dstufft
Member
dstufft commented Nov 15, 2015

--root?

Sent from my iPhone

On Nov 15, 2015, at 5:02 PM, Domen Kožar notifications@github.com wrote:

I'd like to tell pip to install wheels at a certain prefix, which is currently not possible at all (there is --target but then there is no way to tell where to install scripts).


Reply to this email directly or view it on GitHub.

@domenkozar
Contributor

That's not exactly working:

pip install *.whl --root=/nix/store/0nr8sy9bgngqi5ijk2izxxmlm4dmvx25-python2.7-pbr-1.8.1/
...
/nix/store/0nr8sy9bgngqi5ijk2izxxmlm4dmvx25-python2.7-pbr-1.8.1/nix/store/rfcsnszaw2jlkqdgg65h3phss8xa0mlp-python-2.7.10/bin/pbr
...

root and prefix are different, each meaning:

  • prefix: what path goes before bin, lib, include, etc
  • root: where to put all files including prefix (I think this option doesn't make much sense once you have prefix, but I guess it's around for legacy reasons)
@domenkozar
Contributor

Opened #3252 to address --prefix

@matham
Contributor
matham commented Nov 26, 2015

I just ran into this. We're trying to use namespace packages as a way to separate our optional binary deps on Windows.

For each of the deps, the binary parts are installed to e.g. python/shared or python/include etc. Also each dep has a module that is installed to kivy/deps/module_name. Where kivy is the main project.

The problem is when the the root project, kivy, is not in site-packages but in an alternate location. I would need to tell wheel to install the namespace module to the alternate kivy place. These binary packages are wheels, so the pip stuff doesn't work.

Using --root doesn't work as it creates intermediate site-packages directories. Anyway, root installs both the python stuff and the bins to the new location, which is not what I wanted.

Although maybe this is an issue with namespace packages, and not wheel?

Maybe when building, wheel should have the option to specify which files are allowed to be located elsewhere and allow during the install to specify the location? This would get around the no existing setup.py problem as with this it may not be needed anymore for these kind of settings.

@LeoHuckvale

+1 for "--install-scripts" with Wheels.

Sorry to add to the noise, but I thought I might share a use-case.

We have an application which we drive with a Python package (library and accompanying scripts). We want to ship the Python package (and updates) in a format which is clean (for distribution to customers), so a source distribution isn't an option. The library component installs fine into site-packages (which is what we want), but we need to install the scripts to a custom location, where our application looks for them.

@anntzer
anntzer commented Sep 21, 2016

It would be nice to clarify the error message when the pip command invoked is actually pip wheel itself. Note that this use if explictly mentioned in the docs (even though discouraged) at https://pip.pypa.io/en/stable/reference/pip_wheel/#customising-the-build:

pip wheel --global-option bdist_ext --global-option -DFOO wheel

This currently outputs the nonsensical(?) warning

/usr/lib/python3.5/site-packages/pip/commands/wheel.py:126: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.

Obviously this use of wheels is not (should not be?) disabled.

@GhostLyrics GhostLyrics referenced this issue in echocat/puppet-graphite Nov 16, 2016
Open

ubuntu 16.04 not supported #298

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment