--install-option (in the cli) gets passed to deps being installed

[s0undt3ch]

$ pip install --install-option='--salt-root=/foo' salt
Downloading/unpacking salt
  Downloading salt-2014.1.5.tar.gz (2.8MB): 2.8MB downloaded
  Running setup.py (path:/tmp/ve-test/build/salt/setup.py) egg_info for package salt
    2014.1.5

    package init file 'salt/templates/__init__.py' not found (or not a regular file)
Downloading/unpacking Jinja2 (from salt)
  Downloading Jinja2-2.7.3.tar.gz (378kB): 378kB downloaded
  Running setup.py (path:/tmp/ve-test/build/Jinja2/setup.py) egg_info for package Jinja2

    warning: no files found matching '*' under directory 'custom_fixers'
    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
Downloading/unpacking M2Crypto (from salt)
  Downloading M2Crypto-0.22.3.tar.gz (74kB): 74kB downloaded
  Running setup.py (path:/tmp/ve-test/build/M2Crypto/setup.py) egg_info for package M2Crypto

Downloading/unpacking msgpack-python (from salt)
  Downloading msgpack-python-0.4.2.tar.gz (114kB): 114kB downloaded
  Running setup.py (path:/tmp/ve-test/build/msgpack-python/setup.py) egg_info for package msgpack-python

Downloading/unpacking pycrypto (from salt)
  Downloading pycrypto-2.6.1.tar.gz (446kB): 446kB downloaded
  Running setup.py (path:/tmp/ve-test/build/pycrypto/setup.py) egg_info for package pycrypto

Downloading/unpacking PyYAML (from salt)
  Downloading PyYAML-3.11.tar.gz (248kB): 248kB downloaded
  Running setup.py (path:/tmp/ve-test/build/PyYAML/setup.py) egg_info for package PyYAML

Downloading/unpacking pyzmq>=2.2.0 (from salt)
  Downloading pyzmq-14.3.1.tar.gz (982kB): 982kB downloaded
  Running setup.py (path:/tmp/ve-test/build/pyzmq/setup.py) egg_info for package pyzmq

    no previously-included directories found matching 'docs/build'
    no previously-included directories found matching 'docs/gh-pages'
    warning: no previously-included files found matching 'bundled/zeromq/src/Makefile*'
    warning: no previously-included files found matching 'bundled/zeromq/src/platform.hpp'
    warning: no previously-included files found matching 'setup.cfg'
    warning: no previously-included files found matching 'zmq/libzmq*'
    warning: no previously-included files matching '__pycache__/*' found anywhere in distribution
    warning: no previously-included files matching '.deps/*' found anywhere in distribution
    warning: no previously-included files matching '*.so' found anywhere in distribution
    warning: no previously-included files matching '*.pyd' found anywhere in distribution
    warning: no previously-included files matching '.git*' found anywhere in distribution
    warning: no previously-included files matching '.DS_Store' found anywhere in distribution
    warning: no previously-included files matching '.mailmap' found anywhere in distribution
    warning: no previously-included files matching 'Makefile.am' found anywhere in distribution
    warning: no previously-included files matching 'Makefile.in' found anywhere in distribution
Downloading/unpacking MarkupSafe (from salt)
  Downloading MarkupSafe-0.23.tar.gz
  Running setup.py (path:/tmp/ve-test/build/MarkupSafe/setup.py) egg_info for package MarkupSafe

Downloading/unpacking apache-libcloud (from salt)
  Downloading apache_libcloud-0.14.1-py2.py3-none-any.whl (1.2MB): 1.2MB downloaded
Installing collected packages: salt, Jinja2, M2Crypto, msgpack-python, pycrypto, PyYAML, pyzmq, MarkupSafe, apache-libcloud
  Running setup.py install for salt
    2014.1.5
    package init file 'salt/templates/__init__.py' not found (or not a regular file)

    Installing salt-run script to /tmp/ve-test/bin
    Installing salt-minion script to /tmp/ve-test/bin
    Installing salt script to /tmp/ve-test/bin
    Installing salt-key script to /tmp/ve-test/bin
    Installing salt-call script to /tmp/ve-test/bin
    Installing salt-cp script to /tmp/ve-test/bin
    Installing salt-syndic script to /tmp/ve-test/bin
    Installing salt-ssh script to /tmp/ve-test/bin
    Installing salt-master script to /tmp/ve-test/bin
    Installing salt-cloud script to /tmp/ve-test/bin
  Running setup.py install for Jinja2
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: -c --help [cmd1 cmd2 ...]
       or: -c --help-commands
       or: -c cmd --help

    error: option --salt-root not recognized
    Complete output from command /tmp/ve-test/bin/python2 -c "import setuptools, tokenize;__file__='/tmp/ve-test/build/Jinja2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-gFIT34-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/ve-test/include/site/python2.7 --salt-root=/foo:
    usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]

   or: -c --help [cmd1 cmd2 ...]

   or: -c --help-commands

   or: -c cmd --help



error: option --salt-root not recognized

----------------------------------------
Cleaning up...
Command /tmp/ve-test/bin/python2 -c "import setuptools, tokenize;__file__='/tmp/ve-test/build/Jinja2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-gFIT34-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/ve-test/include/site/python2.7 --salt-root=/foo failed with error code 1 in /tmp/ve-test/build/Jinja2
Storing debug log for failure in /home/vampas/.pip/pip.log

[holbech]

I'm affected by this one as well. For now I'm resorting to ugly hacks using environment variables instead.

[johnyf]

Same here. What is the point of providing --install-option, if it will get passed to the dependencies and cause a failure ?
This makes it practically impossible to use --install-option for any package that has dependencies.

[johnyf]

It would be expected that --global-options are supposed to be passed to dependencies, but not --install-options. This is not the case in RequirementSet.install though.

[qwcode]

starting with pip v7, pip supports using --install-option per line (i.e. per requirement) in a requirements file, which provides a solution for your need. in this use case, --install-option is not passed down to dependencies.

so like this

`pip install -r requirements.txt'

where requirements.txt contains

salt --install-option='--salt-root-dir=/foo'

using --install-option directly in the cli currently makes it global to the whole execution of pip, i.e. all top-level requirements and subdependencies.

I guess it could be changed to only apply to top-level requirements, but I'm not sure given the new support in requirements files.

i.e. supposing pip install --install-option='blah' pkg1 pkg2 pkg3, it would only apply to pkg1, pkg2, and pkg3, but not any dependencies.

[johnyf]

Thanks a lot! I was aware of the newly introduced --install-option in requirement files, but after observing the command-line behavior, I assumed that it will be the same. This solves the main issue I had, which was to automate fetching a C library for compiling an extension of a dependency.

I cannot make this the default behavior (as it fetches a tarball external to PyPI, and also, the user may already have the library installed somewhere, and so want to set CFLAGS, LDFLAGS instead). So it has to be an extra option to setup.py. The only way to selectively pass an option to a dependency is through a requirements file, so no problem there.

The only other use case is when someone installs directly the package, and cannot pass this as a command line option via pip. Since this is a direct install, it wouldn't be very different to unpack the package and run setup.py directly, or to clone and do the same thing (after all, this seems to be the only way for passing compiler directives as environment variables).

A temporary workaround for this use case may be:

pip install foo
pip install --no-deps --force-reinstall --upgrade --install-option="--do-some-magic" foo

This should first install dependencies w/o passing any flags, then re-install the package only, passing --do-some-magic. (I added --upgrade, because it didn't work w/o it.)

[t-neumann]

Is the a resolution for that issue available yet?

This temporary workaround of first installing the package with dependencies and then re-installing the package with --install-option is not working for me:

Additional software is installed during the installation of the package and the --install-option could skip this step. But there's no use for this flag to simply install everything including the software first, just to then uninstall it again.

[pradyunsg]

I'm pretty sure that the only reason this has not been fixed is because no one has come around to fixing this.

[fpelliccioni]

Same problem here.
Does anyone know how to capture a --global-option from my setup.py?

[vcolano]

Going on 4 years this has been open and still no fix... any updates on when we might see some progress?

[ollamh]

Any updates on this one? For those who can't use requirements file this is crucial option to have.

[chrahunt]

Assuming we want to NOT pass options to dependencies by default (and possibly remove the ability entirely), one possible approach would be:

1. Introduce a flag --pass-options-to-deps that makes pip behave the way it currently does. --no-pass-options-to-deps makes pip not pass options to dependencies.
2. By default assume that we should pass options to dependencies, but if no explicit option was passed then print a deprecation warning stating that the behavior is changing in (if implemented right now) 20.2.
3. In 20.2, change the default and remove the deprecation warning

If we want to remove the options getting passed to all dependencies behavior then the deprecation warning should also be traced if --pass-options-to-deps is used.

One reason we may want to remove the behavior entirely is to transition fully over to config settings (#5771), which we could incorporate this kind of configuration into.

[chrahunt]

Removing bug label since this was almost certainly done intentionally initially for location-related options (like those deprecated in #7309).

[DiddiLeija]

Hi everyone! How is this issue going? Is this still happening with newer versions of pip? If not, can we close it?

[johnyf]

This issue remains the same (#1883 (comment)). Passing an --install-option from the command line to pip results in the option being propagated to packages that are dependencies of the package given to pip. Then, when pip attempts to install those dependencies, the installation fails, because the dependencies do not recognize the --install-option. For example:

pip install --verbose dd --install-option='--cudd' --install-option='--cudd_zdd'

raises an error (below intermediate output has been replaced with [...], and parts of paths replaced with <...>):

/<...>/.virtualenvs/py39_7/lib/python3.9/site-packages/pip/_internal/commands/install.py:229: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Using pip 21.2.4 from /<...>/.virtualenvs/py39_7/lib/python3.9/site-packages/pip (python 3.9)
Collecting dd
  Using cached dd-0.5.6.tar.gz (639 kB)

[...]

Installing collected packages: pyparsing, ply, pydot, psutil, astutils, dd
    Running command /<...>/.virtualenvs/py39_7/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/<...>/pip-install-3mq4ucs1/pyparsing_79f31590008e46d5a1d51adbd262851c/setup.py'"'"'; __file__='"'"'/<...>/pip-install-3mq4ucs1/pyparsing_79f31590008e46d5a1d51adbd262851c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /<...>/pip-record-xoxr6gev/install-record.txt --single-version-externally-managed --compile --install-headers /<...>/.virtualenvs/py39_7/include/site/python3.9/pyparsing --cudd --cudd_zdd
    usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
       or: setup.py --help [cmd1 cmd2 ...]
       or: setup.py --help-commands
       or: setup.py cmd --help

    error: option --cudd not recognized
    Running setup.py install for pyparsing ... error
ERROR: Command errored out with exit status 1: /<...>/.virtualenvs/py39_7/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/<...>/pip-install-3mq4ucs1/pyparsing_79f31590008e46d5a1d51adbd262851c/setup.py'"'"'; __file__='"'"'/<...>/pip-install-3mq4ucs1/pyparsing_79f31590008e46d5a1d51adbd262851c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /<...>/pip-record-xoxr6gev/install-record.txt --single-version-externally-managed --compile --install-headers /<...>/.virtualenvs/py39_7/include/site/python3.9/pyparsing --cudd --cudd_zdd Check the logs for full command output.

[johnyf]

The issue can be addressed by using process substitution:

pip install --verbose \
    -r <(echo "dd --install-option='--fetch' --install-option='--cudd' --install-option='--cudd_zdd'")

This works, because it emulates a requirements file, and thus changes the scope of --install-option (the options then do not propagate to dependencies: #1883 (comment)).

This approach is much better than the install-then-reinstall approach that I described above (#1883 (comment)).

Also, the install-then-reinstall approach is possible for those packages that have a pure Python variant, as well as an optimized alternative (i.e., same API implemented in both pure Python and using built extensions, e.g., Cython). The package dd is such an example.

For packages that necessarily need --install-option to be installed, process substitution addresses the issue (whereas install-then-reinstall would not).

This approach works also for installing from the local directory:

pip install --verbose --use-feature=in-tree-build \
    -r <(echo ". --install-option='--fetch' --install-option='--cudd' --install-option='--cudd_zdd'")

[johnyf]

As of pip == 21.3, the above (#1883 (comment)) call for installing from the local directory becomes:

pip install --verbose \
    -r <(echo ". --install-option='--fetch' --install-option='--cudd' --install-option='--cudd_zdd'")

[johnyf]

Process substitution (suggested above: #1883 (comment)) is not in POSIX. For example, it is not available in /bin/sh, and so not inside makefiles, unless workarounds are used. Environment variables are another option, when process substitution is unavailable, to emulate the requirements-file behavior of --install-option, from the command-line. This observation motivates having the command-line --install-option behave the same way that the requirements-file --install-option behaves.

(It seems useful to also allow multiple command-line parameters and arguments to be passed within the argument of a single occurrence of --install-option, so that the users would need to write --install-option fewer times on each invocation of pip install. Perhaps shlex is relevant to this possibility.)

[wudstrand]

Any update on this issue?

[sbidoul]

It is highly unlikely that we will change this behaviour, since the --install-option and --global-option force pip to use setup.py's install command, which is deprecated.

As a temporary workaround, you can try creating a requirements.in file with one line:

salt --install-option='--salt-root=/foo'

and install with pip install -r requirements.in.

OTOH, now might be a good time to confirm this is the behaviour we want for --config-settings too. cc/ @pypa/pip-committers.

[pfmoore]

In my view, yes, we should continue with the current behaviour:

1. Specifying config settings on the command line should apply to all builds by default
2. Specifying config settings in a requirements file should apply just to the given requirement, to give users a way to specify requirements on a per-user basis

The first case is because otherwise, how would you specify an option that applies to everything? The second is because that is how any per-requirement options are set currently.

Any other approach would need justifying (as it's a deviation from how we handle every similar option) and given that the semantics of config options are currently entirely backend-dependent, we can't realistically make such a judgement. If there's a case for a different approach, it should be made on Discourse, so that all backend maintainers have a chance to get involved, and any conclusions can be properly recorded (as standards, if necessary).

[pradyunsg]

Discourse

https://discuss.python.org/c/packaging/14

[sbidoul]

how would you specify an option that applies to everything

Personally I'm not quite sure it makes sense to pass the same config settings to all builds. But I lack concrete use cases to help form an opinion.

[uranusjr]

For --config-settings (and the legacy --build-option), automatically cascading to dependencies actually is the default for a lot of build systems out there. This kind of things is usually for build-time flags, and for a lot of those, you actually need the flags to be cascaded, otherwise your dependencies won’t link correctly. So the general solution in these build systems is generally tell packagers they should be careful to handle unknown options correctly, which I feel should also be how we approach this.

[pfmoore]

https://discuss.python.org/c/packaging/14

Broken link?

But I lack concrete use cases to help form an opinion.

So do I. As @uranusjr says, other build systems cascade by default (as does the common "set environment variables to configure your compiler" model). But it's not clear that config options in Python will correspond to that sort of option. Until build backends start making use of config options, we're guessing.

As far as I know, only setuptools implements config options:

1. For configuring the editable mode. I have no feel for whether this makes sense to cascade, but my first guess is that it does no harm to cascade it. Dependencies aren't editable, after all.
2. As a way to pass the historical build option flags - for which "preserve existing behaviour" is certainly as good as we have now, and possibly as good as we can get, given that setuptools lets projects use any semantics they want for build options. Given the original issue raised here, maybe setuptools should default to ignoring unknown build options¹, rather than failing, but that's a matter for them.

Footnotes

1. Which is how config options work. ↩