Command-line arguments to pip install should take precedence over options configured in requirements.txt

[andornaut]

• Pip version: 8.1.1
• Python version: 2.7.11
• Operating System: Ubuntu 16.04

Description:

Command-line arguments to pip install should take precedence over options configured in requirements.txt files. For example, if --index-url is specified on the command-line, then it should override the index-url set in a requirements.txt file.

What I've run:

pushd $(mktemp -d) >/dev/null
virtualenv -q .
source bin/activate >/dev/null
pip install -qU pip
echo "\
--index-url=https://example.com/DONOTQUERY
examplethatdoesnotexist
" > r.txt
pip install --isolated --index-url=https://pypi.python.org/simple/ -vr r.txt > log
grep DONOTQUERY -B 2 log  && echo "Queried https://example.com/DONOTQUERY"
grep pypi log || echo "Did not query pypi.python.org"

This outputs

Collecting examplethatdoesnotexist (from -r r.txt (line 2))
  1 location(s) to search for versions of examplethatdoesnotexist:
  * https://example.com/DONOTQUERY/examplethatdoesnotexist/
  Getting page https://example.com/DONOTQUERY/examplethatdoesnotexist/
  Starting new HTTPS connection (1): example.com
  "GET /DONOTQUERY/examplethatdoesnotexist/ HTTP/1.1" 404 606
  Could not fetch URL https://example.com/DONOTQUERY/examplethatdoesnotexist/: 404 Client Error: Not Found for url: https://example.com/DONOTQUERY/examplethatdoesnotexist/ - skipping
Queried https://example.com/DONOTQUERY
Did not query pypi.python.org

I would expect that the location searched for examplethatdoesnotexist would be the one specified on the command line "https://pypi.python.org/simple/" not the one specified in the requirements.txt file "https://example.com/DONOTQUERY".

[pradyunsg]

@pypa/pip-committers Thoughts?

[pfmoore]

The order of priority should certainly be clearly defined in the docs. I haven't checked - is it?

As to whether the current order is correct or not, I'm unclear. What is established practice for other, similar programs? I'd like to see a non-artificial use case that demonstrates why the proposed behaviour is better than the currently implemented one.

(As a counter to the proposal, if you have a requirements file with private packages and an --index-url line pointing to the private index holding them, it makes no sense for the user to be able to override the index URL, so requirements file taking precedence seems logical there).

[foobacca]

We have a case where we have a local pip server which we use in development. So our requirements file includes --extra-index-url

For deployment we build the wheels and package them up in an RPM, and have the RPM spec file install the requirements with --no-index --find-link=... - with the expectation that these command line arguments will override the arguments in a file.

A quick test using my local venv and an internal package. First uninstall it and check pip install with no flags fails to install it:

% pip uninstall internal_package
Uninstalling internal_package-1.1.6:
  Would remove:
    /home/hdowner/.venv/cindexd/bin/internal_package
    /home/hdowner/.venv/cindexd/lib/python2.7/site-packages/internal_package-1.1.6.dist-info/*
    /home/hdowner/.venv/cindexd/lib/python2.7/site-packages/internal_package/*
Proceed (y/n)? y
  Successfully uninstalled internal_package-1.1.6

% pip install internal_package
Collecting internal_package
  Could not find a version that satisfies the requirement internal_package (from versions: )
No matching distribution found for internal_package

Now try using both flags on the command line:

% pip install --no-index --extra-index-url https://pip.mycompany/simple internal_package
Collecting internal_package
  Could not find a version that satisfies the requirement internal_package (from versions: )
No matching distribution found for internal_package

Which is what we want. (Although what would be expected when both flags are used is an another question - maybe it should be an error? At any rate I don't really mind.)

Next try using the set up we have, with --no-index on the command line and the --extra-index-url in a requirements file:

% cat test-req.txt
--extra-index-url https://pip.mycompany/simple

internal_package

% pip install --no-index -r test-req.txt
Looking in indexes: https://pip.mycompany/simple
Collecting internal_package (from -r test-req.txt (line 3))
  Downloading https://pip.mycompany/packages/internal_package-1.1.7-py2.py3-none-any.whl
Requirement already satisfied: requests==2.13.0 in /home/hdowner/.venv/cindexd/lib/python2.7/site-packages (from internal_package->-r test-req.txt (line 3)) (2.13.0)
Requirement already satisfied: six~=1.9 in /home/hdowner/.venv/cindexd/lib/python2.7/site-packages (from internal_package->-r test-req.txt (line 3)) (1.11.0)
Installing collected packages: internal_package
Successfully installed internal_package-1.1.7

This case is where I want the command line arguments to override the flags in the requirements file.

As to what other systems do, I found this answer on stack overflow about precedence which broadly agrees with my understanding, as does Microsoft ASP.NET and these two python packages.

[andornaut]

it makes no sense for the user to be able to override the index URL, so requirements file taking precedence seems logical there

IMHO, it would make sense for the user to be able to override any config. Perhaps, the user is a CI system with its own package index cache.

Shouldn't the user be the ultimate decision maker? I struggle to recall software where this is not the case.

[pradyunsg]

Honestly, I'm personally not sure how we want to handle this situation. I do think this is a valid feature request that would need to be investigated further.

[ddormer]

I have a use case for this. We have an internal index which is accessible by developers and our CI, however the index is overridden in a deployment environment in order to use a different internal index that is always accessible to production servers in the case of network outages.

I believe it's quite standard for command-line arguments to override config options.

[dquitmann-op]

First: thanks for the awesome work with pip!

I just stumbled across this and while it seems to have not many attention, I think it really is a game breaker.

The order of priority should certainly be clearly defined in the docs. I haven't checked - is it?

Yes and no: https://pip.pypa.io/en/stable/user_guide/#config-precedence clearly described the precedence of config files, environment variables and cli arguments, but neither there nor in https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format the precedence of in-line options in requirements files is clarified.

My humble option: these should be in between env variables and cli arguments, thus overwriting all configuration done in ones environment (config files and env variables) but being overwritten by explicit cli arguments.

I have the following usecase (similar to #3614 (comment)):

• we have a custom PyPI server and the requirements file contains the --index-url
• I wanted to install the project on a system without access to the internal PyPI server, so I downloaded all packages (pip download -r requirements.txt) and wanted to install them on the target system with pip install --no-index --find-links <path_with_packages> --requirement requirements.txt, but the index-url in the file overwrites the no-index in the cli

[niander]

I also would like to see more clear precedence order in the documentation. As @dquitmann-op pointed out, the current documentation doesn't mention that configuration in requirements.txt take precedence over everything.

Nevertheless, I believe the configuration in the requirements.txt should not take precedence over command-line and global environment variables for security and other reasons.

[pradyunsg]

Happy to accept a PR adding this information to the documentation! ^.^

[niander]

@pradyunsg I am happy to do it but I am not 100% sure if all options in requirements.txt take precedence over command line, environment and local configuration.

Still, I wouldn't consider this issue as just documentation. I think the suggestion of removing the precedence of options in requirements.txt is relevant. Do you agree with that?

[victorusachev]

First of all, thanks for a job well done with pip!
Well I just ran into the same issues as described here.