--process-dependency-links not recognized in requirements.txt

[rassie]

No description provided.

[digitalresistor]

This commit should be reversed.

Doing a pip install on a requirements.txt should not be able to go download source code from anywhere on the net if found in a setup.py file, only PyPi. If the user adds that as a specific command line while using pip, that's one thing, but dependency links were considered harmful.

Specifically see this comment for the security concerns: #2023 (comment)

Also, they were to be deprecated.

[goodwillcoding]

I completely agree with @bertjwregeer ... an option that lets pip wander around the internet like this needs to be something a user explicitly enabled.

[xavfernandez]

I don't see any security concern: the --process-dependency-links does not magically pop in the requirements.txt file.

If you run pip install --requirement on some file without checking its content, then you're likely to have other security problems...
Direct link like http://badomain.com/lets_run_some_arbitrary_setup_py_code.tar.gz#egg=pip are also allowed in requirements files.

[dstufft]

Pip does not read a requirements file that was not even explicitly passed in by the user invoking pip, or which was explicitly included in said requirements file. It has the same trust level as typing stuff on the CLI and it is appropriate to allow this option there (as much as it's appropriate to allow this option anywhere).

[digitalresistor]

There are plenty of examples on the internet where one of the steps is to download a repository or code, and pip install a requirements.txt

That this now has the side effect of following dependency links worries me. Especially since I can't know that package x.y.z from PyPi has dependency links or not... or does this limit it to JUST the package that has --process-dependency-links added to it, and not the whole requirements.txt file (which is how the change reads now)

[dstufft]

It's the whole installation process.

[digitalresistor]

So just 1 package in the requirements.txt requires it, but it is now followed for every single package. Auditing that becomes a nightmare. At least when the user had to manually type it out on the command line there was a lot more thought that went into it (and they could do it for the one package that failed), instead of scattershot across all of the packages.

I understand what you are saying about requirements.txt being explicitly run by the user, and that links to bad eggs are allowed, but links to bad eggs are easier to spot than a dependency link in the requires of a package 2 dependencies down from where the user starter.

[dstufft]

I'm not going to argue that dependency links themselves are a good feature, they aren't. In the future we'll have a better feature for them but in the meantime we don't want to punish people who are using dependency links within their own organizations (e.g., not from PyPI, but to operate in a sort of "repo-less" mode) because we don't have that feature done yet.

[digitalresistor]

To better safeguard users, could packages download from PyPi be limited so that they don't have their dependency_links followed. This way only packages in repo-less mode will have them followed (if this is set in requirements.txt).

That will at least make it easier to audit the requirements.txt files.

[xavfernandez]

I'm sorry but I don't follow. Maybe it is not clear.

This does not implicitly enable --process-dependency-links on all requirement files.

This change allows to add the option --process-dependency-links (as a new text line) in a requirement file so that the user only has to type pip install -r requirements_with_all_my_options.txt instead of pip install -r requirements.txt --process-dependency-links before.

So before, you audited the requirements.txt files and had no idea whether the user would add the --process-dependency-links option to its command line.

Now, you will audit the requirements.txt, will raise an alert if you see the --process-dependency-links option in the audited file and still have no idea whether the user will add the --process-dependency-links option to its command line ^^

[digitalresistor]

@xavfernandez I understand that this does not enable it for all requirement files...

But --process-dependency-links in a requirements.txt file now applies to all packages inside that requirements file. So now if I audit the file, I see "--process-dependency-links" and I need to manually go through EVERY single line in the file and find out the following:

1. What packages/eggs are installed from where
2. Do any of the above have dependency links?
3. Find all dependencies for the packages/eggs
4. Goto step 2 until the tree is exhausted
5. Consider if it is all acceptable or not, make sure to pin to a particular version for EVERYTHING to make sure that future changes don't add a dependency_link.

i.e. before it was fairly easy to see if we were going to scattershot go across the internet to download stuff:

1. See what packages/eggs are installed from where
2. See bad-egg directly listed in the requirements file

If it was a requirement for a company to use --process-dependency-links they would have to add it to the command line, but it would have to be a conscious choice.

For example, I have an internal project that does depend on --process-dependency-links, but I won't ever add that particular project to my requirements.txt along with a --process-dependency-links because I don't want pip downloading stuff from around the internet. Instead my build process does:

pip install -r requirements.txt
# Assume requirements.txt includes all the requirements for the project
pip install --process-dependency-links privateproject.tar.gz

This way I can be sure that only the dependency links that I explicitly control (in privateproject.tar.gz or more likely our git repo directly) are followed, and everything else is installed from PyPi without following dependency links (even if a project does use them).

My privateproject.tar.gz doesn't depend on ANY packages on PyPi and only on my dependency_links. Yes this means someone running python setup.py dev or something along those lines won't download all the dependencies, but it is a way I can protect me and my infrastructure from outside code being downloaded from all over the web.

[pfmoore]

Surely, though, you can just add a further rule to your company audit rules, that --process-dependency-links is not allowed in requirements files? Then you're in just the same situation as currently. This doesn't sound like a reason we should remove the feature from pip.

[toddjames]

I was just trying to figure out why my dependency links weren't working for my private repositories and I found the --process-dependency-links flag. I was happy to see that it fixed the issue I was having, but dismayed that it was deprecated.

After some searching to see what I should be using instead, I found this issue. My question is this: is there a tracking issue that I can follow for the replacement feature I will need to use in the future once dependency links processing is removed? I'd like to stay on top of that so I can be prepared when it is removed.

[xavfernandez]

The dependency links will likely be replaced by what PEP508 allows.