New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--process-dependency-links not recognized in requirements.txt #1274

Closed
rassie opened this Issue Oct 31, 2013 · 14 comments

Comments

Projects
None yet
7 participants
@rassie

rassie commented Oct 31, 2013

No description provided.

xavfernandez added a commit to xavfernandez/pip that referenced this issue Aug 14, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Aug 14, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Sep 3, 2015

xavfernandez added a commit to xavfernandez/pip that referenced this issue Sep 5, 2015

@bertjwregeer

This comment has been minimized.

Show comment
Hide comment
@bertjwregeer

bertjwregeer Jan 20, 2016

This commit should be reversed.

Doing a pip install on a requirements.txt should not be able to go download source code from anywhere on the net if found in a setup.py file, only PyPi. If the user adds that as a specific command line while using pip, that's one thing, but dependency links were considered harmful.

Specifically see this comment for the security concerns: #2023 (comment)

Also, they were to be deprecated.

bertjwregeer commented Jan 20, 2016

This commit should be reversed.

Doing a pip install on a requirements.txt should not be able to go download source code from anywhere on the net if found in a setup.py file, only PyPi. If the user adds that as a specific command line while using pip, that's one thing, but dependency links were considered harmful.

Specifically see this comment for the security concerns: #2023 (comment)

Also, they were to be deprecated.

@goodwillcoding

This comment has been minimized.

Show comment
Hide comment
@goodwillcoding

goodwillcoding Jan 20, 2016

I completely agree with @bertjwregeer ... an option that lets pip wander around the internet like this needs to be something a user explicitly enabled.

goodwillcoding commented Jan 20, 2016

I completely agree with @bertjwregeer ... an option that lets pip wander around the internet like this needs to be something a user explicitly enabled.

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Jan 20, 2016

Contributor

I don't see any security concern: the --process-dependency-links does not magically pop in the requirements.txt file.

If you run pip install --requirement on some file without checking its content, then you're likely to have other security problems...
Direct link like http://badomain.com/lets_run_some_arbitrary_setup_py_code.tar.gz#egg=pip are also allowed in requirements files.

Contributor

xavfernandez commented Jan 20, 2016

I don't see any security concern: the --process-dependency-links does not magically pop in the requirements.txt file.

If you run pip install --requirement on some file without checking its content, then you're likely to have other security problems...
Direct link like http://badomain.com/lets_run_some_arbitrary_setup_py_code.tar.gz#egg=pip are also allowed in requirements files.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Jan 20, 2016

Member

Pip does not read a requirements file that was not even explicitly passed in by the user invoking pip, or which was explicitly included in said requirements file. It has the same trust level as typing stuff on the CLI and it is appropriate to allow this option there (as much as it's appropriate to allow this option anywhere).

Member

dstufft commented Jan 20, 2016

Pip does not read a requirements file that was not even explicitly passed in by the user invoking pip, or which was explicitly included in said requirements file. It has the same trust level as typing stuff on the CLI and it is appropriate to allow this option there (as much as it's appropriate to allow this option anywhere).

@bertjwregeer

This comment has been minimized.

Show comment
Hide comment
@bertjwregeer

bertjwregeer Jan 20, 2016

There are plenty of examples on the internet where one of the steps is to download a repository or code, and pip install a requirements.txt

That this now has the side effect of following dependency links worries me. Especially since I can't know that package x.y.z from PyPi has dependency links or not... or does this limit it to JUST the package that has --process-dependency-links added to it, and not the whole requirements.txt file (which is how the change reads now)

bertjwregeer commented Jan 20, 2016

There are plenty of examples on the internet where one of the steps is to download a repository or code, and pip install a requirements.txt

That this now has the side effect of following dependency links worries me. Especially since I can't know that package x.y.z from PyPi has dependency links or not... or does this limit it to JUST the package that has --process-dependency-links added to it, and not the whole requirements.txt file (which is how the change reads now)

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Jan 20, 2016

Member

It's the whole installation process.

Member

dstufft commented Jan 20, 2016

It's the whole installation process.

@bertjwregeer

This comment has been minimized.

Show comment
Hide comment
@bertjwregeer

bertjwregeer Jan 20, 2016

So just 1 package in the requirements.txt requires it, but it is now followed for every single package. Auditing that becomes a nightmare. At least when the user had to manually type it out on the command line there was a lot more thought that went into it (and they could do it for the one package that failed), instead of scattershot across all of the packages.

I understand what you are saying about requirements.txt being explicitly run by the user, and that links to bad eggs are allowed, but links to bad eggs are easier to spot than a dependency link in the requires of a package 2 dependencies down from where the user starter.

bertjwregeer commented Jan 20, 2016

So just 1 package in the requirements.txt requires it, but it is now followed for every single package. Auditing that becomes a nightmare. At least when the user had to manually type it out on the command line there was a lot more thought that went into it (and they could do it for the one package that failed), instead of scattershot across all of the packages.

I understand what you are saying about requirements.txt being explicitly run by the user, and that links to bad eggs are allowed, but links to bad eggs are easier to spot than a dependency link in the requires of a package 2 dependencies down from where the user starter.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Jan 20, 2016

Member

I'm not going to argue that dependency links themselves are a good feature, they aren't. In the future we'll have a better feature for them but in the meantime we don't want to punish people who are using dependency links within their own organizations (e.g., not from PyPI, but to operate in a sort of "repo-less" mode) because we don't have that feature done yet.

Member

dstufft commented Jan 20, 2016

I'm not going to argue that dependency links themselves are a good feature, they aren't. In the future we'll have a better feature for them but in the meantime we don't want to punish people who are using dependency links within their own organizations (e.g., not from PyPI, but to operate in a sort of "repo-less" mode) because we don't have that feature done yet.

@bertjwregeer

This comment has been minimized.

Show comment
Hide comment
@bertjwregeer

bertjwregeer Jan 20, 2016

To better safeguard users, could packages download from PyPi be limited so that they don't have their dependency_links followed. This way only packages in repo-less mode will have them followed (if this is set in requirements.txt).

That will at least make it easier to audit the requirements.txt files.

bertjwregeer commented Jan 20, 2016

To better safeguard users, could packages download from PyPi be limited so that they don't have their dependency_links followed. This way only packages in repo-less mode will have them followed (if this is set in requirements.txt).

That will at least make it easier to audit the requirements.txt files.

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Jan 20, 2016

Contributor

I'm sorry but I don't follow. Maybe it is not clear.

This does not implicitly enable --process-dependency-links on all requirement files.

This change allows to add the option --process-dependency-links (as a new text line) in a requirement file so that the user only has to type pip install -r requirements_with_all_my_options.txt instead of pip install -r requirements.txt --process-dependency-links before.

So before, you audited the requirements.txt files and had no idea whether the user would add the --process-dependency-links option to its command line.

Now, you will audit the requirements.txt, will raise an alert if you see the --process-dependency-links option in the audited file and still have no idea whether the user will add the --process-dependency-links option to its command line ^^

Contributor

xavfernandez commented Jan 20, 2016

I'm sorry but I don't follow. Maybe it is not clear.

This does not implicitly enable --process-dependency-links on all requirement files.

This change allows to add the option --process-dependency-links (as a new text line) in a requirement file so that the user only has to type pip install -r requirements_with_all_my_options.txt instead of pip install -r requirements.txt --process-dependency-links before.

So before, you audited the requirements.txt files and had no idea whether the user would add the --process-dependency-links option to its command line.

Now, you will audit the requirements.txt, will raise an alert if you see the --process-dependency-links option in the audited file and still have no idea whether the user will add the --process-dependency-links option to its command line ^^

@bertjwregeer

This comment has been minimized.

Show comment
Hide comment
@bertjwregeer

bertjwregeer Jan 20, 2016

@xavfernandez I understand that this does not enable it for all requirement files...

But --process-dependency-links in a requirements.txt file now applies to all packages inside that requirements file. So now if I audit the file, I see "--process-dependency-links" and I need to manually go through EVERY single line in the file and find out the following:

  1. What packages/eggs are installed from where
  2. Do any of the above have dependency links?
  3. Find all dependencies for the packages/eggs
  4. Goto step 2 until the tree is exhausted
  5. Consider if it is all acceptable or not, make sure to pin to a particular version for EVERYTHING to make sure that future changes don't add a dependency_link.

i.e. before it was fairly easy to see if we were going to scattershot go across the internet to download stuff:

  1. See what packages/eggs are installed from where
  2. See bad-egg directly listed in the requirements file

If it was a requirement for a company to use --process-dependency-links they would have to add it to the command line, but it would have to be a conscious choice.

For example, I have an internal project that does depend on --process-dependency-links, but I won't ever add that particular project to my requirements.txt along with a --process-dependency-links because I don't want pip downloading stuff from around the internet. Instead my build process does:

pip install -r requirements.txt
# Assume requirements.txt includes all the requirements for the project
pip install --process-dependency-links privateproject.tar.gz

This way I can be sure that only the dependency links that I explicitly control (in privateproject.tar.gz or more likely our git repo directly) are followed, and everything else is installed from PyPi without following dependency links (even if a project does use them).

My privateproject.tar.gz doesn't depend on ANY packages on PyPi and only on my dependency_links. Yes this means someone running python setup.py dev or something along those lines won't download all the dependencies, but it is a way I can protect me and my infrastructure from outside code being downloaded from all over the web.

bertjwregeer commented Jan 20, 2016

@xavfernandez I understand that this does not enable it for all requirement files...

But --process-dependency-links in a requirements.txt file now applies to all packages inside that requirements file. So now if I audit the file, I see "--process-dependency-links" and I need to manually go through EVERY single line in the file and find out the following:

  1. What packages/eggs are installed from where
  2. Do any of the above have dependency links?
  3. Find all dependencies for the packages/eggs
  4. Goto step 2 until the tree is exhausted
  5. Consider if it is all acceptable or not, make sure to pin to a particular version for EVERYTHING to make sure that future changes don't add a dependency_link.

i.e. before it was fairly easy to see if we were going to scattershot go across the internet to download stuff:

  1. See what packages/eggs are installed from where
  2. See bad-egg directly listed in the requirements file

If it was a requirement for a company to use --process-dependency-links they would have to add it to the command line, but it would have to be a conscious choice.

For example, I have an internal project that does depend on --process-dependency-links, but I won't ever add that particular project to my requirements.txt along with a --process-dependency-links because I don't want pip downloading stuff from around the internet. Instead my build process does:

pip install -r requirements.txt
# Assume requirements.txt includes all the requirements for the project
pip install --process-dependency-links privateproject.tar.gz

This way I can be sure that only the dependency links that I explicitly control (in privateproject.tar.gz or more likely our git repo directly) are followed, and everything else is installed from PyPi without following dependency links (even if a project does use them).

My privateproject.tar.gz doesn't depend on ANY packages on PyPi and only on my dependency_links. Yes this means someone running python setup.py dev or something along those lines won't download all the dependencies, but it is a way I can protect me and my infrastructure from outside code being downloaded from all over the web.

@pfmoore

This comment has been minimized.

Show comment
Hide comment
@pfmoore

pfmoore Jan 20, 2016

Member

Surely, though, you can just add a further rule to your company audit rules, that --process-dependency-links is not allowed in requirements files? Then you're in just the same situation as currently. This doesn't sound like a reason we should remove the feature from pip.

Member

pfmoore commented Jan 20, 2016

Surely, though, you can just add a further rule to your company audit rules, that --process-dependency-links is not allowed in requirements files? Then you're in just the same situation as currently. This doesn't sound like a reason we should remove the feature from pip.

@toddjames

This comment has been minimized.

Show comment
Hide comment
@toddjames

toddjames Mar 19, 2016

I was just trying to figure out why my dependency links weren't working for my private repositories and I found the --process-dependency-links flag. I was happy to see that it fixed the issue I was having, but dismayed that it was deprecated.

After some searching to see what I should be using instead, I found this issue. My question is this: is there a tracking issue that I can follow for the replacement feature I will need to use in the future once dependency links processing is removed? I'd like to stay on top of that so I can be prepared when it is removed.

toddjames commented Mar 19, 2016

I was just trying to figure out why my dependency links weren't working for my private repositories and I found the --process-dependency-links flag. I was happy to see that it fixed the issue I was having, but dismayed that it was deprecated.

After some searching to see what I should be using instead, I found this issue. My question is this: is there a tracking issue that I can follow for the replacement feature I will need to use in the future once dependency links processing is removed? I'd like to stay on top of that so I can be prepared when it is removed.

@xavfernandez

This comment has been minimized.

Show comment
Hide comment
@xavfernandez

xavfernandez Apr 3, 2016

Contributor

The dependency links will likely be replaced by what PEP508 allows.

Contributor

xavfernandez commented Apr 3, 2016

The dependency links will likely be replaced by what PEP508 allows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment