New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore external and unsafe urls aka PEP438 #985

Merged
merged 9 commits into from Jun 11, 2013

Conversation

Projects
None yet
3 participants
@dstufft
Member

dstufft commented Jun 7, 2013

Implements Phase2 of PEP438.

Adds a number of index options:

# Allow an externally hosted and verifiable file to be installed for PROJECT
#     AKA Files directly linked from the simple index with a Hash
--allow-external PROJECT

# Allows externally hosted and verifiable files to be installed for all projects
--allow-all-external

# Allows an insecure and unverifiable file to be installed for PROJECT
#    AKA files directly linked from the simple index without a hash
#    AKA files linked from the homepage or download url
--allow-insecure PROJECT

# Turn on future behavior with regards to external urls
--no-allow-external

# Turn on future behavior with regards to insecure urls
--no-allow-insecure

All changes are gated on the api version of the page being >= 2. This means that older indexes, apache indexes, etc will continue to use the old processing rules.

There is also a speed boost involved here as pip will ignore the homepage and download urls if the current options would not allow installing anything from them anyways.

The default action currently is to install but warn. In the future (1.5?) pip will default to --no-allow-insecure and no-allow-external and those flags will be noop'd.

URLs directly passed in via the command line, requirements files, etc are always considered verifiable and secure.

dstufft added some commits Jun 2, 2013

Differentiate between internal and external links where possible
* By default ignore external links
* Add the ``--allow-external`` flag that enables external links
  globally
* Fallback to allowing all links if we cannot determine the
  API version of the parsed page
* Inform the user of ``--allow-external`` if nothing was found
  to install
In accordance with PEP438 default to allowing external urls
* After one release has been made allowing external urls, future
  releases will disallow by default
Differentiate between safe and unsafe urls where possible
* Links and HTMLPages know if they are "trusted"
* File Links know if they are safe or not
* A "Safe" file Link comes from a trusted Link/HTMLPage and has
  a hash allowing verification of the download
* Adds a --allow-unsafe PACKAGE argument to allow unsafe files on
  a per package basis
* Optimizes scraping external sites by short circuiting if
  the current trust rules won't allow using it's files anyways
@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Jun 7, 2013

Member

I'm going to add this to the 1.4 milestone because i'd like to add it to 1.4 because I think it's an important change to both speed up installs and enable a path towards a more secure installation story. That being said if it's decided not to add this to 1.4 I can live with that (but I'd really love it if I didn't have to!).

Member

dstufft commented Jun 7, 2013

I'm going to add this to the 1.4 milestone because i'd like to add it to 1.4 because I think it's an important change to both speed up installs and enable a path towards a more secure installation story. That being said if it's decided not to add this to 1.4 I can live with that (but I'd really love it if I didn't have to!).

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Jun 7, 2013

Member

This fixes #818 as well.

Member

dstufft commented Jun 7, 2013

This fixes #818 as well.

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft

dstufft Jun 7, 2013

Member

This also (more or less) fixes #623

Member

dstufft commented Jun 7, 2013

This also (more or less) fixes #623

dstufft added a commit that referenced this pull request Jun 11, 2013

Merge pull request #985 from pypa/ignore-external-and-unsafe-urls
Ignore external and unsafe urls aka PEP438

@dstufft dstufft merged commit 499f45d into develop Jun 11, 2013

1 check passed

default The Travis CI build passed
Details

@dstufft dstufft deleted the ignore-external-and-unsafe-urls branch Jun 11, 2013

@asmeurer

This comment has been minimized.

Show comment
Hide comment
@asmeurer

asmeurer Jun 11, 2013

It's awesome to see this finally fixed!

asmeurer commented Jun 11, 2013

It's awesome to see this finally fixed!

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Jul 3, 2013

Contributor

@dstufft can you add a good summary in the change log for this.

Contributor

qwcode commented Jul 3, 2013

@dstufft can you add a good summary in the change log for this.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Jul 3, 2013

Contributor

add it in release-1.4 branch and it will get merged over periodically

Contributor

qwcode commented Jul 3, 2013

add it in release-1.4 branch and it will get merged over periodically

@dstufft

This comment has been minimized.

Show comment
Hide comment
@dstufft
Member

dstufft commented Jul 3, 2013

@qwcode Done in 15c2a73

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment