Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

SSL certificate verification. #791

Merged
merged 59 commits into from

9 participants

@qwcode
Owner

builds on original pull #789

Done:

  • move ssl and match_hostname imports to backwardcompat
  • added match_hostname function to pip (no external dependency)
  • cert path moved to locations module
  • better error message when UrlError.reason is CertificateError or SSLError
  • added license for CA Root ceriticates
  • miscellaneous test fixes
  • --cert-path option to override path to pem file
  • add new tests
  • when scheme is 'https' and have ssl module, add in ssl cert verifying handler (and exclude http handler). otherwise, use build_opener to generate a "standard" chain of handlers.
  • --allow-no-ssl option to allow lack of cert verification when no ssl module (for py25 users who won't install ssl)
  • update to latest certs securely (https://gist.github.com/jjb/996292)
  • update authors file and release notes
  • update pip docs
  • py25 socket patch to work with ssl backport
  • only show --allow-no-ssl when no ssl
  • use the words "CA bundle" instead of "certificate file"
  • get an install test working that installs ssl backport for py25
  • refactor backwardcompat to a package
  • py25 ssl backport install test works locally, but skipping on travis for now.
  • fix the "pip list" tests to use our local packages for testing
  • wait for PSF's new cert, then remove old cacert.org cert from pem file
  • http://cheeseshop.python.org/ 503 errors have been fixed
  • updates based on jannis review

Todo:

  • merge and release an RC.
pip/util.py
@@ -664,3 +666,18 @@ def call_subprocess(cmd, show_stdout=True,
% (command_desc, proc.returncode, cwd))
if stdout is not None:
return ''.join(all_output)
+
+
+def warn_if_no_ssl():
+ """Warn when there's no ssl"""
+ if not ssl:
+ logger.warn(textwrap.dedent("""
+ #############################################################
+ ## WARNING!! ##
+ ## You don't have an importable ssl module. ##
+ ## We can not provide ssl certified downloads from PyPI. ##
+ ## Install this: http://pypi.python.org/pypi/ssl/ ##
@rasky
rasky added a note

You might want to use "https" in this URL :)

@qwcode Owner
qwcode added a note

yes : )

@rasky
rasky added a note

As an additional note, I think that we should probably wait for a keypress, or wait either 10 seconds or a keypress, (or exit and ask to be relaunched with a command line option, eg: --exploit-me). I know it's annoying, but there is a remote code execution vulnerability here; if you simply print a warning and get on downloading the file and running its setup.py, it might be too late for the user.

@qwcode Owner
qwcode added a note

I'll go with the consensus, but I was concerned about breaking people's automated processes. @jezdez, @pnasrat ?

@rasky
rasky added a note

If that's an (understandable) concern, a delay of 10 seconds (shortened by a keypress) should probably solve it.

@pnasrat Owner
pnasrat added a note

I don't like --exploit-me as an option just --nossl is probably sufficient. We probably don't want to break scripts, just announce vocally.

For package maintainers we should also have this in the installation docs.

@qwcode Owner
qwcode added a note

@pnasrat but if they don't use --nossl ?

which option?
1) just a message with no delay
2) 10s delay which gives people time to quit or continue
3) fail

@dstufft Owner
dstufft added a note

If they don't explictly disable SSL fail w/ an error telling them how to bypass it.

@jezdez Owner
jezdez added a note

3)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@dstufft
Owner

We should ensure that any SSL enabled index url is only ever accessed via SSL. In particular this protects against a SSL stripping[1] attack where an attacker is able to convert the request to plaintext (and thus be able to modify the request) before sending it via SSL itself to PyPI.

[1] http://www.thoughtcrime.org/software/sslstrip/

@rasky

@dstufft I don't think it's a problem in the context of the patch since the PyPI index URL does not (currently) redirect to the HTTPS version, and pip will hardcode the HTTPS version anyway.

@dstufft
Owner

@rasky What will happen if pip requests the HTTPS version but receives a plaintext response?

@rasky

@dstufft Given that the patch explicitly wraps the TCP socket with ssl.wrap_socket, the SSL negotiation would fail because they won't speak the same protocol ( = ssl.wrap_socket() will raise an exception).

This said, please do test it :)

@dstufft
Owner
@radiosilence

With regards to CAcert, is the strategy to go with CAcert "for now", and then remove it once PyPI has a good certificate?

For a free Class 1 (no EV) certificate, I'd definitely look into StartSSL.

@qwcode
Owner

I think PyPI is close to having a new cert. The idea right now, is that I would remove the cacert.org cert before merge.

pip/util.py
@@ -664,3 +666,18 @@ def call_subprocess(cmd, show_stdout=True,
% (command_desc, proc.returncode, cwd))
if stdout is not None:
return ''.join(all_output)
+
+
+def raise_no_ssl_exception():
+ """Raise when there's no ssl and not using '--no-ssl'"""
+ raise PipError(textwrap.dedent("""
+ #############################################################
+ ## You don't have an importable ssl module. ##
+ ## We can not provide ssl certified downloads. ##
+ ## Do one of 2 things: ##
+ ## 1) Install this: https://pypi.python.org/pypi/ssl/ ##
+ ## (It provides ssl support for older Pythons ) ##
+ ## 2) Use the --no-ssl option to allow this insecurity ##
@rasky
rasky added a note

It's --allow-no-ssl now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@rasky rasky commented on the diff
tests/test_config.py
((7 lines not shown))
reset_env(environ)
result = run_pip('install', '-vvv', 'INITools', expect_error=True)
assert "Analyzing links from page http://pypi.pinaxproject.com" in result.stdout, result.stdout
- assert "Analyzing links from page http://example.com" in result.stdout, result.stdout
+ assert "Skipping link %s" % find_links in result.stdout
@rasky
rasky added a note

Since we're at it: shouldn't --find-links also require --allow-no-ssl? It can be a separate patch, of course (I can work on it).

@dstufft Owner
dstufft added a note

Forcing SSL on find links doesn't really buy near as much as forcing it on PyPI does. Further more we don't know for sure that find links will even have SSL (or if it is, valid SSL), and we don't want to have people disable SSL for their Find links and have it disable their main repo protection as well.

@rasky
rasky added a note

Well, the URL pointed by --find-links can still be MITM'd to serve compromised packages, as far as I can see. Obviously it's less important than fixing the PyPI one, but it's probably something to consider while we enforce pip security.

Does --find-links work with HTTPS urls? Maybe we should emit a warning if the URL being passed is HTTP?

@dstufft Owner
dstufft added a note

I think a warning with HTTP would def be good. --find-links requires user actions to enable it so we'll be secure out of the box still. The biggest concern I have with tying it to --allow-no-ssl is losing PyPI SSL protection b/c one of your find links doesn't have a valid SSL cert.

@qwcode Owner
qwcode added a note

the way it's written now, the use of the ssl url opener is for any connection, index urls or find_link urls, whether they are "http" or "https"

and --allow-no-ssl just means "it's ok to use the non-ssl url opener when ssl is not importable".

e.g.:

pip -vv  install --no-index --find-links=https://pypi.python.org/simple/peppercorn/ peppercorn
[...]
 Analyzing links from page https://pypi.python.org/simple/peppercorn/
[...]
 Found link https://pypi.python.org/packages/source/p/peppercorn/peppercorn- [..]

this all seems to work, but should it be changed? should we use the non-ssl opener for "http:" urls?

@qwcode Owner
qwcode added a note

should we use the non-ssl opener for "http:" urls?

when accessing "http" urls, our opener (a normal opener plus the https handler) will try and use the http handler first (and succeed), before exercising the https handler (which it would fail on), so I think it's ok.

@rasky
rasky added a note

Oh, that's a security issue then. It means that an attacker can still MITM by having a non-SSL HTTP proxy server on port 443.

The opener used by pip to access PyPI (and all https URLs) should only allow SSL connections; there should not be any fallback to HTTP (not even if SSL fails).

@qwcode Owner
qwcode added a note

ok, if it really does fail in this case (and I agree it seems it could given the way it processes thru the chain of handlers; just me looking at urlllib2 code; not an actual experiment), I think it can be prevented pretty easily, by passing in our own dummy HTTPHandler that fails in all cases, in addition to our verification handler.

from the urllib2 docs for build_opener:
"Instances of the following classes will be in front of the handlers, unless the handlers contain them, instances of them or subclasses of them: ProxyHandler, UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler, HTTPErrorProcessor"

@qwcode Owner
qwcode added a note

also, to be clear, there is no "fallback" to the http handler if the https handler fails. the issue is that it seems to try the http handler first.

I guess the most direct thing is to specifically contruct an OpenerDirector instance with one handler, and not use build_opener

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/basecommand.py
@@ -106,6 +107,12 @@ def main(self, args, initial_options):
if options.exists_action:
os.environ['PIP_EXISTS_ACTION'] = ''.join(options.exists_action)
+ if options.allow_no_ssl:
+ os.environ['PIP_ALLOW_NO_SSL'] = '1'
@rasky
rasky added a note

Is there any security implication in using the environmentfor security-related configuration (PIP_ALLOW_NO_SSL, PIP_CERT_PATH)? I guess that if an attacker can modify the environment in which pip is run, she can as well change the PYTHONPATH to her hacked version of pip, so maybe it's not a big problem.

Comments?

@dstufft Owner
dstufft added a note

I wouldn't be real worried about it. If an attacker can set envvars we've lost the game for that computer already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
tests/test_pip.py
@@ -11,6 +11,10 @@
from scripttest import TestFileEnvironment, FoundDir
from tests.path import Path, curdir, u
from pip.util import rmtree
+from pip.backwardcompat import ssl
+
+#allow py25 unit tests to work
+os.environ['PIP_ALLOW_NO_SSL'] = '1'
@hltbra Owner
hltbra added a note

Should all tests skip ssl by default? I see there is the condition to include that if no ssl module found at run_pip, it should not be here (at top), right?

@qwcode Owner
qwcode added a note

--allow-no-ssl means that it's ok to use the non-ssl url opener when ssl is not installed (which is only true on py25)
it doesn't mean "don't use ssl". to be clear though, I will add a condition such that's only set for py25

also, I'm going to add a specific test case that runs just for py25 that installs ssl (http://pypi.python.org/pypi/ssl/) and confirm that an install against pypi works using that library

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/download.py
((24 lines not shown))
+ self.sock = ssl.wrap_socket(sock,
+ self.key_file,
+ self.cert_file,
+ cert_reqs=ssl.CERT_REQUIRED,
+ ca_certs=cert_path)
+ match_hostname(self.sock.getpeercert(), self.host)
+
+
+class VerifiedHTTPSHandler(urllib2.HTTPSHandler, urllib2.HTTPHandler):
+ """
+ A HTTPSHandler that wraps connections with ssl certificate verification.
+ By inheriting from both HTTPSHandler and HTTPHandler, this overrides
+ the default https *and* http handlers during the 'build_opener' routine.
+ We specifically *don't* want a http handler in the chain of handlers
+ to prevent MITM attacks that spoof https servers with http content.
+ """
@qwcode Owner
qwcode added a note

although the double inheritance works to eliminate the httphandler, this feels weird.
maybe just better to delete the httphandler from the handler list after the OpenerDirector instance is built.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/baseparser.py
@@ -350,4 +350,20 @@ def create_main_parser():
metavar='action',
help="Default action when a path already exists: "
"(s)witch, (i)gnore, (w)ipe, (b)ackup."),
+
+ optparse.make_option(
+ '--allow-no-ssl',
+ dest='allow_no_ssl',
+ action='store_true',
+ default=False,
+ help = "Allow lack of certificate checking when ssl is not installed."),
+
+ optparse.make_option(
+ '--cert-path',
+ dest='cert_path',
+ type='str',
+ default='',
+ metavar='path',
+ help = "Path to alternate certificate file."),
@rasky
rasky added a note

The most common name is probably "CA bundle" instead of "certificate file". If I read certificate file, I think of a single certificate (and thus I cannot understand why pip would need a certificate).

@qwcode Owner
qwcode added a note

ok, I'll change the help line. "Path to alternate CA bundle."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/baseparser.py
@@ -350,4 +350,20 @@ def create_main_parser():
metavar='action',
help="Default action when a path already exists: "
"(s)witch, (i)gnore, (w)ipe, (b)ackup."),
+
+ optparse.make_option(
+ '--allow-no-ssl',
+ dest='allow_no_ssl',
+ action='store_true',
+ default=False,
+ help = "Allow lack of certificate checking when ssl is not installed."),
@rasky
rasky added a note

Shouldn't this option itself be guarded by if ssl of if py25, so not to clutter the --help output for most users? It does not have any effect for them anyway.

@qwcode Owner
qwcode added a note

agreed, I'll do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@jezdez
Owner

@qwcode Would you mind moving the backwardcompat_ssl.py into backwardcompat/ssl_match_hostname.py please?

@qwcode
Owner

@jezdez, sure, np.

@qwcode
Owner

@jezdez, you mentioned on twitter, pip using the requests project for ssl verification. is that effort happening somewhere? should we reconsider this pull?

@hltbra
Owner

What about pip search that uses XMLRPC and HTTP?

@coderanger

New SSL layout on PyPI is live. Gogogo :runner: :runner: :runner:

@qwcode
Owner

sorting out test failures that are arising as a result....
couple of tests end up hitting http://cheeseshop.python.org which now returns 503...
intentional?

@jezdez
Owner

D'oh!

@jezdez jezdez commented on the diff
pip/backwardcompat/socket_create_connection.py
((21 lines not shown))
+ host, port = address
+ err = None
+ for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
+ af, socktype, proto, canonname, sa = res
+ sock = None
+ try:
+ sock = socket.socket(af, socktype, proto)
+ if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
+ sock.settimeout(timeout)
+ if source_address:
+ sock.bind(source_address)
+ sock.connect(sa)
+ return sock
+
+ except socket.error:
+ err = sys.exc_info()[1]
@jezdez Owner
jezdez added a note

sys is undefined here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
NOTICE.txt
@@ -0,0 +1,18 @@
@jezdez Owner
jezdez added a note

This file can be ported over to LICENSE.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
docs/installing.txt
@@ -3,11 +3,21 @@
Installation
============
-Python Support
---------------
+.. warning::
+
+ Prior to version 1.3, pip did not use SSL for downloading packages from PyPI, and thus left
+ users more vulnerable to security threats. We advise installing at least version 1.3.
+ If you're using `virtualenv <http://www.virtualenv.org>`_ to install pip, we advise installing
+ at least version 1.8.5, which contains pip version 1.3.
@jezdez Owner
jezdez added a note

I think we should make a 1.9 release to mark pip 1.3 inclusion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
docs/installing.txt
((35 lines not shown))
+
+Using a Package Manager
++++++++++++++++++++++++
+
+On Linux, pip is packaged by most distributions. For instance, on an Ubuntu
+system, you can install it with::
+
+ $ sudo apt-get install python-pip
+
+On a Fedora system, you can install it with::
+
+ $ yum install python-pip
+
+The latest versions of pip may not be available using this method.
+
+
@jezdez Owner
jezdez added a note

The package manager section is clearly to help people get the safest experience. The problem though is that they almost always have older versions of pip available (e.g. http://packages.ubuntu.com/search?keywords=python-pip) that would make the latest efforts to secure pip moot. Please remove this section till we have confirmation that the latest pip is available through the native package managers.

@rasky
rasky added a note

Just before this paragraph, there is a warning explaining that you really want pip 1.3.

What about adding a line saying "make sure that your distribution packages pip 1.3 or later, otherwise use a manual installation as explained below".

@jezdez Owner
jezdez added a note

See, I don't believe people actually read all the paragraphs, but only look for the apt-get line they need to copy/paste. So even providing that part is a no-go for me.

That said, maybe we should reach out to the distros activately and ask them to update. Or.. even provide packages for the different distros (e.g. an own PPA).

@pnasrat Owner
pnasrat added a note
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
docs/logic.txt
@@ -151,17 +151,37 @@ pip offers a set of :ref:`Package Index Options <Package Index Options>` for mod
See the :ref:`pip install Examples<pip install Examples>`.
+.. _`SSL Certificate Verification`:
+
+SSL Certificate Verification
+============================
+
+Starting with v1.3, pip provides SSL certificate verification over https, for the purpose
+of providing secure, certified downloads from PyPI.
+
+This is supported by default in all Python versions pip supports, except Python 2.5.
+
+Python 2.5 users can install https://pypi.python.org/pypi/ssl/, which provides ssl support for older pythons.
@jezdez Owner
jezdez added a note

We should extend that paragraph a bit, with a step by step guide how to install the ssl package for *NIX and Windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/basecommand.py
@@ -69,11 +69,15 @@ def _copy_option_group(self, parser, group):
def merge_options(self, initial_options, options):
# Make sure we have all global options carried over
- for attr in ['log', 'proxy', 'require_venv',
+ attrs = ['log', 'proxy', 'require_venv',
'log_explicit_levels', 'log_file',
@jezdez Owner
jezdez added a note

Indentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/basecommand.py
@@ -104,6 +108,12 @@ def main(self, args, initial_options):
if options.exists_action:
os.environ['PIP_EXISTS_ACTION'] = ''.join(options.exists_action)
+ if not ssl and options.allow_no_ssl:
+ os.environ['PIP_ALLOW_NO_SSL'] = '1'
@jezdez Owner
jezdez added a note

Why do we need to set an environment variable here? The option system in pip was built so it can accept env vars and pass them as part of the optparse options to the functions that need it, not the other way around.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/basecommand.py
@@ -104,6 +108,12 @@ def main(self, args, initial_options):
if options.exists_action:
os.environ['PIP_EXISTS_ACTION'] = ''.join(options.exists_action)
+ if not ssl and options.allow_no_ssl:
+ os.environ['PIP_ALLOW_NO_SSL'] = '1'
+
+ if options.cert_path:
+ os.environ['PIP_CERT_PATH'] = options.cert_path
@jezdez Owner
jezdez added a note

Same as above.

@qwcode Owner
qwcode added a note

these options are used in places where the options object in the command isn't currently tunneled into.
we'd need to pass options down a couple of levels. what do you recommend?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/baseparser.py
@@ -356,4 +356,21 @@ def create_main_parser():
metavar='action',
help="Default action when a path already exists: "
"(s)witch, (i)gnore, (w)ipe, (b)ackup."),
+
+ optparse.make_option(
+ '--cert-path',
@jezdez Owner
jezdez added a note

Should we maybe just call that --cert, less to type :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/baseparser.py
((13 lines not shown))
]
+
+if not ssl:
+ standard_options.append(optparse.make_option(
+ '--allow-no-ssl',
@jezdez Owner
jezdez added a note

For this option I'd like to make it even clearer to the user what they are doing if they don't use this. Let's call it --insecure or some-such.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/util.py
@@ -664,3 +666,18 @@ def call_subprocess(cmd, show_stdout=True,
% (command_desc, proc.returncode, cwd))
if stdout is not None:
return ''.join(all_output)
+
+
+def raise_no_ssl_exception():
+ """Raise when there's no ssl and not using '--no-ssl'"""
+ raise PipError(textwrap.dedent("""
+ #############################################################
+ ## You don't have an importable ssl module. ##
+ ## We can not provide ssl certified downloads. ##
+ ## Do one of 2 things: ##
+ ## 1) Install this: https://pypi.python.org/pypi/ssl/ ##
+ ## (It provides ssl support for older Pythons ) ##
+ ## 2) Use the --allow-no-ssl option to allow insecurity ##
+ #############################################################
@jezdez Owner
jezdez added a note

This is pretty good already, I fear that people will freak out if they see it for the first time though. Let's explain why we can't ship ssl support and link to our documentation explaining in detail how to install it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@qwcode
Owner

just read @jezdez comments. will post updates later....

@jezdez
Owner

w00t!, @qwcode!

@qwcode
Owner

@hltbra , for pip search, I can switch the index url to 'https', but the xmlrpc lib is written internally to use the HTTP connection, and since we're not downloading using that command, I wasn't going to do much work right now to use a cert verifying connection. not sure it would work?

@jezdez
Owner

Win!

@qwcode
Owner

signing off for now.

will check tomorrow for any more updates to be made.
hoping to be able cut RCs tomorrow.

@merwok

Indent has two extra spaces.

@qwcode qwcode merged commit beb9dea into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Feb 4, 2013
  1. Changing the default index URL to use HTTPS.

    James Cleveland authored
  2. Added certificate validation against root CA file.

    James Cleveland authored
    Credits to Joseph Turner for this code:
    
    http://thejosephturner.com/blog/2011/03/19/https-certificate-verification-in-python-with-urllib2/
    
    And also to Kenneth Reitz for the CA dump (with addition of
    CAcert.org chain because that's what PyPI uses).
    
    Which, combined with pip, enables us to go a step towards
    validated SSL and reducing the impact of MITM attacks.
  3. Added CA file to MANIFEST.in

    James Cleveland authored
  4. Adding PEM file to package data.

    James Cleveland authored
  5. Adding code to match_hostname.

    James Cleveland authored
  6. added install_requires

    James Cleveland authored
Commits on Feb 6, 2013
  1. @qwcode

    'package_dir' not needed

    qwcode authored
  2. @qwcode
  3. @qwcode
  4. @qwcode

    match_hostname from py32

    qwcode authored
  5. @qwcode
  6. @qwcode
  7. @qwcode
  8. @qwcode
  9. @qwcode

    py25 import fixes

    qwcode authored
Commits on Feb 7, 2013
  1. @qwcode
  2. @qwcode

    misc test fixes

    qwcode authored
  3. @qwcode
  4. @qwcode
  5. @qwcode

    more py25 test fixes

    qwcode authored
Commits on Feb 8, 2013
  1. @qwcode
  2. @qwcode
  3. @qwcode
  4. @qwcode

    ssl cert tests

    qwcode authored
  5. @qwcode
Commits on Feb 9, 2013
  1. @qwcode

    pypy ssl test fix

    qwcode authored
  2. @qwcode
  3. @qwcode

    update authors and changelog

    qwcode authored
  4. @qwcode
  5. @qwcode

    ssl cert docs updates

    qwcode authored
  6. @qwcode
Commits on Feb 10, 2013
  1. @qwcode
  2. @qwcode
  3. @qwcode

    ssl docs fix

    qwcode authored
  4. @qwcode

    --allow-no-ssl test fix

    qwcode authored
Commits on Feb 11, 2013
  1. @qwcode
  2. @qwcode
  3. @qwcode
  4. @qwcode

    remove HTTPHandler explictly

    qwcode authored
Commits on Feb 12, 2013
  1. @qwcode
Commits on Feb 16, 2013
  1. @qwcode
  2. @qwcode

    more local test packages

    qwcode authored
  3. @qwcode

    merge with develop

    qwcode authored
  4. @qwcode
  5. @qwcode

    merge with develop

    qwcode authored
  6. @qwcode

    remove old cert from pem file

    qwcode authored
  7. @qwcode

    fix import syntax

    qwcode authored
Commits on Feb 18, 2013
  1. @qwcode

    add missing sys import

    qwcode authored
  2. @qwcode
  3. @qwcode
  4. @qwcode
  5. @qwcode

    proper list indents

    qwcode authored
  6. @qwcode

    from --cert-path to --cert

    qwcode authored
  7. @qwcode
  8. @qwcode
  9. @qwcode
  10. @qwcode
Commits on Feb 19, 2013
  1. @qwcode

    remove extra indents in docs

    qwcode authored
  2. @qwcode
This page is out of date. Refresh to see the latest.
View
1  AUTHORS.txt
@@ -25,6 +25,7 @@ Ian Bicking
Igor Sobreira
Ionel Maries Cristian
Jakub Vysoky
+James Cleveland
Jannis Leidel
Jay Graves
John-Scott Atlakson
View
3  CHANGES.txt
@@ -4,6 +4,9 @@ Changelog
develop (unreleased)
--------------------
+* SSL Cert Verification; Make https the default for PyPI access.
+ Thanks James Cleveland, Giovanni Bajo and many others (Pull #789).
+
* Added "pip list" for listing installed packages and the latest version
available. Thanks Rafael Caricio, Miguel Araujo, Dmitry Gladkov (Pull #752)
View
19 LICENSE.txt
@@ -18,3 +18,22 @@ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+
+License for Bundle of CA Root Certificates (pip/cacert.pem)
+===========================================================
+
+This library is free software; you can redistribute it and/or
+modify it under the terms of the GNU Lesser General Public
+License as published by the Free Software Foundation; either
+version 2.1 of the License, or (at your option) any later version.
+
+This library is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+Lesser General Public License for more details.
+
+You should have received a copy of the GNU Lesser General Public
+License along with this library; if not, write to the Free Software
+Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
+02110-1301
View
1  MANIFEST.in
@@ -2,6 +2,7 @@ include AUTHORS.txt
include LICENSE.txt
include CHANGES.txt
include PROJECT.txt
+include pip/cacert.pem
recursive-include docs *.txt
recursive-include docs *.html
recursive-exclude docs/_build *.txt
View
2  docs/index.txt
@@ -6,7 +6,7 @@ A tool for installing and managing Python packages.
`Mailing list <http://groups.google.com/group/python-virtualenv>`_ ``|``
`Issues <https://github.com/pypa/pip/issues>`_ ``|``
`Github <https://github.com/pypa/pip>`_ ``|``
-`PyPI <http://pypi.python.org/pypi/pip/>`_ ``|``
+`PyPI <https://pypi.python.org/pypi/pip/>`_ ``|``
irc:#pip
View
48 docs/installing.txt
@@ -3,11 +3,21 @@
Installation
============
-Python Support
---------------
+.. warning::
+
+ Prior to version 1.3, pip did not use SSL for downloading packages from PyPI, and thus left
+ users more vulnerable to security threats. We advise installing at least version 1.3.
+ If you're using `virtualenv <http://www.virtualenv.org>`_ to install pip, we advise installing
+ at least version 1.9, which contains pip version 1.3.
+
+
+Python & OS Support
+-------------------
pip works with CPython versions 2.5, 2.6, 2.7, 3.1, 3.2, 3.3 and also pypy.
+pip works on Unix/Linux, OS X, and Windows.
+
Using virtualenv
----------------
@@ -33,27 +43,35 @@ Installing Globally
pip can be installed globally in order to manage global packages.
Often this requires the installation to be performed as root.
-Prerequisites
-+++++++++++++
+.. warning::
+
+ We advise against using easy_install to install pip, because easy_install
+ does not download from PyPI over SSL, so the installation might be insecure.
+ Since pip can then be used to install packages (which execute code on
+ your computer), it is better to go through a trusted path.
+
-A global install of pip requires either `setuptools <http://pypi.python.org/pypi/setuptools>`_
-or `distribute <http://pypi.python.org/pypi/distribute>`_ to be installed globally as well.
+Requirements
+++++++++++++
-In many cases, these can be installed using your OS package manager.
+pip requires either `setuptools <https://pypi.python.org/pypi/setuptools>`_
+or `distribute <https://pypi.python.org/pypi/distribute>`_.
-See the `Distribute Install Instructions <http://pypi.python.org/pypi/distribute/>`_ or the
-`Setuptools Install Instructions <http://pypi.python.org/pypi/setuptools#installation-instructions>`_
+See the `Distribute Install Instructions <https://pypi.python.org/pypi/distribute/>`_ or the
+`Setuptools Install Instructions <https://pypi.python.org/pypi/setuptools#installation-instructions>`_
+
+If installing pip using a linux package manager, these requirements will be installed for you.
.. warning::
If you are using Python 3.X you **must** use distribute; setuptools doesn't
support Python 3.X.
+
Using get-pip
+++++++++++++
-Download `get-pip.py <https://raw.github.com/pypa/pip/master/contrib/get-pip.py>`_
-and execute it using Python. This will only install pip, not the prerequisites.
+After installing the requirements:
::
@@ -61,12 +79,14 @@ and execute it using Python. This will only install pip, not the prerequisites.
$ [sudo] python get-pip.py
-From source
-+++++++++++
+Installing from source
+++++++++++++++++++++++
+
+After installing the requirements:
::
- $ curl -O http://pypi.python.org/packages/source/p/pip/pip-X.X.tar.gz
+ $ curl -O https://pypi.python.org/packages/source/p/pip/pip-X.X.tar.gz
$ tar xvfz pip-X.X.tar.gz
$ cd pip-X.X
$ [sudo] python setup.py install
View
60 docs/logic.txt
@@ -151,17 +151,73 @@ pip offers a set of :ref:`Package Index Options <Package Index Options>` for mod
See the :ref:`pip install Examples<pip install Examples>`.
+.. _`SSL Certificate Verification`:
+
+SSL Certificate Verification
+============================
+
+Starting with v1.3, pip provides SSL certificate verification over https, for the purpose
+of providing secure, certified downloads from PyPI.
+
+This is supported by default in all Python versions pip supports, except Python 2.5.
+
+Python 2.5 users can :ref:`install an SSL backport <SSL Backport>`, which provides ssl support for older pythons.
+Pip does not try to install this automatically because it requires a compiler, which not all systems will have.
+
+Although not recommended, Python 2.5 users who are unable to install ssl, can use the global option,
+``--insecure``, to allow access to PyPI w/o attempting SSL certificate verification. This option will only be visible
+when ssl is not importable. This is *not* a general option.
+
+
+.. _`SSL Backport`:
+
+Installing the SSL Backport
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. warning::
+
+ We advise against using ``pip`` itself to install the ssl backport, because it won't be secure
+ until *after* installing ssl. Likewise, ``easy_install`` is not advised, because it
+ does not currently support ssl.
+
+
+1. Download the ssl archive:
+
+ * Using a Browser:
+
+ 1. Go to `this url <https://pypi.python.org/pypi/ssl/1.15>`_.
+ 2. Confirm the identity of the site is valid.
+ Most browsers provide this information to the left of the URL bar in the form of padlock icon that you can click on to confirm the site is verified.
+ 3. Scroll down, and click to download ``ssl-1.15.tar.gz``.
+
+ * Using curl, which supports ssl certificate verification:
+ ::
+
+ $ curl -O https://pypi.python.org/packages/source/s/ssl/ssl-1.15.tar.gz
+
+2. Confirm the md5sum:
+ ::
+ $ md5sum ssl-1.15.tar.gz
+ 81ea8a1175e437b4c769ae65b3290e0c ssl-1.15.tar.gz
+
+3. Unpack the archive, and change into the ``ssl-1.15`` directory.
+4. Run: ``python setup.py install``.
+
+
Hash Verification
=================
-PyPI provides an md5 hash of a package by having the link to the
-package include an #md5=<hash>.
+PyPI provides md5 hashes in the hash fragment of package download urls.
pip supports checking this, as well as any of the
guaranteed hashlib algorithms (sha1, sha224, sha384, sha256, sha512, md5).
The hash fragment is case sensitive (i.e. sha1 not SHA1).
+This check is only intended to provide basic download corruption protection.
+It is not intended to provide security against tampering. For that,
+see :ref:`SSL Certificate Verification`
+
Download Cache
==============
View
21 pip/backwardcompat.py → pip/backwardcompat/__init__.py
@@ -112,3 +112,24 @@ def home_lib(home):
else:
lib = os.path.join('lib', 'python')
return os.path.join(home, lib)
+
+
+## py25 has no builtin ssl module
+## only >=py32 has ssl.match_hostname and ssl.CertificateError
+try:
+ import ssl
+ try:
+ from ssl import match_hostname, CertificateError
+ except ImportError:
+ from pip.backwardcompat.ssl_match_hostname import match_hostname, CertificateError
+except ImportError:
+ ssl = None
+
+
+# patch for py25 socket to work with http://pypi.python.org/pypi/ssl/
+import socket
+if not hasattr(socket, 'create_connection'): # for Python 2.5
+ # monkey-patch socket module
+ from pip.backwardcompat.socket_create_connection import create_connection
+ socket.create_connection = create_connection
+
View
44 pip/backwardcompat/socket_create_connection.py
@@ -0,0 +1,44 @@
+"""
+patch for py25 socket to work with http://pypi.python.org/pypi/ssl/
+copy-paste from py2.6 stdlib socket.py
+https://gist.github.com/zed/1347055
+"""
+import socket
+import sys
+
+_GLOBAL_DEFAULT_TIMEOUT = getattr(socket, '_GLOBAL_DEFAULT_TIMEOUT', object())
+def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
+ source_address=None):
+ """Connect to *address* and return the socket object.
+
+ Convenience function. Connect to *address* (a 2-tuple ``(host,
+ port)``) and return the socket object. Passing the optional
+ *timeout* parameter will set the timeout on the socket instance
+ before attempting to connect. If no *timeout* is supplied, the
+ global default timeout setting returned by :func:`getdefaulttimeout`
+ is used.
+ """
+
+ host, port = address
+ err = None
+ for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
+ af, socktype, proto, canonname, sa = res
+ sock = None
+ try:
+ sock = socket.socket(af, socktype, proto)
+ if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
+ sock.settimeout(timeout)
+ if source_address:
+ sock.bind(source_address)
+ sock.connect(sa)
+ return sock
+
+ except socket.error:
+ err = sys.exc_info()[1]
@jezdez Owner
jezdez added a note

sys is undefined here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ if sock is not None:
+ sock.close()
+
+ if err is not None:
+ raise err
+ else:
+ raise socket.error("getaddrinfo returns an empty list")
View
60 pip/backwardcompat/ssl_match_hostname.py
@@ -0,0 +1,60 @@
+"""The match_hostname() function from Python 3.2, essential when using SSL."""
+
+import re
+
+__version__ = '3.2a3'
+
+class CertificateError(ValueError):
+ pass
+
+def _dnsname_to_pat(dn):
+ pats = []
+ for frag in dn.split(r'.'):
+ if frag == '*':
+ # When '*' is a fragment by itself, it matches a non-empty dotless
+ # fragment.
+ pats.append('[^.]+')
+ else:
+ # Otherwise, '*' matches any dotless fragment.
+ frag = re.escape(frag)
+ pats.append(frag.replace(r'\*', '[^.]*'))
+ return re.compile(r'\A' + r'\.'.join(pats) + r'\Z', re.IGNORECASE)
+
+def match_hostname(cert, hostname):
+ """Verify that *cert* (in decoded format as returned by
+ SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 rules
+ are mostly followed, but IP addresses are not accepted for *hostname*.
+
+ CertificateError is raised on failure. On success, the function
+ returns nothing.
+ """
+ if not cert:
+ raise ValueError("empty or no certificate")
+ dnsnames = []
+ san = cert.get('subjectAltName', ())
+ for key, value in san:
+ if key == 'DNS':
+ if _dnsname_to_pat(value).match(hostname):
+ return
+ dnsnames.append(value)
+ if not san:
+ # The subject is only checked when subjectAltName is empty
+ for sub in cert.get('subject', ()):
+ for key, value in sub:
+ # XXX according to RFC 2818, the most specific Common Name
+ # must be used.
+ if key == 'commonName':
+ if _dnsname_to_pat(value).match(hostname):
+ return
+ dnsnames.append(value)
+ if len(dnsnames) > 1:
+ raise CertificateError("hostname %r "
+ "doesn't match either of %s"
+ % (hostname, ', '.join(map(repr, dnsnames))))
+ elif len(dnsnames) == 1:
+ raise CertificateError("hostname %r "
+ "doesn't match %r"
+ % (hostname, dnsnames[0]))
+ else:
+ raise CertificateError("no appropriate commonName or "
+ "subjectAltName fields were found")
View
25 pip/basecommand.py
@@ -12,7 +12,7 @@
from pip.download import urlopen
from pip.exceptions import (BadCommand, InstallationError, UninstallationError,
CommandError)
-from pip.backwardcompat import StringIO
+from pip.backwardcompat import StringIO, ssl
from pip.baseparser import ConfigOptionParser, UpdatingDefaultsHelpFormatter
from pip.status_codes import SUCCESS, ERROR, UNKNOWN_ERROR, VIRTUALENV_NOT_FOUND
from pip.util import get_prog
@@ -69,11 +69,15 @@ def _copy_option_group(self, parser, group):
def merge_options(self, initial_options, options):
# Make sure we have all global options carried over
- for attr in ['log', 'proxy', 'require_venv',
- 'log_explicit_levels', 'log_file',
- 'timeout', 'default_vcs',
- 'skip_requirements_regex',
- 'no_input', 'exists_action']:
+ attrs = ['log', 'proxy', 'require_venv',
+ 'log_explicit_levels', 'log_file',
+ 'timeout', 'default_vcs',
+ 'skip_requirements_regex',
+ 'no_input', 'exists_action',
+ 'cert']
+ if not ssl:
+ attrs.append('insecure')
+ for attr in attrs:
setattr(options, attr, getattr(initial_options, attr) or getattr(options, attr))
options.quiet += initial_options.quiet
options.verbose += initial_options.verbose
@@ -98,12 +102,21 @@ def main(self, args, initial_options):
self.setup_logging()
+ #TODO: try to get these passing down from the command?
+ # without resorting to os.environ to hold these.
+
if options.no_input:
os.environ['PIP_NO_INPUT'] = '1'
if options.exists_action:
os.environ['PIP_EXISTS_ACTION'] = ''.join(options.exists_action)
+ if not ssl and options.insecure:
+ os.environ['PIP_INSECURE'] = '1'
+
+ if options.cert:
+ os.environ['PIP_CERT'] = options.cert
+
if options.require_venv:
# If a venv is required check if it can really be found
if not os.environ.get('VIRTUAL_ENV'):
View
19 pip/baseparser.py
@@ -6,7 +6,7 @@
import os
import textwrap
from distutils.util import strtobool
-from pip.backwardcompat import ConfigParser, string_types
+from pip.backwardcompat import ConfigParser, string_types, ssl
from pip.locations import default_config_file, default_log_file
from pip.util import get_terminal_size, get_prog
@@ -356,4 +356,21 @@ def create_main_parser():
metavar='action',
help="Default action when a path already exists: "
"(s)witch, (i)gnore, (w)ipe, (b)ackup."),
+
+ optparse.make_option(
+ '--cert',
+ dest='cert',
+ type='str',
+ default='',
+ metavar='path',
+ help = "Path to alternate CA bundle."),
+
]
+
+if not ssl:
+ standard_options.append(optparse.make_option(
+ '--insecure',
+ dest='insecure',
+ action='store_true',
+ default=False,
+ help = "Allow lack of certificate checking when ssl is not installed."))
View
3,895 pip/cacert.pem
3,895 additions, 0 deletions not shown
View
2  pip/cmdoptions.py
@@ -21,7 +21,7 @@ def make_option_group(group, parser):
'-i', '--index-url', '--pypi-url',
dest='index_url',
metavar='URL',
- default='http://pypi.python.org/simple/',
+ default='https://pypi.python.org/simple/',
help='Base URL of Python Package Index (default %default).')
extra_index_url = make_option(
View
2  pip/commands/search.py
@@ -24,7 +24,7 @@ def __init__(self, *args, **kw):
'--index',
dest='index',
metavar='URL',
- default='http://pypi.python.org/pypi',
+ default='https://pypi.python.org/pypi',
help='Base URL of Python Package Index (default %default)')
self.parser.insert_option_group(0, self.cmd_opts)
View
89 pip/download.py
@@ -5,18 +5,21 @@
import os
import re
import shutil
+import socket
import sys
import tempfile
-from pip.backwardcompat import (xmlrpclib, urllib, urllib2,
- urlparse, string_types)
-from pip.exceptions import InstallationError
+from pip.backwardcompat import (xmlrpclib, urllib, urllib2, httplib,
+ urlparse, string_types, ssl)
+if ssl:
+ from pip.backwardcompat import match_hostname
+from pip.exceptions import InstallationError, PipError, NoSSLError
from pip.util import (splitext, rmtree, format_size, display_path,
backup_dir, ask_path_exists, unpack_file,
create_download_cache_folder, cache_download)
from pip.vcs import vcs
from pip.log import logger
-
+from pip.locations import default_cert_path
__all__ = ['xmlrpclib_transport', 'get_file_content', 'urlopen',
'is_url', 'url_to_path', 'path_to_url', 'path_to_url2',
@@ -66,6 +69,53 @@ def get_file_content(url, comes_from=None):
_scheme_re = re.compile(r'^(http|https|file):', re.I)
_url_slash_drive_re = re.compile(r'/*([a-z])\|', re.I)
+class VerifiedHTTPSConnection(httplib.HTTPSConnection):
+ """
+ A connection that wraps connections with ssl certificate verification.
+ """
+ def connect(self):
+
+ self.connection_kwargs = {}
+
+ #TODO: refactor compatibility logic into backwardcompat?
+
+ # for > py2.5
+ if hasattr(self, 'timeout'):
+ self.connection_kwargs.update(timeout = self.timeout)
+
+ # for >= py2.7
+ if hasattr(self, 'source_address'):
+ self.connection_kwargs.update(source_address = self.source_address)
+
+ sock = socket.create_connection((self.host, self.port), **self.connection_kwargs)
+
+ # for >= py2.7
+ if getattr(self, '_tunnel_host', None):
+ self.sock = sock
+ self._tunnel()
+
+ # get alternate bundle or use our included bundle
+ cert_path = os.environ.get('PIP_CERT', '') or default_cert_path
+
+ self.sock = ssl.wrap_socket(sock,
+ self.key_file,
+ self.cert_file,
+ cert_reqs=ssl.CERT_REQUIRED,
+ ca_certs=cert_path)
+
+ match_hostname(self.sock.getpeercert(), self.host)
+
+
+class VerifiedHTTPSHandler(urllib2.HTTPSHandler):
+ """
+ A HTTPSHandler that uses our own VerifiedHTTPSConnection.
+ """
+ def __init__(self, connection_class = VerifiedHTTPSConnection):
+ self.specialized_conn_class = connection_class
+ urllib2.HTTPSHandler.__init__(self)
+ def https_open(self, req):
+ return self.do_open(self.specialized_conn_class, req)
+
class URLOpener(object):
"""
@@ -81,10 +131,10 @@ def __call__(self, url):
auth.
"""
- url, username, password = self.extract_credentials(url)
+ url, username, password, scheme = self.extract_credentials(url)
if username is None:
try:
- response = urllib2.urlopen(self.get_request(url))
+ response = self.get_opener(scheme=scheme).open(url)
except urllib2.HTTPError:
e = sys.exc_info()[1]
if e.code != 401:
@@ -121,10 +171,31 @@ def get_response(self, url, username=None, password=None):
self.passman.add_password(None, netloc, username, password)
stored_username, stored_password = self.passman.find_user_password(None, netloc)
authhandler = urllib2.HTTPBasicAuthHandler(self.passman)
- opener = urllib2.build_opener(authhandler)
+ opener = self.get_opener(authhandler, scheme=scheme)
# FIXME: should catch a 401 and offer to let the user reenter credentials
return opener.open(req)
+ def get_opener(self, *args, **kwargs):
+ """
+ Build an OpenerDirector instance based on the scheme, whether ssl is
+ importable and the --insecure parameter.
+ """
+ if kwargs.get('scheme') == 'https':
+ if ssl:
+ https_handler = VerifiedHTTPSHandler()
+ director = urllib2.build_opener(https_handler, *args)
+ #strip out HTTPHandler to prevent MITM spoof
+ for handler in director.handlers:
+ if isinstance(handler, urllib2.HTTPHandler):
+ director.handlers.remove(handler)
+ return director
+ elif os.environ.get('PIP_INSECURE', '') == '1':
+ return urllib2.build_opener(*args)
+ else:
+ raise NoSSLError()
+ else:
+ return urllib2.build_opener(*args)
+
def setup(self, proxystr='', prompting=True):
"""
Sets the proxy handler given the option passed on the command
@@ -161,7 +232,7 @@ def extract_credentials(self, url):
username, password = self.parse_credentials(netloc)
if username is None:
- return url, None, None
+ return url, None, None, scheme
elif password is None and self.prompting:
# remove the auth credentials from the url part
netloc = netloc.replace('%s@' % username, '', 1)
@@ -173,7 +244,7 @@ def extract_credentials(self, url):
netloc = netloc.replace('%s:%s@' % (username, password), '', 1)
target_url = urlparse.urlunsplit((scheme, netloc, path, query, frag))
- return target_url, username, password
+ return target_url, username, password, scheme
def get_proxy(self, proxystr=''):
"""
View
26 pip/exceptions.py
@@ -1,5 +1,6 @@
"""Exceptions used throughout package"""
+import textwrap
class PipError(Exception):
"""Base pip exception"""
@@ -28,3 +29,28 @@ class BadCommand(PipError):
class CommandError(PipError):
"""Raised when there is an error in command-line arguments"""
+
+
+class NoSSLError(PipError):
+ """Raised when there's no ssl and not using '--insecure'"""
+
+ def __str__(self):
+ return textwrap.dedent("""
+ ###################################################################
+ ## You don't have an importable ssl module. You are most ##
+ ## likely using Python 2.5, which did not include ssl ##
+ ## support by default. In this state, we can not provide ##
+ ## ssl certified downloads from PyPI. ##
+ ## ##
+ ## You can do one of 2 things: ##
+ ## 1) Install this: https://pypi.python.org/pypi/ssl/ ##
+ ## (It provides ssl support for older Pythons ) ##
+ ## 2) Use the --insecure option to allow this insecurity ##
+ ## ##
+ ## For more details, go to the "SSL Certificate Verification" ##
+ ## section located here: ##
+ ## http://www.pip-installer.org/en/latest/logic.html ##
+ ## ##
+ ###################################################################
+ """)
+
View
7 pip/index.py
@@ -23,8 +23,10 @@
from pip.backwardcompat import (WindowsError, BytesIO,
Queue, urlparse,
URLError, HTTPError, u,
- product, url2pathname,
+ product, url2pathname, ssl,
Empty as QueueEmpty)
+if ssl:
+ from pip.backwardcompat import CertificateError
from pip.download import urlopen, path_to_url2, url_to_path, geturl, Urllib2HeadRequest
__all__ = ['PackageFinder']
@@ -486,6 +488,9 @@ def get_page(cls, link, req, cache=None, skip_archives=True):
level =1
desc = 'timed out'
elif isinstance(e, URLError):
+ #ssl/certificate error
+ if ssl and hasattr(e, 'reason') and (isinstance(e.reason, ssl.SSLError) or isinstance(e.reason, CertificateError)):
+ desc = 'there was a problem confirming the ssl certificate %s' % e
log_meth = logger.info
if hasattr(e, 'reason') and isinstance(e.reason, socket.timeout):
desc = 'timed out'
View
1  pip/locations.py
@@ -8,6 +8,7 @@
from pip.backwardcompat import get_python_lib
import pip.exceptions
+default_cert_path = os.path.join(os.path.dirname(__file__), 'cacert.pem')
def running_under_virtualenv():
"""
View
4 pip/log.py
@@ -4,7 +4,7 @@
import sys
import logging
-import pip.backwardcompat
+from pip import backwardcompat
class Logger(object):
@@ -72,7 +72,7 @@ def log(self, level, msg, *args, **kw):
rendered = '%02i %s' % (level, rendered)
if hasattr(consumer, 'write'):
rendered += '\n'
- pip.backwardcompat.fwrite(consumer, rendered)
+ backwardcompat.fwrite(consumer, rendered)
else:
consumer(rendered)
View
6 pip/util.py
@@ -8,8 +8,10 @@
import zipfile
import tarfile
import subprocess
-from pip.exceptions import InstallationError, BadCommand
-from pip.backwardcompat import WindowsError, string_types, raw_input, console_to_str, user_site
+import textwrap
+from pip.exceptions import InstallationError, BadCommand, PipError
+from pip.backwardcompat import(WindowsError, string_types, raw_input,
+ console_to_str, user_site, ssl)
from pip.locations import site_packages, running_under_virtualenv, virtualenv_no_global
from pip.log import logger
View
3  setup.py
@@ -47,7 +47,8 @@ def find_version(*file_paths):
author_email='python-virtualenv@groups.google.com',
url='http://www.pip-installer.org',
license='MIT',
- packages=['pip', 'pip.commands', 'pip.vcs'],
+ packages=['pip', 'pip.commands', 'pip.vcs', 'pip.backwardcompat'],
+ package_data={'pip': ['*.pem']},
entry_points=dict(console_scripts=['pip=pip:main', 'pip-%s=pip:main' % sys.version[:3]]),
test_suite='nose.collector',
tests_require=tests_require,
View
4 tests/packages/README.txt
@@ -58,9 +58,9 @@ pkgwithmpkg-1.0.tar.gz; pkgwithmpkg-1.0-py2.7-macosx10.7.mpkg.zip
-----------------------------------------------------------------
used for osx test case (tests.test_finder:test_no_mpkg)
-simple-[123].0.tar.gz
+simple[2]-[123].0.tar.gz
---------------------
-contains "simple" package; good for basic testing and version logic.
+contains "simple[2]" package; good for basic testing and version logic.
Upper-[12].0.tar.gz and requiresuppper-1.0.tar.gz
--------------------------------------------------
View
BIN  tests/packages/simple2-1.0.tar.gz
Binary file not shown
View
BIN  tests/packages/simple2-2.0.tar.gz
Binary file not shown
View
BIN  tests/packages/simple2-3.0.tar.gz
Binary file not shown
View
8 tests/test_basic.py
@@ -10,6 +10,7 @@
from pip.util import rmtree, find_command
from pip.exceptions import BadCommand
+from pip.backwardcompat import ssl
from tests.test_pip import (here, reset_env, run_pip, pyversion, mkdir,
src_folder, write_file, path_to_url)
@@ -48,7 +49,12 @@ def test_pip_second_command_line_interface_works():
Check if ``pip-<PYVERSION>`` commands behaves equally
"""
e = reset_env()
- result = e.run('pip-%s' % pyversion, 'install', 'INITools==0.2')
+
+ args = ['pip-%s' % pyversion]
+ if not ssl:
+ args.append('--insecure')
+ args.extend(['install', 'INITools==0.2'])
+ result = e.run(*args)
egg_info_folder = e.site_packages / 'INITools-0.2-py%s.egg-info' % pyversion
initools_folder = e.site_packages / 'initools'
assert egg_info_folder in result.files_created, str(result)
View
12 tests/test_config.py
@@ -1,7 +1,7 @@
import os
import tempfile
import textwrap
-from tests.test_pip import reset_env, run_pip, clear_environ, write_file
+from tests.test_pip import reset_env, run_pip, clear_environ, write_file, path_to_url, here
def test_options_from_env_vars():
@@ -79,9 +79,10 @@ def test_command_line_append_flags():
result = run_pip('install', '-vvv', 'INITools', expect_error=True)
assert "Analyzing links from page http://pypi.pinaxproject.com" in result.stdout
reset_env(environ)
- result = run_pip('install', '-vvv', '--find-links', 'http://example.com', 'INITools', expect_error=True)
+ find_links = path_to_url(os.path.join(here, 'packages'))
+ result = run_pip('install', '-vvv', '--find-links', find_links, 'INITools', expect_error=True)
assert "Analyzing links from page http://pypi.pinaxproject.com" in result.stdout
- assert "Analyzing links from page http://example.com" in result.stdout
+ assert "Skipping link %s" % find_links in result.stdout
def test_command_line_appends_correctly():
@@ -90,12 +91,13 @@ def test_command_line_appends_correctly():
"""
environ = clear_environ(os.environ.copy())
- environ['PIP_FIND_LINKS'] = 'http://pypi.pinaxproject.com http://example.com'
+ find_links = path_to_url(os.path.join(here, 'packages'))
+ environ['PIP_FIND_LINKS'] = 'http://pypi.pinaxproject.com %s' % find_links
reset_env(environ)
result = run_pip('install', '-vvv', 'INITools', expect_error=True)
assert "Analyzing links from page http://pypi.pinaxproject.com" in result.stdout, result.stdout
- assert "Analyzing links from page http://example.com" in result.stdout, result.stdout
+ assert "Skipping link %s" % find_links in result.stdout
@rasky
rasky added a note

Since we're at it: shouldn't --find-links also require --allow-no-ssl? It can be a separate patch, of course (I can work on it).

@dstufft Owner
dstufft added a note

Forcing SSL on find links doesn't really buy near as much as forcing it on PyPI does. Further more we don't know for sure that find links will even have SSL (or if it is, valid SSL), and we don't want to have people disable SSL for their Find links and have it disable their main repo protection as well.

@rasky
rasky added a note

Well, the URL pointed by --find-links can still be MITM'd to serve compromised packages, as far as I can see. Obviously it's less important than fixing the PyPI one, but it's probably something to consider while we enforce pip security.

Does --find-links work with HTTPS urls? Maybe we should emit a warning if the URL being passed is HTTP?

@dstufft Owner
dstufft added a note

I think a warning with HTTP would def be good. --find-links requires user actions to enable it so we'll be secure out of the box still. The biggest concern I have with tying it to --allow-no-ssl is losing PyPI SSL protection b/c one of your find links doesn't have a valid SSL cert.

@qwcode Owner
qwcode added a note

the way it's written now, the use of the ssl url opener is for any connection, index urls or find_link urls, whether they are "http" or "https"

and --allow-no-ssl just means "it's ok to use the non-ssl url opener when ssl is not importable".

e.g.:

pip -vv  install --no-index --find-links=https://pypi.python.org/simple/peppercorn/ peppercorn
[...]
 Analyzing links from page https://pypi.python.org/simple/peppercorn/
[...]
 Found link https://pypi.python.org/packages/source/p/peppercorn/peppercorn- [..]

this all seems to work, but should it be changed? should we use the non-ssl opener for "http:" urls?

@qwcode Owner
qwcode added a note

should we use the non-ssl opener for "http:" urls?

when accessing "http" urls, our opener (a normal opener plus the https handler) will try and use the http handler first (and succeed), before exercising the https handler (which it would fail on), so I think it's ok.

@rasky
rasky added a note

Oh, that's a security issue then. It means that an attacker can still MITM by having a non-SSL HTTP proxy server on port 443.

The opener used by pip to access PyPI (and all https URLs) should only allow SSL connections; there should not be any fallback to HTTP (not even if SSL fails).

@qwcode Owner
qwcode added a note

ok, if it really does fail in this case (and I agree it seems it could given the way it processes thru the chain of handlers; just me looking at urlllib2 code; not an actual experiment), I think it can be prevented pretty easily, by passing in our own dummy HTTPHandler that fails in all cases, in addition to our verification handler.

from the urllib2 docs for build_opener:
"Instances of the following classes will be in front of the handlers, unless the handlers contain them, instances of them or subclasses of them: ProxyHandler, UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler, HTTPErrorProcessor"

@qwcode Owner
qwcode added a note

also, to be clear, there is no "fallback" to the http handler if the https handler fails. the issue is that it seems to try the http handler first.

I guess the most direct thing is to specifically contruct an OpenerDirector instance with one handler, and not use build_opener

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
def test_config_file_override_stack():
View
50 tests/test_list.py
@@ -1,9 +1,10 @@
import os
import re
import textwrap
-from tests.test_pip import pyversion, reset_env, run_pip, write_file
+from tests.test_pip import pyversion, reset_env, run_pip, write_file, path_to_url, here
from tests.local_repos import local_checkout
+find_links = path_to_url(os.path.join(here, 'packages'))
def test_list_command():
"""
@@ -11,10 +12,10 @@ def test_list_command():
"""
reset_env()
- run_pip('install', 'INITools==0.2', 'mock==0.7.0')
+ run_pip('install', '-f', find_links, '--no-index', 'simple==1.0', 'simple2==3.0')
result = run_pip('list')
- assert 'INITools (0.2)' in result.stdout, str(result)
- assert 'mock (0.7.0)' in result.stdout, str(result)
+ assert 'simple (1.0)' in result.stdout, str(result)
+ assert 'simple2 (3.0)' in result.stdout, str(result)
def test_local_flag():
@@ -23,9 +24,9 @@ def test_local_flag():
"""
reset_env()
- run_pip('install', 'mock==0.7.0')
+ run_pip('install', '-f', find_links, '--no-index', 'simple==1.0')
result = run_pip('list', '--local')
- assert 'mock (0.7.0)' in result.stdout
+ assert 'simple (1.0)' in result.stdout
def test_uptodate_flag():
@@ -34,15 +35,12 @@ def test_uptodate_flag():
"""
reset_env()
- total_re = re.compile('INSTALLED: +([0-9.\w]+)')
- run_pip('install', 'pytz', 'mock==0.8.0')
+ run_pip('install', '-f', find_links, '--no-index', 'simple==1.0', 'simple2==3.0')
run_pip('install', '-e', 'git+https://github.com/pypa/pip-test-package.git#egg=pip-test-package')
- result = run_pip('search', 'pytz')
- pytz_ver = total_re.search(str(result)).group(1)
- result = run_pip('list', '--uptodate')
- assert not 'mock (0.8.0)' in result.stdout
+ result = run_pip('list', '-f', find_links, '--no-index', '--uptodate')
+ assert 'simple (1.0)' not in result.stdout #3.0 is latest
assert 'pip-test-package' not in result.stdout #editables excluded
- assert 'pytz (%s)' % pytz_ver in result.stdout
+ assert 'simple2 (3.0)' in result.stdout, str(result)
def test_outdated_flag():
@@ -50,23 +48,13 @@ def test_outdated_flag():
Test the behavior of --outdated flag in the list command
"""
- env = reset_env()
- total_re = re.compile('LATEST: +([0-9.\w]+)')
- write_file('req.txt', textwrap.dedent("""\
- INITools==0.2
- # and something else to test out:
- mock==0.7.0
- -e git+https://github.com/pypa/pip-test-package.git#egg=pip-test-package
- """))
- run_pip('install', '-r', env.scratch_path/'req.txt')
- result = run_pip('search', 'mock')
- mock_ver = total_re.search(str(result)).group(1)
- result = run_pip('search', 'INITools')
- initools_ver = total_re.search(str(result)).group(1)
- result = run_pip('list', '--outdated', expect_stderr=True)
- assert 'INITools (Current: 0.2 Latest: %s)' % initools_ver in result.stdout, str(result)
+ reset_env()
+ run_pip('install', '-f', find_links, '--no-index', 'simple==1.0', 'simple2==3.0')
+ run_pip('install', '-e', 'git+https://github.com/pypa/pip-test-package.git#egg=pip-test-package')
+ result = run_pip('list', '-f', find_links, '--no-index', '--outdated')
+ assert 'simple (Current: 1.0 Latest: 3.0)' in result.stdout
assert 'pip-test-package' not in result.stdout #editables excluded
- assert 'mock (Current: 0.7.0 Latest: %s)' % mock_ver in result.stdout, str(result)
+ assert 'simple2' not in result.stdout, str(result) #3.0 is latest
def test_editables_flag():
@@ -74,9 +62,9 @@ def test_editables_flag():
Test the behavior of --editables flag in the list command
"""
reset_env()
- run_pip('install', 'mock==0.7.0')
+ run_pip('install', '-f', find_links, '--no-index', 'simple==1.0')
result = run_pip('install', '-e', 'git+https://github.com/pypa/pip-test-package.git#egg=pip-test-package')
result = run_pip('list', '--editable')
- assert 'mock (0.7.0)' not in result.stdout, str(result)
+ assert 'simple (1.0)' not in result.stdout, str(result)
assert os.path.join('src', 'pip-test-package') in result.stdout, str(result)
View
36 tests/test_pip.py
@@ -1,6 +1,7 @@
#!/usr/bin/env python
import os
import sys
+import re
import tempfile
import shutil
import glob
@@ -11,6 +12,11 @@
from scripttest import TestFileEnvironment, FoundDir
from tests.path import Path, curdir, u
from pip.util import rmtree
+from pip.backwardcompat import ssl
+
+#allow py25 unit tests to work
+if sys.version_info[:2] == (2, 5) and not ssl:
+ os.environ['PIP_INSECURE'] = '1'
pyversion = sys.version[:3]
@@ -101,7 +107,11 @@ def install_setuptools(env):
env = None
-def reset_env(environ=None, use_distribute=None, system_site_packages=False, sitecustomize=None):
+def reset_env(environ=None,
+ use_distribute=None,
+ system_site_packages=False,
+ sitecustomize=None,
+ insecure=True):
"""Return a test environment.
Keyword arguments:
@@ -109,6 +119,7 @@ def reset_env(environ=None, use_distribute=None, system_site_packages=False, sit
use_distribute: use distribute, not setuptools.
system_site_packages: create a virtualenv that simulates --system-site-packages.
sitecustomize: a string containing python code to add to sitecustomize.py.
+ insecure: how to set the --insecure option for py25 tests.
"""
global env
@@ -125,6 +136,10 @@ def reset_env(environ=None, use_distribute=None, system_site_packages=False, sit
#hence, this workaround
(env.lib_path/'no-global-site-packages.txt').rm()
+ if sys.version_info[:2] == (2, 5) and (not ssl) and insecure:
+ #allow py25 tests to work
+ env.environ['PIP_INSECURE'] = '1'
+
return env
@@ -186,10 +201,14 @@ def __str__(self):
def __str__(self):
return str(self._impl)
- def assert_installed(self, pkg_name, with_files=[], without_files=[], without_egg_link=False, use_user_site=False):
+ def assert_installed(self, pkg_name, editable=True, with_files=[], without_files=[], without_egg_link=False, use_user_site=False):
e = self.test_env
- pkg_dir = e.venv/ 'src'/ pkg_name.lower()
+ if editable:
+ pkg_dir = e.venv/ 'src'/ pkg_name.lower()
+ else:
+ without_egg_link = True
+ pkg_dir = e.site_packages / pkg_name
if use_user_site:
egg_link_path = e.user_site / pkg_name + '.egg-link'
@@ -681,6 +700,17 @@ def main():
cwd=version_pkg_path, expect_stderr=True)
+def assert_raises_regexp(exception, reg, run, *args, **kwargs):
+ """Like assertRaisesRegexp in unittest"""
+ try:
+ run(*args, **kwargs)
+ assert False, "%s should have been thrown" %exception
+ except Exception:
+ e = sys.exc_info()[1]
+ p = re.compile(reg)
+ assert p.search(str(e)), str(e)
+
+
if __name__ == '__main__':
sys.stderr.write("Run pip's tests using nosetests. Requires virtualenv, ScriptTest, mock, and nose.\n")
sys.exit(1)
View
140 tests/test_ssl.py
@@ -0,0 +1,140 @@
+
+import sys
+import os
+from mock import Mock, patch
+from pip.download import urlopen, VerifiedHTTPSHandler
+from tests.test_pip import assert_raises_regexp, here, reset_env, run_pip
+from nose import SkipTest
+from nose.tools import assert_raises
+from pip.backwardcompat import urllib2, ssl, URLError
+from pip.exceptions import PipError
+
+pypi_https = 'https://pypi.python.org/simple/'
+pypi_http = 'http://pypi.python.org/simple/'
+
+class Tests_py25:
+ """py25 tests"""
+
+ def setup(self):
+ if sys.version_info >= (2, 6):
+ raise SkipTest()
+
+ def teardown(self):
+ #make sure this is set back for other tests
+ os.environ['PIP_INSECURE'] = '1'
+
+ def test_https_fails(self):
+ """
+ Test py25 access https fails
+ """
+ os.environ['PIP_INSECURE'] = ''
+ assert_raises_regexp(PipError, 'ssl certified', urlopen.get_opener, scheme='https')
+
+ def test_https_ok_with_flag(self):
+ """
+ Test py25 access https url ok with --insecure flag
+ This doesn't mean it's doing cert verification, just accessing over https
+ """
+ os.environ['PIP_INSECURE'] = '1'
+ response = urlopen.get_opener().open(pypi_https)
+ assert response.code == 200, str(dir(response))
+
+ def test_http_ok(self):
+ """
+ Test http pypi access with pip urlopener
+ """
+ os.environ['PIP_INSECURE'] = ''
+ response = urlopen.get_opener().open(pypi_http)
+ assert response.code == 200, str(dir(response))
+
+ def test_install_fails_with_no_ssl_backport(self):
+ """
+ Test installing w/o ssl backport fails
+ """
+ reset_env(insecure=False)
+ #expect error because ssl's setup.py is hard coded to install test data to global prefix
+ result = run_pip('install', 'INITools', expect_error=True)
+ assert "You don't have an importable ssl module" in result.stdout
+
+ def test_install_with_ssl_backport(self):
+ """
+ Test installing with ssl backport
+ """
+
+ #unable to get ssl backport to install on travis.
+ raise SkipTest()
+
+ # insecure=True so we can install ssl first
+ env = reset_env(insecure=True)
+ #expect error because ssl's setup.py is hard coded to install test data to global prefix
+ result = run_pip('install', 'ssl', expect_error=True)
+
+ #set it back to false
+ env.environ['PIP_INSECURE'] = ''
+ result = run_pip('install', 'INITools', expect_error=True)
+ result.assert_installed('initools', editable=False)
+
+
+class Tests_not_py25:
+ """non py25 tests"""
+
+ def setup(self):
+ if sys.version_info < (2, 6):
+ raise SkipTest()
+
+ def teardown(self):
+ os.environ['PIP_CERT'] = ''
+
+
+ def test_https_ok(self):
+ """
+ Test https pypi access with pip urlopener
+ """
+ response = urlopen.get_opener(scheme='https').open(pypi_https)
+ assert response.getcode() == 200, str(dir(response))
+
+ def test_http_ok(self):
+ """
+ Test http pypi access with pip urlopener
+ """
+ response = urlopen.get_opener().open(pypi_http)
+ assert response.getcode() == 200, str(dir(response))
+
+ def test_https_opener_director_handlers(self):
+ """
+ Confirm the expected handlers in our https OpenerDirector instance
+ We're specifically testing it does *not* contain the default http handler
+ """
+ o = urlopen.get_opener(scheme='https')
+ handler_types = [h.__class__ for h in o.handlers]
+
+ assert handler_types == [
+ urllib2.UnknownHandler,
+ urllib2.HTTPDefaultErrorHandler,
+ urllib2.HTTPRedirectHandler,
+ urllib2.FTPHandler,
+ urllib2.FileHandler,
+ VerifiedHTTPSHandler, #our cert check handler
+ urllib2.HTTPErrorProcessor
+ ], str(handler_types)
+
+ @patch('ssl.SSLSocket.getpeercert')
+ def test_fails_with_no_cert_returning(self, mock_getpeercert):
+ """
+ Test get ValueError if pypi returns no cert.
+ """
+ mock_getpeercert.return_value = None
+ o = urlopen.get_opener(scheme='https')
+ assert_raises_regexp(ValueError, 'empty or no certificate', o.open, pypi_https)
+
+
+ def test_bad_pem_fails(self):
+ """
+ Test ssl verification fails with bad pem file.
+ Also confirms alternate --cert-path option works
+ """
+ bad_cert = os.path.join(here, 'packages', 'README.txt')
+ os.environ['PIP_CERT'] = bad_cert
+ o = urlopen.get_opener(scheme='https')
+ assert_raises_regexp(URLError, '[sS][sS][lL]', o.open, pypi_https)
+
Something went wrong with that request. Please try again.