Skip to content
This repository

SSL certificate verification. #791

Merged
merged 59 commits into from over 1 year ago

9 participants

Marcus Smith Donald Stufft Giovanni Bajo James Cleveland Jannis Leidel Hugo Lopes Tavares Noah Kantrowitz Paul Nasrat Éric Araujo
Marcus Smith
Owner

builds on original pull #789

Done:

  • move ssl and match_hostname imports to backwardcompat
  • added match_hostname function to pip (no external dependency)
  • cert path moved to locations module
  • better error message when UrlError.reason is CertificateError or SSLError
  • added license for CA Root ceriticates
  • miscellaneous test fixes
  • --cert-path option to override path to pem file
  • add new tests
  • when scheme is 'https' and have ssl module, add in ssl cert verifying handler (and exclude http handler). otherwise, use build_opener to generate a "standard" chain of handlers.
  • --allow-no-ssl option to allow lack of cert verification when no ssl module (for py25 users who won't install ssl)
  • update to latest certs securely (https://gist.github.com/jjb/996292)
  • update authors file and release notes
  • update pip docs
  • py25 socket patch to work with ssl backport
  • only show --allow-no-ssl when no ssl
  • use the words "CA bundle" instead of "certificate file"
  • get an install test working that installs ssl backport for py25
  • refactor backwardcompat to a package
  • py25 ssl backport install test works locally, but skipping on travis for now.
  • fix the "pip list" tests to use our local packages for testing
  • wait for PSF's new cert, then remove old cacert.org cert from pem file
  • http://cheeseshop.python.org/ 503 errors have been fixed
  • updates based on jannis review

Todo:

  • merge and release an RC.
pip/util.py
@@ -664,3 +666,18 @@ def call_subprocess(cmd, show_stdout=True,
664 666 % (command_desc, proc.returncode, cwd))
665 667 if stdout is not None:
666 668 return ''.join(all_output)
  669 +
  670 +
  671 +def warn_if_no_ssl():
  672 + """Warn when there's no ssl"""
  673 + if not ssl:
  674 + logger.warn(textwrap.dedent("""
  675 + #############################################################
  676 + ## WARNING!! ##
  677 + ## You don't have an importable ssl module. ##
  678 + ## We can not provide ssl certified downloads from PyPI. ##
  679 + ## Install this: http://pypi.python.org/pypi/ssl/ ##
9
Giovanni Bajo
rasky added a note

You might want to use "https" in this URL :)

Marcus Smith Owner
qwcode added a note

yes : )

Giovanni Bajo
rasky added a note

As an additional note, I think that we should probably wait for a keypress, or wait either 10 seconds or a keypress, (or exit and ask to be relaunched with a command line option, eg: --exploit-me). I know it's annoying, but there is a remote code execution vulnerability here; if you simply print a warning and get on downloading the file and running its setup.py, it might be too late for the user.

Marcus Smith Owner
qwcode added a note

I'll go with the consensus, but I was concerned about breaking people's automated processes. @jezdez, @pnasrat ?

Giovanni Bajo
rasky added a note

If that's an (understandable) concern, a delay of 10 seconds (shortened by a keypress) should probably solve it.

Paul Nasrat Collaborator
pnasrat added a note

I don't like --exploit-me as an option just --nossl is probably sufficient. We probably don't want to break scripts, just announce vocally.

For package maintainers we should also have this in the installation docs.

Marcus Smith Owner
qwcode added a note

@pnasrat but if they don't use --nossl ?

which option?
1) just a message with no delay
2) 10s delay which gives people time to quit or continue
3) fail

Donald Stufft Owner
dstufft added a note

If they don't explictly disable SSL fail w/ an error telling them how to bypass it.

Jannis Leidel Owner
jezdez added a note

3)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Donald Stufft
Owner

We should ensure that any SSL enabled index url is only ever accessed via SSL. In particular this protects against a SSL stripping[1] attack where an attacker is able to convert the request to plaintext (and thus be able to modify the request) before sending it via SSL itself to PyPI.

[1] http://www.thoughtcrime.org/software/sslstrip/

Giovanni Bajo

@dstufft I don't think it's a problem in the context of the patch since the PyPI index URL does not (currently) redirect to the HTTPS version, and pip will hardcode the HTTPS version anyway.

Donald Stufft
Owner

@rasky What will happen if pip requests the HTTPS version but receives a plaintext response?

Giovanni Bajo

@dstufft Given that the patch explicitly wraps the TCP socket with ssl.wrap_socket, the SSL negotiation would fail because they won't speak the same protocol ( = ssl.wrap_socket() will raise an exception).

This said, please do test it :)

Donald Stufft
Owner
James Cleveland

With regards to CAcert, is the strategy to go with CAcert "for now", and then remove it once PyPI has a good certificate?

For a free Class 1 (no EV) certificate, I'd definitely look into StartSSL.

Marcus Smith
Owner

I think PyPI is close to having a new cert. The idea right now, is that I would remove the cacert.org cert before merge.

pip/util.py
@@ -664,3 +666,18 @@ def call_subprocess(cmd, show_stdout=True,
664 666 % (command_desc, proc.returncode, cwd))
665 667 if stdout is not None:
666 668 return ''.join(all_output)
  669 +
  670 +
  671 +def raise_no_ssl_exception():
  672 + """Raise when there's no ssl and not using '--no-ssl'"""
  673 + raise PipError(textwrap.dedent("""
  674 + #############################################################
  675 + ## You don't have an importable ssl module. ##
  676 + ## We can not provide ssl certified downloads. ##
  677 + ## Do one of 2 things: ##
  678 + ## 1) Install this: https://pypi.python.org/pypi/ssl/ ##
  679 + ## (It provides ssl support for older Pythons ) ##
  680 + ## 2) Use the --no-ssl option to allow this insecurity ##
1
Giovanni Bajo
rasky added a note

It's --allow-no-ssl now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Giovanni Bajo rasky commented on the diff
tests/test_config.py
((7 lines not shown))
94 96 reset_env(environ)
95 97 result = run_pip('install', '-vvv', 'INITools', expect_error=True)
96 98
97 99 assert "Analyzing links from page http://pypi.pinaxproject.com" in result.stdout, result.stdout
98   - assert "Analyzing links from page http://example.com" in result.stdout, result.stdout
  100 + assert "Skipping link %s" % find_links in result.stdout
9
Giovanni Bajo
rasky added a note

Since we're at it: shouldn't --find-links also require --allow-no-ssl? It can be a separate patch, of course (I can work on it).

Donald Stufft Owner
dstufft added a note

Forcing SSL on find links doesn't really buy near as much as forcing it on PyPI does. Further more we don't know for sure that find links will even have SSL (or if it is, valid SSL), and we don't want to have people disable SSL for their Find links and have it disable their main repo protection as well.

Giovanni Bajo
rasky added a note

Well, the URL pointed by --find-links can still be MITM'd to serve compromised packages, as far as I can see. Obviously it's less important than fixing the PyPI one, but it's probably something to consider while we enforce pip security.

Does --find-links work with HTTPS urls? Maybe we should emit a warning if the URL being passed is HTTP?

Donald Stufft Owner
dstufft added a note

I think a warning with HTTP would def be good. --find-links requires user actions to enable it so we'll be secure out of the box still. The biggest concern I have with tying it to --allow-no-ssl is losing PyPI SSL protection b/c one of your find links doesn't have a valid SSL cert.

Marcus Smith Owner
qwcode added a note

the way it's written now, the use of the ssl url opener is for any connection, index urls or find_link urls, whether they are "http" or "https"

and --allow-no-ssl just means "it's ok to use the non-ssl url opener when ssl is not importable".

e.g.:

pip -vv  install --no-index --find-links=https://pypi.python.org/simple/peppercorn/ peppercorn
[...]
 Analyzing links from page https://pypi.python.org/simple/peppercorn/
[...]
 Found link https://pypi.python.org/packages/source/p/peppercorn/peppercorn- [..]

this all seems to work, but should it be changed? should we use the non-ssl opener for "http:" urls?

Marcus Smith Owner
qwcode added a note

should we use the non-ssl opener for "http:" urls?

when accessing "http" urls, our opener (a normal opener plus the https handler) will try and use the http handler first (and succeed), before exercising the https handler (which it would fail on), so I think it's ok.

Giovanni Bajo
rasky added a note

Oh, that's a security issue then. It means that an attacker can still MITM by having a non-SSL HTTP proxy server on port 443.

The opener used by pip to access PyPI (and all https URLs) should only allow SSL connections; there should not be any fallback to HTTP (not even if SSL fails).

Marcus Smith Owner
qwcode added a note

ok, if it really does fail in this case (and I agree it seems it could given the way it processes thru the chain of handlers; just me looking at urlllib2 code; not an actual experiment), I think it can be prevented pretty easily, by passing in our own dummy HTTPHandler that fails in all cases, in addition to our verification handler.

from the urllib2 docs for build_opener:
"Instances of the following classes will be in front of the handlers, unless the handlers contain them, instances of them or subclasses of them: ProxyHandler, UnknownHandler, HTTPHandler, HTTPDefaultErrorHandler, HTTPRedirectHandler, FTPHandler, FileHandler, HTTPErrorProcessor"

Marcus Smith Owner
qwcode added a note

also, to be clear, there is no "fallback" to the http handler if the https handler fails. the issue is that it seems to try the http handler first.

I guess the most direct thing is to specifically contruct an OpenerDirector instance with one handler, and not use build_opener

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/basecommand.py
@@ -106,6 +107,12 @@ def main(self, args, initial_options):
106 107 if options.exists_action:
107 108 os.environ['PIP_EXISTS_ACTION'] = ''.join(options.exists_action)
108 109
  110 + if options.allow_no_ssl:
  111 + os.environ['PIP_ALLOW_NO_SSL'] = '1'
2
Giovanni Bajo
rasky added a note

Is there any security implication in using the environmentfor security-related configuration (PIP_ALLOW_NO_SSL, PIP_CERT_PATH)? I guess that if an attacker can modify the environment in which pip is run, she can as well change the PYTHONPATH to her hacked version of pip, so maybe it's not a big problem.

Comments?

Donald Stufft Owner
dstufft added a note

I wouldn't be real worried about it. If an attacker can set envvars we've lost the game for that computer already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
tests/test_pip.py
@@ -11,6 +11,10 @@
11 11 from scripttest import TestFileEnvironment, FoundDir
12 12 from tests.path import Path, curdir, u
13 13 from pip.util import rmtree
  14 +from pip.backwardcompat import ssl
  15 +
  16 +#allow py25 unit tests to work
  17 +os.environ['PIP_ALLOW_NO_SSL'] = '1'
2
Hugo Lopes Tavares Collaborator
hltbra added a note

Should all tests skip ssl by default? I see there is the condition to include that if no ssl module found at run_pip, it should not be here (at top), right?

Marcus Smith Owner
qwcode added a note

--allow-no-ssl means that it's ok to use the non-ssl url opener when ssl is not installed (which is only true on py25)
it doesn't mean "don't use ssl". to be clear though, I will add a condition such that's only set for py25

also, I'm going to add a specific test case that runs just for py25 that installs ssl (http://pypi.python.org/pypi/ssl/) and confirm that an install against pypi works using that library

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/download.py
((24 lines not shown))
  92 + self.sock = ssl.wrap_socket(sock,
  93 + self.key_file,
  94 + self.cert_file,
  95 + cert_reqs=ssl.CERT_REQUIRED,
  96 + ca_certs=cert_path)
  97 + match_hostname(self.sock.getpeercert(), self.host)
  98 +
  99 +
  100 +class VerifiedHTTPSHandler(urllib2.HTTPSHandler, urllib2.HTTPHandler):
  101 + """
  102 + A HTTPSHandler that wraps connections with ssl certificate verification.
  103 + By inheriting from both HTTPSHandler and HTTPHandler, this overrides
  104 + the default https *and* http handlers during the 'build_opener' routine.
  105 + We specifically *don't* want a http handler in the chain of handlers
  106 + to prevent MITM attacks that spoof https servers with http content.
  107 + """
1
Marcus Smith Owner
qwcode added a note

although the double inheritance works to eliminate the httphandler, this feels weird.
maybe just better to delete the httphandler from the handler list after the OpenerDirector instance is built.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/baseparser.py
@@ -350,4 +350,20 @@ def create_main_parser():
350 350 metavar='action',
351 351 help="Default action when a path already exists: "
352 352 "(s)witch, (i)gnore, (w)ipe, (b)ackup."),
  353 +
  354 + optparse.make_option(
  355 + '--allow-no-ssl',
  356 + dest='allow_no_ssl',
  357 + action='store_true',
  358 + default=False,
  359 + help = "Allow lack of certificate checking when ssl is not installed."),
  360 +
  361 + optparse.make_option(
  362 + '--cert-path',
  363 + dest='cert_path',
  364 + type='str',
  365 + default='',
  366 + metavar='path',
  367 + help = "Path to alternate certificate file."),
2
Giovanni Bajo
rasky added a note

The most common name is probably "CA bundle" instead of "certificate file". If I read certificate file, I think of a single certificate (and thus I cannot understand why pip would need a certificate).

Marcus Smith Owner
qwcode added a note

ok, I'll change the help line. "Path to alternate CA bundle."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/baseparser.py
@@ -350,4 +350,20 @@ def create_main_parser():
350 350 metavar='action',
351 351 help="Default action when a path already exists: "
352 352 "(s)witch, (i)gnore, (w)ipe, (b)ackup."),
  353 +
  354 + optparse.make_option(
  355 + '--allow-no-ssl',
  356 + dest='allow_no_ssl',
  357 + action='store_true',
  358 + default=False,
  359 + help = "Allow lack of certificate checking when ssl is not installed."),
2
Giovanni Bajo
rasky added a note

Shouldn't this option itself be guarded by if ssl of if py25, so not to clutter the --help output for most users? It does not have any effect for them anyway.

Marcus Smith Owner
qwcode added a note

agreed, I'll do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Jannis Leidel
Owner

@qwcode Would you mind moving the backwardcompat_ssl.py into backwardcompat/ssl_match_hostname.py please?

Marcus Smith
Owner

@jezdez, sure, np.

Marcus Smith
Owner

@jezdez, you mentioned on twitter, pip using the requests project for ssl verification. is that effort happening somewhere? should we reconsider this pull?

Hugo Lopes Tavares
Collaborator

What about pip search that uses XMLRPC and HTTP?

Noah Kantrowitz

New SSL layout on PyPI is live. Gogogo :runner::runner::runner:

Marcus Smith
Owner

sorting out test failures that are arising as a result....
couple of tests end up hitting http://cheeseshop.python.org which now returns 503...
intentional?

Jannis Leidel
Owner

D'oh!

Jannis Leidel jezdez commented on the diff
pip/backwardcompat/socket_create_connection.py
((21 lines not shown))
  21 + host, port = address
  22 + err = None
  23 + for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
  24 + af, socktype, proto, canonname, sa = res
  25 + sock = None
  26 + try:
  27 + sock = socket.socket(af, socktype, proto)
  28 + if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
  29 + sock.settimeout(timeout)
  30 + if source_address:
  31 + sock.bind(source_address)
  32 + sock.connect(sa)
  33 + return sock
  34 +
  35 + except socket.error:
  36 + err = sys.exc_info()[1]
1
Jannis Leidel Owner
jezdez added a note

sys is undefined here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
NOTICE.txt
... ... @@ -0,0 +1,18 @@
1
Jannis Leidel Owner
jezdez added a note

This file can be ported over to LICENSE.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
docs/installing.txt
@@ -3,11 +3,21 @@
3 3 Installation
4 4 ============
5 5
6   -Python Support
7   ---------------
  6 +.. warning::
  7 +
  8 + Prior to version 1.3, pip did not use SSL for downloading packages from PyPI, and thus left
  9 + users more vulnerable to security threats. We advise installing at least version 1.3.
  10 + If you're using `virtualenv <http://www.virtualenv.org>`_ to install pip, we advise installing
  11 + at least version 1.8.5, which contains pip version 1.3.
1
Jannis Leidel Owner
jezdez added a note

I think we should make a 1.9 release to mark pip 1.3 inclusion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
docs/installing.txt
((35 lines not shown))
  70 +
  71 +Using a Package Manager
  72 ++++++++++++++++++++++++
  73 +
  74 +On Linux, pip is packaged by most distributions. For instance, on an Ubuntu
  75 +system, you can install it with::
  76 +
  77 + $ sudo apt-get install python-pip
  78 +
  79 +On a Fedora system, you can install it with::
  80 +
  81 + $ yum install python-pip
  82 +
  83 +The latest versions of pip may not be available using this method.
  84 +
  85 +
4
Jannis Leidel Owner
jezdez added a note

The package manager section is clearly to help people get the safest experience. The problem though is that they almost always have older versions of pip available (e.g. http://packages.ubuntu.com/search?keywords=python-pip) that would make the latest efforts to secure pip moot. Please remove this section till we have confirmation that the latest pip is available through the native package managers.

Giovanni Bajo
rasky added a note

Just before this paragraph, there is a warning explaining that you really want pip 1.3.

What about adding a line saying "make sure that your distribution packages pip 1.3 or later, otherwise use a manual installation as explained below".

Jannis Leidel Owner
jezdez added a note

See, I don't believe people actually read all the paragraphs, but only look for the apt-get line they need to copy/paste. So even providing that part is a no-go for me.

That said, maybe we should reach out to the distros activately and ask them to update. Or.. even provide packages for the different distros (e.g. an own PPA).

Paul Nasrat Collaborator
pnasrat added a note
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
docs/logic.txt
@@ -151,17 +151,37 @@ pip offers a set of :ref:`Package Index Options <Package Index Options>` for mod
151 151 See the :ref:`pip install Examples<pip install Examples>`.
152 152
153 153
  154 +.. _`SSL Certificate Verification`:
  155 +
  156 +SSL Certificate Verification
  157 +============================
  158 +
  159 +Starting with v1.3, pip provides SSL certificate verification over https, for the purpose
  160 +of providing secure, certified downloads from PyPI.
  161 +
  162 +This is supported by default in all Python versions pip supports, except Python 2.5.
  163 +
  164 +Python 2.5 users can install https://pypi.python.org/pypi/ssl/, which provides ssl support for older pythons.
1
Jannis Leidel Owner
jezdez added a note

We should extend that paragraph a bit, with a step by step guide how to install the ssl package for *NIX and Windows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/basecommand.py
@@ -69,11 +69,15 @@ def _copy_option_group(self, parser, group):
69 69
70 70 def merge_options(self, initial_options, options):
71 71 # Make sure we have all global options carried over
72   - for attr in ['log', 'proxy', 'require_venv',
  72 + attrs = ['log', 'proxy', 'require_venv',
73 73 'log_explicit_levels', 'log_file',
1
Jannis Leidel Owner
jezdez added a note

Indentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/basecommand.py
@@ -104,6 +108,12 @@ def main(self, args, initial_options):
104 108 if options.exists_action:
105 109 os.environ['PIP_EXISTS_ACTION'] = ''.join(options.exists_action)
106 110
  111 + if not ssl and options.allow_no_ssl:
  112 + os.environ['PIP_ALLOW_NO_SSL'] = '1'
1
Jannis Leidel Owner
jezdez added a note

Why do we need to set an environment variable here? The option system in pip was built so it can accept env vars and pass them as part of the optparse options to the functions that need it, not the other way around.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/basecommand.py
@@ -104,6 +108,12 @@ def main(self, args, initial_options):
104 108 if options.exists_action:
105 109 os.environ['PIP_EXISTS_ACTION'] = ''.join(options.exists_action)
106 110
  111 + if not ssl and options.allow_no_ssl:
  112 + os.environ['PIP_ALLOW_NO_SSL'] = '1'
  113 +
  114 + if options.cert_path:
  115 + os.environ['PIP_CERT_PATH'] = options.cert_path
2
Jannis Leidel Owner
jezdez added a note

Same as above.

Marcus Smith Owner
qwcode added a note

these options are used in places where the options object in the command isn't currently tunneled into.
we'd need to pass options down a couple of levels. what do you recommend?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/baseparser.py
@@ -356,4 +356,21 @@ def create_main_parser():
356 356 metavar='action',
357 357 help="Default action when a path already exists: "
358 358 "(s)witch, (i)gnore, (w)ipe, (b)ackup."),
  359 +
  360 + optparse.make_option(
  361 + '--cert-path',
1
Jannis Leidel Owner
jezdez added a note

Should we maybe just call that --cert, less to type :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/baseparser.py
((13 lines not shown))
359 368 ]
  369 +
  370 +if not ssl:
  371 + standard_options.append(optparse.make_option(
  372 + '--allow-no-ssl',
1
Jannis Leidel Owner
jezdez added a note

For this option I'd like to make it even clearer to the user what they are doing if they don't use this. Let's call it --insecure or some-such.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
pip/util.py
@@ -664,3 +666,18 @@ def call_subprocess(cmd, show_stdout=True,
664 666 % (command_desc, proc.returncode, cwd))
665 667 if stdout is not None:
666 668 return ''.join(all_output)
  669 +
  670 +
  671 +def raise_no_ssl_exception():
  672 + """Raise when there's no ssl and not using '--no-ssl'"""
  673 + raise PipError(textwrap.dedent("""
  674 + #############################################################
  675 + ## You don't have an importable ssl module. ##
  676 + ## We can not provide ssl certified downloads. ##
  677 + ## Do one of 2 things: ##
  678 + ## 1) Install this: https://pypi.python.org/pypi/ssl/ ##
  679 + ## (It provides ssl support for older Pythons ) ##
  680 + ## 2) Use the --allow-no-ssl option to allow insecurity ##
  681 + #############################################################
1
Jannis Leidel Owner
jezdez added a note

This is pretty good already, I fear that people will freak out if they see it for the first time though. Let's explain why we can't ship ssl support and link to our documentation explaining in detail how to install it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Marcus Smith
Owner

just read @jezdez comments. will post updates later....

Jannis Leidel
Owner

w00t!, @qwcode!

Marcus Smith
Owner

@hltbra , for pip search, I can switch the index url to 'https', but the xmlrpc lib is written internally to use the HTTP connection, and since we're not downloading using that command, I wasn't going to do much work right now to use a cert verifying connection. not sure it would work?

Jannis Leidel
Owner

Win!

Marcus Smith
Owner

signing off for now.

will check tomorrow for any more updates to be made.
hoping to be able cut RCs tomorrow.

Éric Araujo

Indent has two extra spaces.

Marcus Smith qwcode merged commit beb9dea into from
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 59 unique commits by 2 authors.

Feb 04, 2013
James Cleveland Changing the default index URL to use HTTPS. b9ea089
James Cleveland Added certificate validation against root CA file.
Credits to Joseph Turner for this code:

http://thejosephturner.com/blog/2011/03/19/https-certificate-verification-in-python-with-urllib2/

And also to Kenneth Reitz for the CA dump (with addition of
CAcert.org chain because that's what PyPI uses).

Which, combined with pip, enables us to go a step towards
validated SSL and reducing the impact of MITM attacks.
246e974
James Cleveland Added CA file to MANIFEST.in db79f83
James Cleveland Adding PEM file to package data. 39d84ab
James Cleveland Adding code to match_hostname. d54c695
James Cleveland added install_requires 7776cfc
Feb 06, 2013
Marcus Smith qwcode 'package_dir' not needed b97969f
Marcus Smith qwcode won't be installing the ssl backport as a dependency 6bd3bd4
Marcus Smith qwcode warn before installing if no ssl module 11ebe01
Marcus Smith qwcode match_hostname from py32 c612db0
Marcus Smith qwcode backwardcompat logic for ssl and match_hostname 73c4614
Marcus Smith qwcode use standard opener when no ssl 4bb5ac6
Marcus Smith qwcode move cert_path to locations module 7e20fd8
Marcus Smith qwcode log relevant message if URLError.reason is SSLError or CertificateError b2e0b6d
Marcus Smith qwcode py25 import fixes a4a9197
Feb 07, 2013
Marcus Smith qwcode license for CA Root Certificates d50a3b7
Marcus Smith qwcode misc test fixes 7fac01a
Marcus Smith qwcode --cert-path and --no-ssl options 8496406
Marcus Smith qwcode allow py25 tests to work without ssl 226768d
Marcus Smith qwcode more py25 test fixes 193562b
Feb 08, 2013
Marcus Smith qwcode tun py25 test logic and add assert_raises_regexp d8fe463
Marcus Smith qwcode fix param name in exception message 456ea80
Marcus Smith qwcode OpenerDirector for ssl should not contain the default http handler fe17bb4
Marcus Smith qwcode ssl cert tests d77380d
Marcus Smith qwcode remove duplicate backwardcompat imports and excess logic 609cfe9
Feb 09, 2013
Marcus Smith qwcode pypy ssl test fix 6dc3212
Marcus Smith qwcode shorter metavar for --cert-path bb7ba1a
Marcus Smith qwcode update authors and changelog e338436
Marcus Smith qwcode latest mozilla ca certs aquired securely and generated to pem form 857f64e
Marcus Smith qwcode ssl cert docs updates 49563cf
Marcus Smith qwcode py25 socket patch to work with ssl backport f92052f
Feb 10, 2013
Marcus Smith qwcode use the more common phrase 'CA bundle' 76b5ebc
Marcus Smith qwcode only show --allow-no-ssl when no ssl 83d8b37
Marcus Smith qwcode ssl docs fix 912784f
Marcus Smith qwcode --allow-no-ssl test fix 9cda4d6
Feb 11, 2013
Marcus Smith qwcode py25 install tests with ssl backport 8fc02eb
Marcus Smith qwcode refactor pip.backwardcompat from module to package 1cf1a7e
Marcus Smith qwcode update installation docs related to ssl a6a0ee3
Marcus Smith qwcode remove HTTPHandler explictly 84f8134
Feb 12, 2013
Marcus Smith qwcode unable to get ssl backport to install on travis f13f879
Feb 16, 2013
Marcus Smith qwcode use local packages, not 'mock', which has invalid ssl download link 9b57f89
Marcus Smith qwcode more local test packages 7c8470a
Marcus Smith qwcode merge with develop d0c3138
Marcus Smith qwcode clear up connection compatibility logic 809a996
Marcus Smith qwcode merge with develop 22bf924
Marcus Smith qwcode remove old cert from pem file b2a17e5
Marcus Smith qwcode fix import syntax 1857ece
Feb 18, 2013
Marcus Smith qwcode add missing sys import c4753b2
Marcus Smith qwcode move CA license to our license file 26f9c14
Marcus Smith qwcode our next virtualenv release will be 1.9 f72ddaa
Marcus Smith qwcode remove yum/apt-get instructions, since they will likely install non-s…
…sl versions for awhile
a2ba2dc
Marcus Smith qwcode proper list indents e3f1ce8
Marcus Smith qwcode from --cert-path to --cert 559d77a
Marcus Smith qwcode from --allow-no-ssl to --insecure 4a4a141
Marcus Smith qwcode have 'pip search' use https index url 039e1fc
Marcus Smith qwcode improve ssl exception text and docs 2cbc7fa
Marcus Smith qwcode custom NoSSLError exception instead of util function 889e1a0
Feb 19, 2013
Marcus Smith qwcode remove extra indents in docs db9c92a
Marcus Smith qwcode add TODO about options passing d6bb9a5
This page is out of date. Refresh to see the latest.

Showing 31 changed files with 4,505 additions and 83 deletions. Show diff stats Hide diff stats

  1. +1 0  AUTHORS.txt
  2. +3 0  CHANGES.txt
  3. +19 0 LICENSE.txt
  4. +1 0  MANIFEST.in
  5. +1 1  docs/index.txt
  6. +34 14 docs/installing.txt
  7. +58 2 docs/logic.txt
  8. +21 0 pip/{backwardcompat.py → backwardcompat/__init__.py}
  9. +44 0 pip/backwardcompat/socket_create_connection.py
  10. +60 0 pip/backwardcompat/ssl_match_hostname.py
  11. +19 6 pip/basecommand.py
  12. +18 1 pip/baseparser.py
  13. +3,895 0 pip/cacert.pem
  14. +1 1  pip/cmdoptions.py
  15. +1 1  pip/commands/search.py
  16. +80 9 pip/download.py
  17. +26 0 pip/exceptions.py
  18. +6 1 pip/index.py
  19. +1 0  pip/locations.py
  20. +2 2 pip/log.py
  21. +4 2 pip/util.py
  22. +2 1  setup.py
  23. +2 2 tests/packages/README.txt
  24. BIN  tests/packages/simple2-1.0.tar.gz
  25. BIN  tests/packages/simple2-2.0.tar.gz
  26. BIN  tests/packages/simple2-3.0.tar.gz
  27. +7 1 tests/test_basic.py
  28. +7 5 tests/test_config.py
  29. +19 31 tests/test_list.py
  30. +33 3 tests/test_pip.py
  31. +140 0 tests/test_ssl.py
1  AUTHORS.txt
@@ -25,6 +25,7 @@ Ian Bicking
25 25 Igor Sobreira
26 26 Ionel Maries Cristian
27 27 Jakub Vysoky
  28 +James Cleveland
28 29 Jannis Leidel
29 30 Jay Graves
30 31 John-Scott Atlakson
3  CHANGES.txt
@@ -4,6 +4,9 @@ Changelog
4 4 develop (unreleased)
5 5 --------------------
6 6
  7 +* SSL Cert Verification; Make https the default for PyPI access.
  8 + Thanks James Cleveland, Giovanni Bajo and many others (Pull #789).
  9 +
7 10 * Added "pip list" for listing installed packages and the latest version
8 11 available. Thanks Rafael Caricio, Miguel Araujo, Dmitry Gladkov (Pull #752)
9 12
19 LICENSE.txt
@@ -18,3 +18,22 @@ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
18 18 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
19 19 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
20 20 WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
  21 +
  22 +
  23 +License for Bundle of CA Root Certificates (pip/cacert.pem)
  24 +===========================================================
  25 +
  26 +This library is free software; you can redistribute it and/or
  27 +modify it under the terms of the GNU Lesser General Public
  28 +License as published by the Free Software Foundation; either
  29 +version 2.1 of the License, or (at your option) any later version.
  30 +
  31 +This library is distributed in the hope that it will be useful,
  32 +but WITHOUT ANY WARRANTY; without even the implied warranty of
  33 +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
  34 +Lesser General Public License for more details.
  35 +
  36 +You should have received a copy of the GNU Lesser General Public
  37 +License along with this library; if not, write to the Free Software
  38 +Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA
  39 +02110-1301
1  MANIFEST.in
@@ -2,6 +2,7 @@ include AUTHORS.txt
2 2 include LICENSE.txt
3 3 include CHANGES.txt
4 4 include PROJECT.txt
  5 +include pip/cacert.pem
5 6 recursive-include docs *.txt
6 7 recursive-include docs *.html
7 8 recursive-exclude docs/_build *.txt
2  docs/index.txt
@@ -6,7 +6,7 @@ A tool for installing and managing Python packages.
6 6 `Mailing list <http://groups.google.com/group/python-virtualenv>`_ ``|``
7 7 `Issues <https://github.com/pypa/pip/issues>`_ ``|``
8 8 `Github <https://github.com/pypa/pip>`_ ``|``
9   -`PyPI <http://pypi.python.org/pypi/pip/>`_ ``|``
  9 +`PyPI <https://pypi.python.org/pypi/pip/>`_ ``|``
10 10 irc:#pip
11 11
12 12
48 docs/installing.txt
@@ -3,11 +3,21 @@
3 3 Installation
4 4 ============
5 5
6   -Python Support
7   ---------------
  6 +.. warning::
  7 +
  8 + Prior to version 1.3, pip did not use SSL for downloading packages from PyPI, and thus left
  9 + users more vulnerable to security threats. We advise installing at least version 1.3.
  10 + If you're using `virtualenv <http://www.virtualenv.org>`_ to install pip, we advise installing
  11 + at least version 1.9, which contains pip version 1.3.
  12 +
  13 +
  14 +Python & OS Support
  15 +-------------------
8 16
9 17 pip works with CPython versions 2.5, 2.6, 2.7, 3.1, 3.2, 3.3 and also pypy.
10 18
  19 +pip works on Unix/Linux, OS X, and Windows.
  20 +
11 21
12 22 Using virtualenv
13 23 ----------------
@@ -33,27 +43,35 @@ Installing Globally
33 43 pip can be installed globally in order to manage global packages.
34 44 Often this requires the installation to be performed as root.
35 45
36   -Prerequisites
37   -+++++++++++++
  46 +.. warning::
  47 +
  48 + We advise against using easy_install to install pip, because easy_install
  49 + does not download from PyPI over SSL, so the installation might be insecure.
  50 + Since pip can then be used to install packages (which execute code on
  51 + your computer), it is better to go through a trusted path.
  52 +
38 53
39   -A global install of pip requires either `setuptools <http://pypi.python.org/pypi/setuptools>`_
40   -or `distribute <http://pypi.python.org/pypi/distribute>`_ to be installed globally as well.
  54 +Requirements
  55 +++++++++++++
41 56
42   -In many cases, these can be installed using your OS package manager.
  57 +pip requires either `setuptools <https://pypi.python.org/pypi/setuptools>`_
  58 +or `distribute <https://pypi.python.org/pypi/distribute>`_.
43 59
44   -See the `Distribute Install Instructions <http://pypi.python.org/pypi/distribute/>`_ or the
45   -`Setuptools Install Instructions <http://pypi.python.org/pypi/setuptools#installation-instructions>`_
  60 +See the `Distribute Install Instructions <https://pypi.python.org/pypi/distribute/>`_ or the
  61 +`Setuptools Install Instructions <https://pypi.python.org/pypi/setuptools#installation-instructions>`_
  62 +
  63 +If installing pip using a linux package manager, these requirements will be installed for you.
46 64
47 65 .. warning::
48 66
49 67 If you are using Python 3.X you **must** use distribute; setuptools doesn't
50 68 support Python 3.X.
51 69
  70 +
52 71 Using get-pip
53 72 +++++++++++++
54 73
55   -Download `get-pip.py <https://raw.github.com/pypa/pip/master/contrib/get-pip.py>`_
56   -and execute it using Python. This will only install pip, not the prerequisites.
  74 +After installing the requirements:
57 75
58 76 ::
59 77
@@ -61,12 +79,14 @@ and execute it using Python. This will only install pip, not the prerequisites.
61 79 $ [sudo] python get-pip.py
62 80
63 81
64   -From source
65   -+++++++++++
  82 +Installing from source
  83 +++++++++++++++++++++++
  84 +
  85 +After installing the requirements:
66 86
67 87 ::
68 88
69   - $ curl -O http://pypi.python.org/packages/source/p/pip/pip-X.X.tar.gz
  89 + $ curl -O https://pypi.python.org/packages/source/p/pip/pip-X.X.tar.gz
70 90 $ tar xvfz pip-X.X.tar.gz
71 91 $ cd pip-X.X
72 92 $ [sudo] python setup.py install
60 docs/logic.txt
@@ -151,17 +151,73 @@ pip offers a set of :ref:`Package Index Options <Package Index Options>` for mod
151 151 See the :ref:`pip install Examples<pip install Examples>`.
152 152
153 153
  154 +.. _`SSL Certificate Verification`:
  155 +
  156 +SSL Certificate Verification
  157 +============================
  158 +
  159 +Starting with v1.3, pip provides SSL certificate verification over https, for the purpose
  160 +of providing secure, certified downloads from PyPI.
  161 +
  162 +This is supported by default in all Python versions pip supports, except Python 2.5.
  163 +
  164 +Python 2.5 users can :ref:`install an SSL backport <SSL Backport>`, which provides ssl support for older pythons.
  165 +Pip does not try to install this automatically because it requires a compiler, which not all systems will have.
  166 +
  167 +Although not recommended, Python 2.5 users who are unable to install ssl, can use the global option,
  168 +``--insecure``, to allow access to PyPI w/o attempting SSL certificate verification. This option will only be visible
  169 +when ssl is not importable. This is *not* a general option.
  170 +
  171 +
  172 +.. _`SSL Backport`:
  173 +
  174 +Installing the SSL Backport
  175 +~~~~~~~~~~~~~~~~~~~~~~~~~~~
  176 +
  177 +.. warning::
  178 +
  179 + We advise against using ``pip`` itself to install the ssl backport, because it won't be secure
  180 + until *after* installing ssl. Likewise, ``easy_install`` is not advised, because it
  181 + does not currently support ssl.
  182 +
  183 +
  184 +1. Download the ssl archive:
  185 +
  186 + * Using a Browser:
  187 +
  188 + 1. Go to `this url <https://pypi.python.org/pypi/ssl/1.15>`_.
  189 + 2. Confirm the identity of the site is valid.
  190 + Most browsers provide this information to the left of the URL bar in the form of padlock icon that you can click on to confirm the site is verified.
  191 + 3. Scroll down, and click to download ``ssl-1.15.tar.gz``.
  192 +
  193 + * Using curl, which supports ssl certificate verification:
  194 + ::
  195 +
  196 + $ curl -O https://pypi.python.org/packages/source/s/ssl/ssl-1.15.tar.gz
  197 +
  198 +2. Confirm the md5sum:
  199 + ::
  200 + $ md5sum ssl-1.15.tar.gz
  201 + 81ea8a1175e437b4c769ae65b3290e0c ssl-1.15.tar.gz
  202 +
  203 +3. Unpack the archive, and change into the ``ssl-1.15`` directory.
  204 +4. Run: ``python setup.py install``.
  205 +
  206 +
154 207 Hash Verification
155 208 =================
156 209
157   -PyPI provides an md5 hash of a package by having the link to the
158   -package include an #md5=<hash>.
  210 +PyPI provides md5 hashes in the hash fragment of package download urls.
159 211
160 212 pip supports checking this, as well as any of the
161 213 guaranteed hashlib algorithms (sha1, sha224, sha384, sha256, sha512, md5).
162 214
163 215 The hash fragment is case sensitive (i.e. sha1 not SHA1).
164 216
  217 +This check is only intended to provide basic download corruption protection.
  218 +It is not intended to provide security against tampering. For that,
  219 +see :ref:`SSL Certificate Verification`
  220 +
165 221
166 222 Download Cache
167 223 ==============
21 pip/backwardcompat.py → pip/backwardcompat/__init__.py
@@ -112,3 +112,24 @@ def home_lib(home):
112 112 else:
113 113 lib = os.path.join('lib', 'python')
114 114 return os.path.join(home, lib)
  115 +
  116 +
  117 +## py25 has no builtin ssl module
  118 +## only >=py32 has ssl.match_hostname and ssl.CertificateError
  119 +try:
  120 + import ssl
  121 + try:
  122 + from ssl import match_hostname, CertificateError
  123 + except ImportError:
  124 + from pip.backwardcompat.ssl_match_hostname import match_hostname, CertificateError
  125 +except ImportError:
  126 + ssl = None
  127 +
  128 +
  129 +# patch for py25 socket to work with http://pypi.python.org/pypi/ssl/
  130 +import socket
  131 +if not hasattr(socket, 'create_connection'): # for Python 2.5
  132 + # monkey-patch socket module
  133 + from pip.backwardcompat.socket_create_connection import create_connection
  134 + socket.create_connection = create_connection
  135 +
44 pip/backwardcompat/socket_create_connection.py
... ... @@ -0,0 +1,44 @@
  1 +"""
  2 +patch for py25 socket to work with http://pypi.python.org/pypi/ssl/
  3 +copy-paste from py2.6 stdlib socket.py
  4 +https://gist.github.com/zed/1347055
  5 +"""
  6 +import socket
  7 +import sys
  8 +
  9 +_GLOBAL_DEFAULT_TIMEOUT = getattr(socket, '_GLOBAL_DEFAULT_TIMEOUT', object())
  10 +def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
  11 + source_address=None):
  12 + """Connect to *address* and return the socket object.
  13 +
  14 + Convenience function. Connect to *address* (a 2-tuple ``(host,
  15 + port)``) and return the socket object. Passing the optional
  16 + *timeout* parameter will set the timeout on the socket instance
  17 + before attempting to connect. If no *timeout* is supplied, the
  18 + global default timeout setting returned by :func:`getdefaulttimeout`
  19 + is used.
  20 + """
  21 +
  22 + host, port = address
  23 + err = None
  24 + for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
  25 + af, socktype, proto, canonname, sa = res
  26 + sock = None
  27 + try:
  28 + sock = socket.socket(af, socktype, proto)
  29 + if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
  30 + sock.settimeout(timeout)
  31 + if source_address:
  32 + sock.bind(source_address)
  33 + sock.connect(sa)
  34 + return sock
  35 +
  36 + except socket.error:
  37 + err = sys.exc_info()[1]
  38 + if sock is not None:
  39 + sock.close()
  40 +
  41 + if err is not None:
  42 + raise err
  43 + else:
  44 + raise socket.error("getaddrinfo returns an empty list")
60 pip/backwardcompat/ssl_match_hostname.py
... ... @@ -0,0 +1,60 @@
  1 +"""The match_hostname() function from Python 3.2, essential when using SSL."""
  2 +
  3 +import re
  4 +
  5 +__version__ = '3.2a3'
  6 +
  7 +class CertificateError(ValueError):
  8 + pass
  9 +
  10 +def _dnsname_to_pat(dn):
  11 + pats = []
  12 + for frag in dn.split(r'.'):
  13 + if frag == '*':
  14 + # When '*' is a fragment by itself, it matches a non-empty dotless
  15 + # fragment.
  16 + pats.append('[^.]+')
  17 + else:
  18 + # Otherwise, '*' matches any dotless fragment.
  19 + frag = re.escape(frag)
  20 + pats.append(frag.replace(r'\*', '[^.]*'))
  21 + return re.compile(r'\A' + r'\.'.join(pats) + r'\Z', re.IGNORECASE)
  22 +
  23 +def match_hostname(cert, hostname):
  24 + """Verify that *cert* (in decoded format as returned by
  25 + SSLSocket.getpeercert()) matches the *hostname*. RFC 2818 rules
  26 + are mostly followed, but IP addresses are not accepted for *hostname*.
  27 +
  28 + CertificateError is raised on failure. On success, the function
  29 + returns nothing.
  30 + """
  31 + if not cert:
  32 + raise ValueError("empty or no certificate")
  33 + dnsnames = []
  34 + san = cert.get('subjectAltName', ())
  35 + for key, value in san:
  36 + if key == 'DNS':
  37 + if _dnsname_to_pat(value).match(hostname):
  38 + return
  39 + dnsnames.append(value)
  40 + if not san:
  41 + # The subject is only checked when subjectAltName is empty
  42 + for sub in cert.get('subject', ()):
  43 + for key, value in sub:
  44 + # XXX according to RFC 2818, the most specific Common Name
  45 + # must be used.
  46 + if key == 'commonName':
  47 + if _dnsname_to_pat(value).match(hostname):
  48 + return
  49 + dnsnames.append(value)
  50 + if len(dnsnames) > 1:
  51 + raise CertificateError("hostname %r "
  52 + "doesn't match either of %s"
  53 + % (hostname, ', '.join(map(repr, dnsnames))))
  54 + elif len(dnsnames) == 1:
  55 + raise CertificateError("hostname %r "
  56 + "doesn't match %r"
  57 + % (hostname, dnsnames[0]))
  58 + else:
  59 + raise CertificateError("no appropriate commonName or "
  60 + "subjectAltName fields were found")
25 pip/basecommand.py
@@ -12,7 +12,7 @@
12 12 from pip.download import urlopen
13 13 from pip.exceptions import (BadCommand, InstallationError, UninstallationError,
14 14 CommandError)
15   -from pip.backwardcompat import StringIO
  15 +from pip.backwardcompat import StringIO, ssl
16 16 from pip.baseparser import ConfigOptionParser, UpdatingDefaultsHelpFormatter
17 17 from pip.status_codes import SUCCESS, ERROR, UNKNOWN_ERROR, VIRTUALENV_NOT_FOUND
18 18 from pip.util import get_prog
@@ -69,11 +69,15 @@ def _copy_option_group(self, parser, group):
69 69
70 70 def merge_options(self, initial_options, options):
71 71 # Make sure we have all global options carried over
72   - for attr in ['log', 'proxy', 'require_venv',
73   - 'log_explicit_levels', 'log_file',
74   - 'timeout', 'default_vcs',
75   - 'skip_requirements_regex',
76   - 'no_input', 'exists_action']:
  72 + attrs = ['log', 'proxy', 'require_venv',
  73 + 'log_explicit_levels', 'log_file',
  74 + 'timeout', 'default_vcs',
  75 + 'skip_requirements_regex',
  76 + 'no_input', 'exists_action',
  77 + 'cert']
  78 + if not ssl:
  79 + attrs.append('insecure')
  80 + for attr in attrs:
77 81 setattr(options, attr, getattr(initial_options, attr) or getattr(options, attr))
78 82 options.quiet += initial_options.quiet
79 83 options.verbose += initial_options.verbose
@@ -98,12 +102,21 @@ def main(self, args, initial_options):
98 102
99 103 self.setup_logging()
100 104
  105 + #TODO: try to get these passing down from the command?
  106 + # without resorting to os.environ to hold these.
  107 +
101 108 if options.no_input:
102 109 os.environ['PIP_NO_INPUT'] = '1'
103 110
104 111 if options.exists_action:
105 112 os.environ['PIP_EXISTS_ACTION'] = ''.join(options.exists_action)
106 113
  114 + if not ssl and options.insecure:
  115 + os.environ['PIP_INSECURE'] = '1'
  116 +
  117 + if options.cert:
  118 + os.environ['PIP_CERT'] = options.cert
  119 +
107 120 if options.require_venv:
108 121 # If a venv is required check if it can really be found
109 122 if not os.environ.get('VIRTUAL_ENV'):
19 pip/baseparser.py
@@ -6,7 +6,7 @@
6 6 import os
7 7 import textwrap
8 8 from distutils.util import strtobool
9   -from pip.backwardcompat import ConfigParser, string_types
  9 +from pip.backwardcompat import ConfigParser, string_types, ssl
10 10 from pip.locations import default_config_file, default_log_file
11 11 from pip.util import get_terminal_size, get_prog
12 12
@@ -356,4 +356,21 @@ def create_main_parser():
356 356 metavar='action',
357 357 help="Default action when a path already exists: "
358 358 "(s)witch, (i)gnore, (w)ipe, (b)ackup."),
  359 +
  360 + optparse.make_option(
  361 + '--cert',
  362 + dest='cert',
  363 + type='str',
  364 + default='',
  365 + metavar='path',
  366 + help = "Path to alternate CA bundle."),
  367 +
359 368 ]
  369 +
  370 +if not ssl:
  371 + standard_options.append(optparse.make_option(
  372 + '--insecure',
  373 + dest='insecure',
  374 + action='store_true',
  375 + default=False,
  376 + help = "Allow lack of certificate checking when ssl is not installed."))
3,895 pip/cacert.pem
3,895 additions, 0 deletions not shown
2  pip/cmdoptions.py
@@ -21,7 +21,7 @@ def make_option_group(group, parser):
21 21 '-i', '--index-url', '--pypi-url',
22 22 dest='index_url',
23 23 metavar='URL',
24   - default='http://pypi.python.org/simple/',
  24 + default='https://pypi.python.org/simple/',
25 25 help='Base URL of Python Package Index (default %default).')
26 26
27 27 extra_index_url = make_option(
2  pip/commands/search.py
@@ -24,7 +24,7 @@ def __init__(self, *args, **kw):
24 24 '--index',
25 25 dest='index',
26 26 metavar='URL',
27   - default='http://pypi.python.org/pypi',
  27 + default='https://pypi.python.org/pypi',
28 28 help='Base URL of Python Package Index (default %default)')
29 29
30 30 self.parser.insert_option_group(0, self.cmd_opts)
89 pip/download.py
@@ -5,18 +5,21 @@
5 5 import os
6 6 import re
7 7 import shutil
  8 +import socket
8 9 import sys
9 10 import tempfile
10 11
11   -from pip.backwardcompat import (xmlrpclib, urllib, urllib2,
12   - urlparse, string_types)
13   -from pip.exceptions import InstallationError
  12 +from pip.backwardcompat import (xmlrpclib, urllib, urllib2, httplib,
  13 + urlparse, string_types, ssl)
  14 +if ssl:
  15 + from pip.backwardcompat import match_hostname
  16 +from pip.exceptions import InstallationError, PipError, NoSSLError
14 17 from pip.util import (splitext, rmtree, format_size, display_path,
15 18 backup_dir, ask_path_exists, unpack_file,
16 19 create_download_cache_folder, cache_download)
17 20 from pip.vcs import vcs
18 21 from pip.log import logger
19   -
  22 +from pip.locations import default_cert_path
20 23
21 24 __all__ = ['xmlrpclib_transport', 'get_file_content', 'urlopen',
22 25 'is_url', 'url_to_path', 'path_to_url', 'path_to_url2',
@@ -66,6 +69,53 @@ def get_file_content(url, comes_from=None):
66 69 _scheme_re = re.compile(r'^(http|https|file):', re.I)
67 70 _url_slash_drive_re = re.compile(r'/*([a-z])\|', re.I)
68 71
  72 +class VerifiedHTTPSConnection(httplib.HTTPSConnection):
  73 + """
  74 + A connection that wraps connections with ssl certificate verification.
  75 + """
  76 + def connect(self):
  77 +
  78 + self.connection_kwargs = {}
  79 +
  80 + #TODO: refactor compatibility logic into backwardcompat?
  81 +
  82 + # for > py2.5
  83 + if hasattr(self, 'timeout'):
  84 + self.connection_kwargs.update(timeout = self.timeout)
  85 +
  86 + # for >= py2.7
  87 + if hasattr(self, 'source_address'):
  88 + self.connection_kwargs.update(source_address = self.source_address)
  89 +
  90 + sock = socket.create_connection((self.host, self.port), **self.connection_kwargs)
  91 +
  92 + # for >= py2.7
  93 + if getattr(self, '_tunnel_host', None):
  94 + self.sock = sock
  95 + self._tunnel()
  96 +
  97 + # get alternate bundle or use our included bundle
  98 + cert_path = os.environ.get('PIP_CERT', '') or default_cert_path
  99 +
  100 + self.sock = ssl.wrap_socket(sock,
  101 + self.key_file,
  102 + self.cert_file,
  103 + cert_reqs=ssl.CERT_REQUIRED,
  104 + ca_certs=cert_path)
  105 +
  106 + match_hostname(self.sock.getpeercert(), self.host)
  107 +
  108 +
  109 +class VerifiedHTTPSHandler(urllib2.HTTPSHandler):
  110 + """
  111 + A HTTPSHandler that uses our own VerifiedHTTPSConnection.
  112 + """
  113 + def __init__(self, connection_class = VerifiedHTTPSConnection):
  114 + self.specialized_conn_class = connection_class
  115 + urllib2.HTTPSHandler.__init__(self)
  116 + def https_open(self, req):
  117 + return self.do_open(self.specialized_conn_class, req)
  118 +
69 119
70 120 class URLOpener(object):
71 121 """
@@ -81,10 +131,10 @@ def __call__(self, url):
81 131 auth.
82 132
83 133 """
84   - url, username, password = self.extract_credentials(url)
  134 + url, username, password, scheme = self.extract_credentials(url)
85 135 if username is None:
86 136 try:
87   - response = urllib2.urlopen(self.get_request(url))
  137 + response = self.get_opener(scheme=scheme).open(url)
88 138 except urllib2.HTTPError:
89 139 e = sys.exc_info()[1]
90 140 if e.code != 401:
@@ -121,10 +171,31 @@ def get_response(self, url, username=None, password=None):
121 171 self.passman.add_password(None, netloc, username, password)
122 172 stored_username, stored_password = self.passman.find_user_password(None, netloc)
123 173 authhandler = urllib2.HTTPBasicAuthHandler(self.passman)
124   - opener = urllib2.build_opener(authhandler)
  174 + opener = self.get_opener(authhandler, scheme=scheme)
125 175 # FIXME: should catch a 401 and offer to let the user reenter credentials
126 176 return opener.open(req)
127 177
  178 + def get_opener(self, *args, **kwargs):
  179 + """
  180 + Build an OpenerDirector instance based on the scheme, whether ssl is
  181 + importable and the --insecure parameter.
  182 + """
  183 + if kwargs.get('scheme') == 'https':
  184 + if ssl:
  185 + https_handler = VerifiedHTTPSHandler()
  186 + director = urllib2.build_opener(https_handler, *args)
  187 + #strip out HTTPHandler to prevent MITM spoof
  188 + for handler in director.handlers:
  189 + if isinstance(handler, urllib2.HTTPHandler):
  190 + director.handlers.remove(handler)
  191 + return director
  192 + elif os.environ.get('PIP_INSECURE', '') == '1':
  193 + return urllib2.build_opener(*args)
  194 + else:
  195 + raise NoSSLError()
  196 + else:
  197 + return urllib2.build_opener(*args)
  198 +
128 199 def setup(self, proxystr='', prompting=True):
129 200 """
130 201 Sets the proxy handler given the option passed on the command
@@ -161,7 +232,7 @@ def extract_credentials(self, url):
161 232
162 233 username, password = self.parse_credentials(netloc)
163 234 if username is None:
164   - return url, None, None
  235 + return url, None, None, scheme
165 236 elif password is None and self.prompting:
166 237 # remove the auth credentials from the url part
167 238 netloc = netloc.replace('%s@' % username, '', 1)
@@ -173,7 +244,7 @@ def extract_credentials(self, url):
173 244 netloc = netloc.replace('%s:%s@' % (username, password), '', 1)
174 245
175 246 target_url = urlparse.urlunsplit((scheme, netloc, path, query, frag))
176   - return target_url, username, password
  247 + return target_url, username, password, scheme
177 248
178 249 def get_proxy(self, proxystr=''):
179 250 """
26 pip/exceptions.py
... ... @@ -1,5 +1,6 @@
1 1 """Exceptions used throughout package"""
2 2
  3 +import textwrap
3 4
4 5 class PipError(Exception):
5 6 """Base pip exception"""
@@ -28,3 +29,28 @@ class BadCommand(PipError):
28 29
29 30 class CommandError(PipError):
30 31 """Raised when there is an error in command-line arguments"""
  32 +
  33 +
  34 +class NoSSLError(PipError):
  35 + """Raised when there's no ssl and not using '--insecure'"""
  36 +
  37 + def __str__(self):