Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

PyOpenSSL for SNI-support on Python2 #156

Merged
merged 10 commits into from

4 participants

@kirkeby

This adds optional SNI-support for Python2, if pyOpenSSL, pyasn1 and ng-httpsclient are installed (pyasn1 and ng-httpsclient are used to parse the subjectAltName of certificates).

kirkeby added some commits
@kirkeby kirkeby We know best if we support SNI. d299a66
@kirkeby kirkeby Use pyOpenSSL if present.
This gives us SNI goodies on all Python versions.
953b5a4
@kirkeby kirkeby Fix subjectAltName support.
The SNI/subjectAltName tests have a one big problem: They access the
great big, bad Internet, because I am not sure how to create the
correct certificates for this.

Also, this only works in all cases if pyasn1 and ndg-httpsclient are
installed (parsing DER-encoded binary data extracted from X.509
extensions is not my idea of fun.)
df1d950
@shazow
Owner

Hmm I'd be -1 on merging this as-is. I'm not a fan of environment flags for libraries if we can avoid it, and this seems to depend on a lot of fragile things.

Perhaps a better strategy would be to refactor the API to allow you to plug in your own ssl_wrap_socket easily and then post this as a recipe somewhere? Or maybe even as a helper inside urrlib3/contrib/.

Does anyone else have feelings on this? (@wolever @wallunit @t-8ch, anyone else?)

@snoack

Has PyOpenSSL any significant drawbacks? Otherwise I would prefer to automatically pick it, if it is installed, and the built-in ssl module doesn't support SNI. I don't like the environment variable approach either.

@t-8ch

+1 for removing the env variables

@shazow: Wouldn't urllib3.util.ssl_wrap_socket = my_awesome_ssl_wrapping also
work?

@kirkeby:
For testing you may want to look at
test/with_dummyerver/test_socketlevel.py:TestSNI.
It is certainly not pretty, but works.

With the following patch this PR could also use the fingerprint
verification of #144.

From 9505d5673b4a0cbd0f501389401f017fe0c8ed69 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= <thomas@t-8ch.de>
Date: Fri, 15 Mar 2013 16:11:24 +0000
Subject: [PATCH] pyopenssl: add param `binary_form` to getpeercert

---
 urllib3/util.py | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/urllib3/util.py b/urllib3/util.py
index 555e753..57977d2 100644
--- a/urllib3/util.py
+++ b/urllib3/util.py
@@ -392,10 +392,16 @@ elif OpenSSL is not None:  # Use PyOpenSSL if installed
         def sendall(self, data):
             return self.connection.sendall(data)

-        def getpeercert(self):
+        def getpeercert(self, binary_form=False):
             x509 = self.connection.get_peer_certificate()
             if not x509:
                 raise SSLError('')
+
+            if binary_form:
+                return OpenSSL.crypto.dump_certificate(
+                    OpenSSL.crypto.FILETYPE_ASN1,
+                    x509)
+
             return {
                 'subject': (
                     (('commonName', x509.get_subject().CN),),
-- 
1.8.2
urllib3/util.py
((21 lines not shown))
+ ext_dat = ext.get_data()
+ decoded_dat = der_decoder.decode(ext_dat,
+ asn1Spec=general_names)
+
+ for name in decoded_dat:
+ if not isinstance(name, SubjectAltName):
+ continue
+ for entry in range(len(name)):
+ component = name.getComponentByPosition(entry)
+ if component.getName() != 'dNSName':
+ continue
+ dns_name.append(str(component.getComponent()))
+
+ return dns_name
+
+ class wrapped_socket(object):
@t-8ch
t-8ch added a note

I think this classname should use camelcase.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@kirkeby

The environment variables are gone now, and the wrapped_socket helper class is renamed WrappedSocket.

I'm open to attempt to refactor the API to allow pluggable ssl_wrap_socket, and moving this code into contrib, if that's what you think is best?

Regarding drawbacks with PyOpenSSL, I can only think of one: This implementation has not been tested as much as the stdlib-backed one.

@shazow
Owner

Looking better.

Alright, let's do this as a first step:

  1. Put it in a module in the urllib3.contrib namespace (not imported automatically).
  2. For now, we can add something like urllib3.contrib.pyopenssl.inject_into_urllib3() or something which monkeypatches urllib3.util.ssl_wrap_socket with its own version when called.
  3. Add some tests. Maybe we need a contrib testing submodule.
  4. At this point I'd be comfortable merging this first revision into the mainline.

Once we have this, then the next step would be to refactor how urllib3.util.ssl_wrap_socket is used and make it more configurable (so you can specify which socket wrapper to use). This will obsolete the need of monkeypatching. I can help with this.

How does that sound?

@kirkeby

Sounds good.

kirkeby added some commits
@kirkeby kirkeby Move PyOpenSSL-backed SSL into contrib. 35485d8
@kirkeby kirkeby Tests for urllib3.contrib.pyopenssl.
This makes the SNI-test raise SkipTest if SNI is not supported, so
it can be reused by test.contrib.test_pyopenssl even when the stdlib
does not support SNI.

It also tests the urllib3.contrib.pyopenssl module when its
dependencies are installed, by re-exporting HTTPS test-cases and
using module-level setup/teardown code to monkey patch urllib3.
a105119
@kirkeby

How does it look now?

@shazow
Owner

Looks decent for a contrib module. :)

Should we revert the changes to test/with_dummyserver/test_socketlevel.py? (Or move them into /test/contrib/...)

If you'd like to add a Sphinx docs section on the contrib module, specifically about your PyOpenSSL addition and how to use it, I would be +1 to that. Could do that later though.

@t-8ch

+1 for pulling in the changes to test_socketlevel.py

@shazow
Owner

@t-8ch Do you mean test/with_dummyserver/test_socketlevel.py or test/contrib/with_dummyserver/test_socketlevel.py?

@t-8ch

test/with_dummyserver/test_socketlevel.py The SkipTest looks right.

@shazow
Owner

Ah fair enough. Let's keep it.

@kirkeby

Okay. I hope I found the right place to add that contrib section :)

@shazow
Owner

@kirkeby Perfect, thanks! Want to add a little narrative section for how to use it? (Just a code sample or something.)

@t-8ch Can you think of anything else missing?

Otherwise I think it's ready to go.

@kirkeby

Now there's a ready-to-use code snippet in the module's doc-string, which is auto-moduled into the contrib page, is that okay?

@shazow shazow merged commit 9471fe9 into from
@shazow
Owner

Merged. :)

@kirkeby

Yay! Thanks :)

@shazow shazow referenced this pull request from a commit
@shazow CHANGES.rst += SNI support (#156) e363eaf
@schlamar schlamar referenced this pull request from a commit in schlamar/urllib3
@shazow CHANGES.rst += SNI support (#156) 8dd4ec1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Mar 12, 2013
  1. @kirkeby
  2. @kirkeby

    Use pyOpenSSL if present.

    kirkeby authored
    This gives us SNI goodies on all Python versions.
  3. @kirkeby

    Fix subjectAltName support.

    kirkeby authored
    The SNI/subjectAltName tests have a one big problem: They access the
    great big, bad Internet, because I am not sure how to create the
    correct certificates for this.
    
    Also, this only works in all cases if pyasn1 and ndg-httpsclient are
    installed (parsing DER-encoded binary data extracted from X.509
    extensions is not my idea of fun.)
Commits on Mar 20, 2013
  1. @kirkeby
  2. @kirkeby
  3. @kirkeby

    CamelCase for class name.

    kirkeby authored
Commits on Mar 21, 2013
  1. @kirkeby
  2. @kirkeby

    Tests for urllib3.contrib.pyopenssl.

    kirkeby authored
    This makes the SNI-test raise SkipTest if SNI is not supported, so
    it can be reused by test.contrib.test_pyopenssl even when the stdlib
    does not support SNI.
    
    It also tests the urllib3.contrib.pyopenssl module when its
    dependencies are installed, by re-exporting HTTPS test-cases and
    using module-level setup/teardown code to monkey patch urllib3.
  3. @kirkeby
  4. @kirkeby
This page is out of date. Refresh to see the latest.
View
3  CONTRIBUTORS.txt
@@ -48,5 +48,8 @@ In chronological order:
* hartator <hartator@gmail.com>
* Corrected multipart behavior for params
+* Sune Kirkeby <mig@ibofobi.dk>
+ * Optional SNI-support for Python 2 via PyOpenSSL.
+
* [Your name or handle] <[email or website]>
* [Brief summary of your changes]
View
10 docs/contrib.rst
@@ -0,0 +1,10 @@
+Contrib Modules
+===============
+
+These modules implement various extra features, that may not be ready for
+prime time.
+
+SNI-support for Python 2
+------------------------
+
+.. automodule:: urllib3.contrib.pyopenssl
View
10 docs/index.rst
@@ -9,6 +9,7 @@ urllib3 Documentation
managers
helpers
collections
+ contrib
Highlights
@@ -144,6 +145,15 @@ but can also be used independently.
helpers
+Contrib Modules
+---------------
+
+These modules implement various extra features, that may not be ready for
+prime time.
+
+.. toctree::
+
+ contrib
Contributing
============
View
0  test/contrib/__init__.py
No changes.
View
17 test/contrib/test_pyopenssl.py
@@ -0,0 +1,17 @@
+try:
+ from urllib3.contrib.pyopenssl import (inject_into_urllib3,
+ extract_from_urllib3)
+except ImportError as e:
+ from nose.plugins.skip import SkipTest
+ raise SkipTest('Could not import pyopenssl: %r' % e)
+
+from ..with_dummyserver.test_https import TestHTTPS, TestHTTPS_TLSv1
+from ..with_dummyserver.test_socketlevel import TestSNI, TestSocketClosing
+
+
+def setup_module():
+ inject_into_urllib3()
+
+
+def teardown_module():
+ extract_from_urllib3()
View
47 test/with_dummyserver/test_socketlevel.py
@@ -1,16 +1,13 @@
from urllib3 import HTTPConnectionPool, HTTPSConnectionPool
from urllib3.poolmanager import proxy_from_url
from urllib3.exceptions import MaxRetryError, TimeoutError, SSLError
+from urllib3 import util
from dummyserver.testcase import SocketDummyServerTestCase
+from nose.plugins.skip import SkipTest
from threading import Event
-try:
- from ssl import HAS_SNI
-except ImportError: # openssl without SNI
- HAS_SNI = False
-
class TestCookies(SocketDummyServerTestCase):
@@ -34,28 +31,30 @@ def multicookie_response_handler(listener):
self.assertEquals(r.headers, {'set-cookie': 'foo=1, bar=1'})
-if HAS_SNI:
- class TestSNI(SocketDummyServerTestCase):
+class TestSNI(SocketDummyServerTestCase):
- def test_hostname_in_first_request_packet(self):
- done_receiving = Event()
- self.buf = b''
+ def test_hostname_in_first_request_packet(self):
+ if not util.HAS_SNI:
+ raise SkipTest('SNI-support not available')
- def socket_handler(listener):
- sock = listener.accept()[0]
+ done_receiving = Event()
+ self.buf = b''
- self.buf = sock.recv(65536) # We only accept one packet
- done_receiving.set() # let the test know it can proceed
-
- self._start_server(socket_handler)
- pool = HTTPSConnectionPool(self.host, self.port)
- try:
- pool.request('GET', '/', retries=0)
- except SSLError: # We are violating the protocol
- pass
- done_receiving.wait()
- self.assertTrue(self.host.encode() in self.buf,
- "missing hostname in SSL handshake")
+ def socket_handler(listener):
+ sock = listener.accept()[0]
+
+ self.buf = sock.recv(65536) # We only accept one packet
+ done_receiving.set() # let the test know it can proceed
+
+ self._start_server(socket_handler)
+ pool = HTTPSConnectionPool(self.host, self.port)
+ try:
+ pool.request('GET', '/', retries=0)
+ except SSLError: # We are violating the protocol
+ pass
+ done_receiving.wait()
+ self.assertTrue(self.host.encode() in self.buf,
+ "missing hostname in SSL handshake")
class TestSocketClosing(SocketDummyServerTestCase):
View
161 urllib3/contrib/pyopenssl.py
@@ -0,0 +1,161 @@
+'''SSL with SNI-support for Python 2.
+
+This needs the following packages installed:
+
+* pyOpenSSL (tested with 0.13)
+* ndg-httpsclient (tested with 0.3.2)
+* pyasn1 (tested with 0.1.6)
+
+To activate it call :func:`~urllib3.contrib.pyopenssl.inject_into_urllib3`.
+This can be done in a ``sitecustomize`` module, or at any other time before
+your application begins using ``urllib3``, like this::
+
+ try:
+ import urllib3.contrib.pyopenssl
+ urllib3.contrib.pyopenssl.inject_into_urllib3()
+ except ImportError:
+ pass
+
+Now you can use :mod:`urllib3` as you normally would, and it will support SNI
+when the required modules are installed.
+'''
+
+from ndg.httpsclient.ssl_peer_verification import (ServerSSLCertVerification,
+ SUBJ_ALT_NAME_SUPPORT)
+from ndg.httpsclient.subj_alt_name import SubjectAltName
+import OpenSSL.SSL
+from pyasn1.codec.der import decoder as der_decoder
+from socket import _fileobject
+import ssl
+
+from .. import connectionpool
+from .. import util
+
+__all__ = ['inject_into_urllib3', 'extract_from_urllib3']
+
+# SNI only *really* works if we can read the subjectAltName of certificates.
+HAS_SNI = SUBJ_ALT_NAME_SUPPORT
+
+# Map from urllib3 to PyOpenSSL compatible parameter-values.
+_openssl_versions = {
+ ssl.PROTOCOL_SSLv23: OpenSSL.SSL.SSLv23_METHOD,
+ ssl.PROTOCOL_SSLv3: OpenSSL.SSL.SSLv3_METHOD,
+ ssl.PROTOCOL_TLSv1: OpenSSL.SSL.TLSv1_METHOD,
+}
+_openssl_verify = {
+ ssl.CERT_NONE: OpenSSL.SSL.VERIFY_NONE,
+ ssl.CERT_OPTIONAL: OpenSSL.SSL.VERIFY_PEER,
+ ssl.CERT_REQUIRED: OpenSSL.SSL.VERIFY_PEER
+ + OpenSSL.SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
+}
+
+
+orig_util_HAS_SNI = util.HAS_SNI
+orig_connectionpool_ssl_wrap_socket = connectionpool.ssl_wrap_socket
+
+
+def inject_into_urllib3():
+ 'Monkey-patch urllib3 with PyOpenSSL-backed SSL-support.'
+
+ connectionpool.ssl_wrap_socket = ssl_wrap_socket
+ util.HAS_SNI = HAS_SNI
+
+
+def extract_from_urllib3():
+ 'Undo monkey-patching by :func:`inject_into_urllib3`.'
+
+ connectionpool.ssl_wrap_socket = orig_connectionpool_ssl_wrap_socket
+ util.HAS_SNI = orig_util_HAS_SNI
+
+
+### Note: This is a slightly bug-fixed version of same from ndg-httpsclient.
+def get_subj_alt_name(peer_cert):
+ # Search through extensions
+ dns_name = []
+ if not SUBJ_ALT_NAME_SUPPORT:
+ return dns_name
+
+ general_names = SubjectAltName()
+ for i in range(peer_cert.get_extension_count()):
+ ext = peer_cert.get_extension(i)
+ ext_name = ext.get_short_name()
+ if ext_name != 'subjectAltName':
+ continue
+
+ # PyOpenSSL returns extension data in ASN.1 encoded form
+ ext_dat = ext.get_data()
+ decoded_dat = der_decoder.decode(ext_dat,
+ asn1Spec=general_names)
+
+ for name in decoded_dat:
+ if not isinstance(name, SubjectAltName):
+ continue
+ for entry in range(len(name)):
+ component = name.getComponentByPosition(entry)
+ if component.getName() != 'dNSName':
+ continue
+ dns_name.append(str(component.getComponent()))
+
+ return dns_name
+
+
+class WrappedSocket(object):
+ '''API-compatibility wrapper for Python OpenSSL's Connection-class.'''
+
+ def __init__(self, connection, socket):
+ self.connection = connection
+ self.socket = socket
+
+ def makefile(self, mode, bufsize=-1):
+ return _fileobject(self.connection, mode, bufsize)
+
+ def settimeout(self, timeout):
+ return self.socket.settimeout(timeout)
+
+ def sendall(self, data):
+ return self.connection.sendall(data)
+
+ def getpeercert(self):
+ x509 = self.connection.get_peer_certificate()
+ if not x509:
+ raise ssl.SSLError('')
+ return {
+ 'subject': (
+ (('commonName', x509.get_subject().CN),),
+ ),
+ 'subjectAltName': [
+ ('DNS', value)
+ for value in get_subj_alt_name(x509)
+ ]
+ }
+
+
+def _verify_callback(cnx, x509, err_no, err_depth, return_code):
+ return err_no == 0
+
+
+def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
+ ca_certs=None, server_hostname=None,
+ ssl_version=None):
+ ctx = OpenSSL.SSL.Context(_openssl_versions[ssl_version])
+ if certfile:
+ ctx.use_certificate_file(certfile)
+ if keyfile:
+ ctx.use_privatekey_file(keyfile)
+ if cert_reqs != ssl.CERT_NONE:
+ ctx.set_verify(_openssl_verify[cert_reqs], _verify_callback)
+ if ca_certs:
+ try:
+ ctx.load_verify_locations(ca_certs, None)
+ except OpenSSL.SSL.Error as e:
+ raise ssl.SSLError('bad ca_certs: %r' % ca_certs, e)
+
+ cnx = OpenSSL.SSL.Connection(ctx, sock)
+ cnx.set_tlsext_host_name(server_hostname)
+ cnx.set_connect_state()
+ try:
+ cnx.do_handshake()
+ except OpenSSL.SSL.Error as e:
+ raise ssl.SSLError('bad handshake', e)
+
+ return WrappedSocket(cnx, sock)
Something went wrong with that request. Please try again.