Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Manual specification of hostname to verify against #140

Merged
merged 9 commits into from

3 participants

Thomas Weißschuh Andrey Petrov Piotr Dobrogost
Thomas Weißschuh

This allows people to manually specify the hostname they want to verify their ssl certs against.
This should help if SNI insn't available, the server returns the wrong cert or one connects directly to IP addresses.

(Reference: kennethreitz/requests#1124, kennethreitz/requests#749)
This could be enhanced to verify against fingerprints.

urllib3/util.py
@@ -24,6 +26,7 @@
import ssl
from ssl import wrap_socket, CERT_NONE, SSLError, PROTOCOL_SSLv23
+ from .exceptions import SSLError
Andrey Petrov Owner
shazow added a note

Add this to the .exceptions import below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
@@ -302,6 +305,43 @@ def resolve_ssl_version(candidate):
return candidate
+
+def match_fingerprint(remote, local):
+ """
+ Compares if both supplied fingerprints match.
+
+ remote -- binary
Andrey Petrov Owner
shazow added a note

Can we keep this consistent with urllib3's style?

:param remote: <Description>

etc

Andrey Petrov Owner
shazow added a note

Not completely clear what "binary" means.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
@@ -302,6 +305,43 @@ def resolve_ssl_version(candidate):
return candidate
+
+def match_fingerprint(remote, local):
+ """
+ Compares if both supplied fingerprints match.
+
+ remote -- binary
+ local -- hexstring, can be separated by colons
+ """
+
+ # maps the raw byte length of a digest to its hash function
Andrey Petrov Owner
shazow added a note

Sentence-case comments please (capitalize first letter, period if it's a sentence).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
@@ -302,6 +305,43 @@ def resolve_ssl_version(candidate):
return candidate
+
+def match_fingerprint(remote, local):
Andrey Petrov Owner
shazow added a note

Since this function doesn't return anything but just raises errors, perhaps we should call it something like assert_fingerprint?

Thomas Weißschuh
t-8ch added a note

I named it after match_hostname.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/connectionpool.py
@@ -80,13 +80,15 @@ class VerifiedHTTPSConnection(HTTPSConnection):
ca_certs = None
ssl_version = None
- def set_cert(self, key_file=None, cert_file=None,
- cert_reqs=None, ca_certs=None):
+ def set_cert(self, key_file=None, cert_file=None, cert_reqs=None,
+ ca_certs=None, verify_hostname=None, verify_fingerprint=None):
Andrey Petrov Owner
shazow added a note

PEP8 indent on (

Ideally, group similar params (e.g. cert_, verify_) per line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/connectionpool.py
((8 lines not shown))
self.key_file = key_file
self.cert_file = cert_file
self.cert_reqs = cert_reqs
self.ca_certs = ca_certs
+ self.verify_hostname = verify_hostname
Andrey Petrov Owner
shazow added a note

Maybe call these s/verify/assert/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
CONTRIBUTORS.txt
@@ -46,7 +46,11 @@ In chronological order:
* Support for explicitly closing pooled connections
* hartator <hartator@gmail.com>
- * Corrected multipart behavior for params
+ * Corrected multipart behavior for params
+
+* Thomas Weißschuh <thomas@t-8ch.de>
+ * various SSL patches
Andrey Petrov Owner
shazow added a note

Could you change this to "Various SSL patches, with tests"?

Or better yet, "SSL fingerprint checking, with tests"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
@@ -302,6 +305,43 @@ def resolve_ssl_version(candidate):
return candidate
+
+def match_fingerprint(remote, local):
+ """
+ Compares if both supplied fingerprints match.
Andrey Petrov Owner
shazow added a note

Could you give a better description of what this method is for and when it should be used?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Thomas Weißschuh

@shazow This doesn't really work for Pools to multiple hosts. Are those "supported" and worth effort?

Andrey Petrov
Owner

PoolManager? Absolutely, it's the most-used use case for urllib3 (requests depends on it, for example).

Andrey Petrov
Owner

Or do you mean multiple hosts in a single HTTP(S)ConnectionPool object? That should not happen.

Thomas Weißschuh

I am thinking about assert_same_host in in urlopen()
If this is false a connection can be used with a different host than constructed.
Am I getting this wrong?

Andrey Petrov
Owner

Oh, by host you mean literally the Host: header? Yes. But the IPs must match.

Thomas Weißschuh

Then the current behaviour of of this PR should be sufficient.
I hope the style is acceptably now, that was really not my day -.-

Thomas Weißschuh

@shazow If you are happy with this I'll add the docs.

urllib3/util.py
((22 lines not shown))
+ fingerprint = fingerprint.replace(':', '').lower()
+
+ digest_length, rest = divmod(len(fingerprint), 2)
+
+ if rest or digest_length not in hashfunc_map:
+ raise SSLError('Fingerprint is of invalid length')
+
+ # We need encode() here for py32, works on py2 and p33
+ fingerprint_bytes = unhexlify(fingerprint.encode())
+
+ hashfunc = hashfunc_map[digest_length]
+
+ cert_digest = hashfunc(cert).digest()
+
+ if not cert_digest == fingerprint_bytes:
+ raise SSLError('Fingerprints did not match!\n'
Andrey Petrov Owner
shazow added a note

We don't have multiline exception messages anywhere. Perhaps make this consistent?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
((12 lines not shown))
+ Fingerprint as string of hexdigits, can be interspersed by colons.
+ """
+
+ # Maps the length of a digest to a possible hash function producing
+ # this digest
+ hashfunc_map = {
+ 16: md5,
+ 20: sha1
+ }
+
+ fingerprint = fingerprint.replace(':', '').lower()
+
+ digest_length, rest = divmod(len(fingerprint), 2)
+
+ if rest or digest_length not in hashfunc_map:
+ raise SSLError('Fingerprint is of invalid length')
Andrey Petrov Owner
shazow added a note

I believe we put periods at the end of message sentences.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Andrey Petrov shazow commented on the diff
urllib3/util.py
@@ -302,6 +304,45 @@ def resolve_ssl_version(candidate):
return candidate
+
+def assert_fingerprint(cert, fingerprint):
+ """
+ Checks if given fingerprint matches the supplied certificate.
+
+ :param cert:
+ Certificate as bytes object.
+ :param fingerprint:
+ Fingerprint as string of hexdigits, can be interspersed by colons.
+ """
+
+ # Maps the length of a digest to a possible hash function producing
+ # this digest
Andrey Petrov Owner
shazow added a note

Periods at the end of sentence comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Andrey Petrov
Owner

Looks good to me. :)

Docs appreciated.

Thomas Weißschuh

Here we go. Any opinions to my other PRs?

Andrey Petrov
Owner

Sorry I haven't had much time to do OSS stuff these last couple of weeks. Next week should be way better though. :)

Thomas Weißschuh

@shazow While you are at it: 'ping' :sunglasses:

Andrey Petrov
Owner

Thank you. :)

Reviewed the latest version of the changes, looks good! I'm happy to merge it next cycle (probably next week).

I wonder if all of the assert_* configuration is getting out of hand and if we should think about a different place or API for those kinds of things. Happy to leave that off for the future though.

Thomas Weißschuh

What about a configuration class Verification, like Retries and Timeout?

Andrey Petrov
Owner

Hmm might be getting out of hand, or maybe not. Out of curiosity, what do you imagine such a class would look like? :)

Thomas Weißschuh

This was just a reference to those other config objects you mentioned on another issue.
I had a use case like this in mind:

pool = HTTPSConnectionpool(host, port,
           verify=Verification(fingerprint='abc', key_file='/foo', cert_file='/bar')

Like some sort of algebraic datatype:

verify = None
verify = Verification(hostname='foo')
verify = Verification(callback=some_func)

My motivation for a fine grained (bloated) API like this is the elimination of all excuses for not verifying certificates in a reasonable manner.

Piotr Dobrogost

I like the idea of grouping parameters pertaining to some aspect of configuration.

Andrey Petrov
Owner

Hmm interesting indeed. Then the verification logic can live in the Verification object, which would leave HTTPSConnectionPool much cleaner. Will need to think about what objects the Verification logic would need access to. Looks like just the VerifiedHTTPSConnection should do the trick.

Thomas Weißschuh

ping @shazow
Rebased to trigger travis. Added two more tests to keep coverage at 100%.

Andrey Petrov shazow merged commit 7f9d8ee into from
Andrey Petrov
Owner

Fantastic. @t-8ch, thank you so much for following through with this. :)

Andrey Petrov
Owner

Hmm. How would you describe this for the CHANGES.rst?

Thomas Weißschuh

"Added mechanism to verify ssl certificates by fingerprint (md5, sha1) or against an arbitrary hostname (when connecting by IP or for misconfigured servers)"

Andrey Petrov
Owner

+1

Andrey Petrov
Owner

Cool. I think this is a pretty sizeable chunk for v1.6. Maybe I should version bump soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
8 CONTRIBUTORS.txt
View
@@ -46,7 +46,13 @@ In chronological order:
* Support for explicitly closing pooled connections
* hartator <hartator@gmail.com>
- * Corrected multipart behavior for params
+ * Corrected multipart behavior for params
+
+* Thomas Weißschuh <thomas@t-8ch.de>
+ * Support for TLS SNI
+ * API unification of ssl_version/cert_reqs
+ * SSL fingerprint and alternative hostname verification
+ * Bugfixes in testsuite
* Sune Kirkeby <mig@ibofobi.dk>
* Optional SNI-support for Python 2 via PyOpenSSL.
49 test/with_dummyserver/test_https.py
View
@@ -137,6 +137,55 @@ def test_set_ssl_version_to_sslv3(self):
self._pool.ssl_version = ssl.PROTOCOL_SSLv3
self.assertRaises(SSLError, self._pool.request, 'GET', '/')
+ def test_assert_specific_hostname(self):
+ https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+ cert_reqs='CERT_REQUIRED')
+
+ https_pool.ca_certs = DEFAULT_CA
+ https_pool.assert_hostname = 'localhost'
+ https_pool.request('GET', '/')
+
+ def test_assert_fingerprint_md5(self):
+ https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+ cert_reqs='CERT_REQUIRED')
+
+ https_pool.ca_certs = DEFAULT_CA
+ https_pool.assert_fingerprint = 'CA:84:E1:AD0E5a:ef:2f:C3:09' \
+ ':E7:30:F8:CD:C8:5B'
+ https_pool.request('GET', '/')
+
+ def test_assert_fingerprint_sha1(self):
+ https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+ cert_reqs='CERT_REQUIRED')
+
+ https_pool.ca_certs = DEFAULT_CA
+ https_pool.assert_fingerprint = 'CC:45:6A:90:82:F7FF:C0:8218:8e:' \
+ '7A:F2:8A:D7:1E:07:33:67:DE'
+ https_pool.request('GET', '/')
+
+ def test_assert_invalid_fingerprint(self):
+ https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+ cert_reqs='CERT_REQUIRED')
+
+ https_pool.ca_certs = DEFAULT_CA
+ https_pool.assert_fingerprint = 'AA:AA:AA:AA:AA:AAAA:AA:AAAA:AA:' \
+ 'AA:AA:AA:AA:AA:AA:AA:AA:AA'
+
+ self.assertRaises(SSLError,
+ https_pool.request, 'GET', '/')
+
+ # invalid length
+ https_pool.assert_fingerprint = 'AA'
+
+ self.assertRaises(SSLError,
+ https_pool.request, 'GET', '/')
+
+ # uneven length
+ https_pool.assert_fingerprint = 'AA:A'
+
+ self.assertRaises(SSLError,
+ https_pool.request, 'GET', '/')
+
if __name__ == '__main__':
unittest.main()
38 urllib3/connectionpool.py
View
@@ -9,7 +9,7 @@
import errno
from socket import error as SocketError, timeout as SocketTimeout
-from .util import resolve_cert_reqs, resolve_ssl_version
+from .util import resolve_cert_reqs, resolve_ssl_version, assert_fingerprint
try: # Python 3
from http.client import HTTPConnection, HTTPException
@@ -81,12 +81,15 @@ class VerifiedHTTPSConnection(HTTPSConnection):
ssl_version = None
def set_cert(self, key_file=None, cert_file=None,
- cert_reqs=None, ca_certs=None):
+ cert_reqs=None, ca_certs=None,
+ assert_hostname=None, assert_fingerprint=None):
self.key_file = key_file
self.cert_file = cert_file
self.cert_reqs = cert_reqs
self.ca_certs = ca_certs
+ self.assert_hostname = assert_hostname
+ self.assert_fingerprint = assert_fingerprint
def connect(self):
# Add certificate verification
@@ -104,8 +107,12 @@ def connect(self):
ssl_version=resolved_ssl_version)
if resolved_cert_reqs != ssl.CERT_NONE:
- match_hostname(self.sock.getpeercert(), self.host)
-
+ if self.assert_fingerprint:
+ assert_fingerprint(self.sock.getpeercert(binary_form=True),
+ self.assert_fingerprint)
+ else:
+ match_hostname(self.sock.getpeercert(),
+ self.assert_hostname or self.host)
## Pool objects
@@ -502,9 +509,13 @@ class HTTPSConnectionPool(HTTPConnectionPool):
:class:`.VerifiedHTTPSConnection` is used, which *can* verify certificates,
instead of :class:`httplib.HTTPSConnection`.
- The ``key_file``, ``cert_file``, ``cert_reqs``, ``ca_certs``, and ``ssl_version``
- are only used if :mod:`ssl` is available and are fed into
- :meth:`urllib3.util.ssl_wrap_socket` to upgrade the connection socket into an SSL socket.
+ :class:`.VerifiedHTTPSConnection` uses one of ``assert_fingerprint``,
+ ``assert_hostname`` and ``host`` in this order to verify connections.
+
+ The ``key_file``, ``cert_file``, ``cert_reqs``, ``ca_certs`` and
+ ``ssl_version`` are only used if :mod:`ssl` is available and are fed into
+ :meth:`urllib3.util.ssl_wrap_socket` to upgrade the connection socket
+ into an SSL socket.
"""
scheme = 'https'
@@ -512,8 +523,9 @@ class HTTPSConnectionPool(HTTPConnectionPool):
def __init__(self, host, port=None,
strict=False, timeout=None, maxsize=1,
block=False, headers=None,
- key_file=None, cert_file=None,
- cert_reqs=None, ca_certs=None, ssl_version=None):
+ key_file=None, cert_file=None, cert_reqs=None,
+ ca_certs=None, ssl_version=None,
+ assert_hostname=None, assert_fingerprint=None):
HTTPConnectionPool.__init__(self, host, port,
strict, timeout, maxsize,
@@ -523,6 +535,8 @@ def __init__(self, host, port=None,
self.cert_reqs = cert_reqs
self.ca_certs = ca_certs
self.ssl_version = ssl_version
+ self.assert_hostname = assert_hostname
+ self.assert_fingerprint = assert_fingerprint
def _new_conn(self):
"""
@@ -532,7 +546,7 @@ def _new_conn(self):
log.info("Starting new HTTPS connection (%d): %s"
% (self.num_connections, self.host))
- if not ssl: # Platform-specific: Python compiled without +ssl
+ if not ssl: # Platform-specific: Python compiled without +ssl
if not HTTPSConnection or HTTPSConnection is object:
raise SSLError("Can't connect to HTTPS URL because the SSL "
"module is not available.")
@@ -545,7 +559,9 @@ def _new_conn(self):
port=self.port,
strict=self.strict)
connection.set_cert(key_file=self.key_file, cert_file=self.cert_file,
- cert_reqs=self.cert_reqs, ca_certs=self.ca_certs)
+ cert_reqs=self.cert_reqs, ca_certs=self.ca_certs,
+ assert_hostname=self.assert_hostname,
+ assert_fingerprint=self.assert_fingerprint)
connection.ssl_version = self.ssl_version
44 urllib3/util.py
View
@@ -8,6 +8,8 @@
from base64 import b64encode
from collections import namedtuple
from socket import error as SocketError
+from hashlib import md5, sha1
+from binascii import hexlify, unhexlify
try:
from select import poll, POLLIN
@@ -23,7 +25,7 @@
HAS_SNI = False
import ssl
- from ssl import wrap_socket, CERT_NONE, SSLError, PROTOCOL_SSLv23
+ from ssl import wrap_socket, CERT_NONE, PROTOCOL_SSLv23
from ssl import SSLContext # Modern SSL?
from ssl import HAS_SNI # Has SNI?
except ImportError:
@@ -31,7 +33,7 @@
from .packages import six
-from .exceptions import LocationParseError
+from .exceptions import LocationParseError, SSLError
class Url(namedtuple('Url', ['scheme', 'auth', 'host', 'port', 'path', 'query', 'fragment'])):
@@ -302,6 +304,44 @@ def resolve_ssl_version(candidate):
return candidate
+
+def assert_fingerprint(cert, fingerprint):
+ """
+ Checks if given fingerprint matches the supplied certificate.
+
+ :param cert:
+ Certificate as bytes object.
+ :param fingerprint:
+ Fingerprint as string of hexdigits, can be interspersed by colons.
+ """
+
+ # Maps the length of a digest to a possible hash function producing
+ # this digest.
Andrey Petrov Owner
shazow added a note

Periods at the end of sentence comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ hashfunc_map = {
+ 16: md5,
+ 20: sha1
+ }
+
+ fingerprint = fingerprint.replace(':', '').lower()
+
+ digest_length, rest = divmod(len(fingerprint), 2)
+
+ if rest or digest_length not in hashfunc_map:
+ raise SSLError('Fingerprint is of invalid length.')
+
+ # We need encode() here for py32; works on py2 and p33.
+ fingerprint_bytes = unhexlify(fingerprint.encode())
+
+ hashfunc = hashfunc_map[digest_length]
+
+ cert_digest = hashfunc(cert).digest()
+
+ if not cert_digest == fingerprint_bytes:
+ raise SSLError('Fingerprints did not match. Expected "{0}", got "{1}".'
+ .format(hexlify(fingerprint_bytes),
+ hexlify(cert_digest)))
+
+
if SSLContext is not None: # Python 3.2+
def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
ca_certs=None, server_hostname=None,
Something went wrong with that request. Please try again.