Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Manual specification of hostname to verify against #140

Merged
merged 9 commits into from

3 participants

@t-8ch

This allows people to manually specify the hostname they want to verify their ssl certs against.
This should help if SNI insn't available, the server returns the wrong cert or one connects directly to IP addresses.

(Reference: kennethreitz/requests#1124, kennethreitz/requests#749)
This could be enhanced to verify against fingerprints.

urllib3/util.py
@@ -24,6 +26,7 @@
import ssl
from ssl import wrap_socket, CERT_NONE, SSLError, PROTOCOL_SSLv23
+ from .exceptions import SSLError
@shazow Owner
shazow added a note

Add this to the .exceptions import below.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
@@ -302,6 +305,43 @@ def resolve_ssl_version(candidate):
return candidate
+
+def match_fingerprint(remote, local):
+ """
+ Compares if both supplied fingerprints match.
+
+ remote -- binary
@shazow Owner
shazow added a note

Can we keep this consistent with urllib3's style?

:param remote: <Description>

etc

@shazow Owner
shazow added a note

Not completely clear what "binary" means.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
@@ -302,6 +305,43 @@ def resolve_ssl_version(candidate):
return candidate
+
+def match_fingerprint(remote, local):
+ """
+ Compares if both supplied fingerprints match.
+
+ remote -- binary
+ local -- hexstring, can be separated by colons
+ """
+
+ # maps the raw byte length of a digest to its hash function
@shazow Owner
shazow added a note

Sentence-case comments please (capitalize first letter, period if it's a sentence).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
@@ -302,6 +305,43 @@ def resolve_ssl_version(candidate):
return candidate
+
+def match_fingerprint(remote, local):
@shazow Owner
shazow added a note

Since this function doesn't return anything but just raises errors, perhaps we should call it something like assert_fingerprint?

@t-8ch
t-8ch added a note

I named it after match_hostname.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/connectionpool.py
@@ -80,13 +80,15 @@ class VerifiedHTTPSConnection(HTTPSConnection):
ca_certs = None
ssl_version = None
- def set_cert(self, key_file=None, cert_file=None,
- cert_reqs=None, ca_certs=None):
+ def set_cert(self, key_file=None, cert_file=None, cert_reqs=None,
+ ca_certs=None, verify_hostname=None, verify_fingerprint=None):
@shazow Owner
shazow added a note

PEP8 indent on (

Ideally, group similar params (e.g. cert_, verify_) per line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/connectionpool.py
((8 lines not shown))
self.key_file = key_file
self.cert_file = cert_file
self.cert_reqs = cert_reqs
self.ca_certs = ca_certs
+ self.verify_hostname = verify_hostname
@shazow Owner
shazow added a note

Maybe call these s/verify/assert/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
CONTRIBUTORS.txt
@@ -46,7 +46,11 @@ In chronological order:
* Support for explicitly closing pooled connections
* hartator <hartator@gmail.com>
- * Corrected multipart behavior for params
+ * Corrected multipart behavior for params
+
+* Thomas Weißschuh <thomas@t-8ch.de>
+ * various SSL patches
@shazow Owner
shazow added a note

Could you change this to "Various SSL patches, with tests"?

Or better yet, "SSL fingerprint checking, with tests"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
@@ -302,6 +305,43 @@ def resolve_ssl_version(candidate):
return candidate
+
+def match_fingerprint(remote, local):
+ """
+ Compares if both supplied fingerprints match.
@shazow Owner
shazow added a note

Could you give a better description of what this method is for and when it should be used?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@t-8ch

@shazow This doesn't really work for Pools to multiple hosts. Are those "supported" and worth effort?

@shazow
Owner

PoolManager? Absolutely, it's the most-used use case for urllib3 (requests depends on it, for example).

@shazow
Owner

Or do you mean multiple hosts in a single HTTP(S)ConnectionPool object? That should not happen.

@t-8ch

I am thinking about assert_same_host in in urlopen()
If this is false a connection can be used with a different host than constructed.
Am I getting this wrong?

@shazow
Owner

Oh, by host you mean literally the Host: header? Yes. But the IPs must match.

@t-8ch

Then the current behaviour of of this PR should be sufficient.
I hope the style is acceptably now, that was really not my day -.-

@t-8ch

@shazow If you are happy with this I'll add the docs.

urllib3/util.py
((22 lines not shown))
+ fingerprint = fingerprint.replace(':', '').lower()
+
+ digest_length, rest = divmod(len(fingerprint), 2)
+
+ if rest or digest_length not in hashfunc_map:
+ raise SSLError('Fingerprint is of invalid length')
+
+ # We need encode() here for py32, works on py2 and p33
+ fingerprint_bytes = unhexlify(fingerprint.encode())
+
+ hashfunc = hashfunc_map[digest_length]
+
+ cert_digest = hashfunc(cert).digest()
+
+ if not cert_digest == fingerprint_bytes:
+ raise SSLError('Fingerprints did not match!\n'
@shazow Owner
shazow added a note

We don't have multiline exception messages anywhere. Perhaps make this consistent?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
urllib3/util.py
((12 lines not shown))
+ Fingerprint as string of hexdigits, can be interspersed by colons.
+ """
+
+ # Maps the length of a digest to a possible hash function producing
+ # this digest
+ hashfunc_map = {
+ 16: md5,
+ 20: sha1
+ }
+
+ fingerprint = fingerprint.replace(':', '').lower()
+
+ digest_length, rest = divmod(len(fingerprint), 2)
+
+ if rest or digest_length not in hashfunc_map:
+ raise SSLError('Fingerprint is of invalid length')
@shazow Owner
shazow added a note

I believe we put periods at the end of message sentences.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shazow shazow commented on the diff
urllib3/util.py
@@ -302,6 +304,45 @@ def resolve_ssl_version(candidate):
return candidate
+
+def assert_fingerprint(cert, fingerprint):
+ """
+ Checks if given fingerprint matches the supplied certificate.
+
+ :param cert:
+ Certificate as bytes object.
+ :param fingerprint:
+ Fingerprint as string of hexdigits, can be interspersed by colons.
+ """
+
+ # Maps the length of a digest to a possible hash function producing
+ # this digest
@shazow Owner
shazow added a note

Periods at the end of sentence comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@shazow
Owner

Looks good to me. :)

Docs appreciated.

@t-8ch

Here we go. Any opinions to my other PRs?

@shazow
Owner

Sorry I haven't had much time to do OSS stuff these last couple of weeks. Next week should be way better though. :)

@t-8ch

@shazow While you are at it: 'ping' :sunglasses:

@shazow
Owner

Thank you. :)

Reviewed the latest version of the changes, looks good! I'm happy to merge it next cycle (probably next week).

I wonder if all of the assert_* configuration is getting out of hand and if we should think about a different place or API for those kinds of things. Happy to leave that off for the future though.

@t-8ch

What about a configuration class Verification, like Retries and Timeout?

@shazow
Owner

Hmm might be getting out of hand, or maybe not. Out of curiosity, what do you imagine such a class would look like? :)

@t-8ch

This was just a reference to those other config objects you mentioned on another issue.
I had a use case like this in mind:

pool = HTTPSConnectionpool(host, port,
           verify=Verification(fingerprint='abc', key_file='/foo', cert_file='/bar')

Like some sort of algebraic datatype:

verify = None
verify = Verification(hostname='foo')
verify = Verification(callback=some_func)

My motivation for a fine grained (bloated) API like this is the elimination of all excuses for not verifying certificates in a reasonable manner.

@piotr-dobrogost

I like the idea of grouping parameters pertaining to some aspect of configuration.

@shazow
Owner

Hmm interesting indeed. Then the verification logic can live in the Verification object, which would leave HTTPSConnectionPool much cleaner. Will need to think about what objects the Verification logic would need access to. Looks like just the VerifiedHTTPSConnection should do the trick.

@t-8ch

ping @shazow
Rebased to trigger travis. Added two more tests to keep coverage at 100%.

@shazow shazow merged commit 7f9d8ee into shazow:master

1 check passed

Details default The Travis build passed
@shazow
Owner

Fantastic. @t-8ch, thank you so much for following through with this. :)

@shazow
Owner

Hmm. How would you describe this for the CHANGES.rst?

@t-8ch

"Added mechanism to verify ssl certificates by fingerprint (md5, sha1) or against an arbitrary hostname (when connecting by IP or for misconfigured servers)"

@shazow
Owner

+1

@shazow
Owner

Cool. I think this is a pretty sizeable chunk for v1.6. Maybe I should version bump soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
8 CONTRIBUTORS.txt
@@ -46,7 +46,13 @@ In chronological order:
* Support for explicitly closing pooled connections
* hartator <hartator@gmail.com>
- * Corrected multipart behavior for params
+ * Corrected multipart behavior for params
+
+* Thomas Weißschuh <thomas@t-8ch.de>
+ * Support for TLS SNI
+ * API unification of ssl_version/cert_reqs
+ * SSL fingerprint and alternative hostname verification
+ * Bugfixes in testsuite
* Sune Kirkeby <mig@ibofobi.dk>
* Optional SNI-support for Python 2 via PyOpenSSL.
View
49 test/with_dummyserver/test_https.py
@@ -137,6 +137,55 @@ def test_set_ssl_version_to_sslv3(self):
self._pool.ssl_version = ssl.PROTOCOL_SSLv3
self.assertRaises(SSLError, self._pool.request, 'GET', '/')
+ def test_assert_specific_hostname(self):
+ https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+ cert_reqs='CERT_REQUIRED')
+
+ https_pool.ca_certs = DEFAULT_CA
+ https_pool.assert_hostname = 'localhost'
+ https_pool.request('GET', '/')
+
+ def test_assert_fingerprint_md5(self):
+ https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+ cert_reqs='CERT_REQUIRED')
+
+ https_pool.ca_certs = DEFAULT_CA
+ https_pool.assert_fingerprint = 'CA:84:E1:AD0E5a:ef:2f:C3:09' \
+ ':E7:30:F8:CD:C8:5B'
+ https_pool.request('GET', '/')
+
+ def test_assert_fingerprint_sha1(self):
+ https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+ cert_reqs='CERT_REQUIRED')
+
+ https_pool.ca_certs = DEFAULT_CA
+ https_pool.assert_fingerprint = 'CC:45:6A:90:82:F7FF:C0:8218:8e:' \
+ '7A:F2:8A:D7:1E:07:33:67:DE'
+ https_pool.request('GET', '/')
+
+ def test_assert_invalid_fingerprint(self):
+ https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+ cert_reqs='CERT_REQUIRED')
+
+ https_pool.ca_certs = DEFAULT_CA
+ https_pool.assert_fingerprint = 'AA:AA:AA:AA:AA:AAAA:AA:AAAA:AA:' \
+ 'AA:AA:AA:AA:AA:AA:AA:AA:AA'
+
+ self.assertRaises(SSLError,
+ https_pool.request, 'GET', '/')
+
+ # invalid length
+ https_pool.assert_fingerprint = 'AA'
+
+ self.assertRaises(SSLError,
+ https_pool.request, 'GET', '/')
+
+ # uneven length
+ https_pool.assert_fingerprint = 'AA:A'
+
+ self.assertRaises(SSLError,
+ https_pool.request, 'GET', '/')
+
if __name__ == '__main__':
unittest.main()
View
38 urllib3/connectionpool.py
@@ -9,7 +9,7 @@
import errno
from socket import error as SocketError, timeout as SocketTimeout
-from .util import resolve_cert_reqs, resolve_ssl_version
+from .util import resolve_cert_reqs, resolve_ssl_version, assert_fingerprint
try: # Python 3
from http.client import HTTPConnection, HTTPException
@@ -81,12 +81,15 @@ class VerifiedHTTPSConnection(HTTPSConnection):
ssl_version = None
def set_cert(self, key_file=None, cert_file=None,
- cert_reqs=None, ca_certs=None):
+ cert_reqs=None, ca_certs=None,
+ assert_hostname=None, assert_fingerprint=None):
self.key_file = key_file
self.cert_file = cert_file
self.cert_reqs = cert_reqs
self.ca_certs = ca_certs
+ self.assert_hostname = assert_hostname
+ self.assert_fingerprint = assert_fingerprint
def connect(self):
# Add certificate verification
@@ -104,8 +107,12 @@ def connect(self):
ssl_version=resolved_ssl_version)
if resolved_cert_reqs != ssl.CERT_NONE:
- match_hostname(self.sock.getpeercert(), self.host)
-
+ if self.assert_fingerprint:
+ assert_fingerprint(self.sock.getpeercert(binary_form=True),
+ self.assert_fingerprint)
+ else:
+ match_hostname(self.sock.getpeercert(),
+ self.assert_hostname or self.host)
## Pool objects
@@ -502,9 +509,13 @@ class HTTPSConnectionPool(HTTPConnectionPool):
:class:`.VerifiedHTTPSConnection` is used, which *can* verify certificates,
instead of :class:`httplib.HTTPSConnection`.
- The ``key_file``, ``cert_file``, ``cert_reqs``, ``ca_certs``, and ``ssl_version``
- are only used if :mod:`ssl` is available and are fed into
- :meth:`urllib3.util.ssl_wrap_socket` to upgrade the connection socket into an SSL socket.
+ :class:`.VerifiedHTTPSConnection` uses one of ``assert_fingerprint``,
+ ``assert_hostname`` and ``host`` in this order to verify connections.
+
+ The ``key_file``, ``cert_file``, ``cert_reqs``, ``ca_certs`` and
+ ``ssl_version`` are only used if :mod:`ssl` is available and are fed into
+ :meth:`urllib3.util.ssl_wrap_socket` to upgrade the connection socket
+ into an SSL socket.
"""
scheme = 'https'
@@ -512,8 +523,9 @@ class HTTPSConnectionPool(HTTPConnectionPool):
def __init__(self, host, port=None,
strict=False, timeout=None, maxsize=1,
block=False, headers=None,
- key_file=None, cert_file=None,
- cert_reqs=None, ca_certs=None, ssl_version=None):
+ key_file=None, cert_file=None, cert_reqs=None,
+ ca_certs=None, ssl_version=None,
+ assert_hostname=None, assert_fingerprint=None):
HTTPConnectionPool.__init__(self, host, port,
strict, timeout, maxsize,
@@ -523,6 +535,8 @@ def __init__(self, host, port=None,
self.cert_reqs = cert_reqs
self.ca_certs = ca_certs
self.ssl_version = ssl_version
+ self.assert_hostname = assert_hostname
+ self.assert_fingerprint = assert_fingerprint
def _new_conn(self):
"""
@@ -532,7 +546,7 @@ def _new_conn(self):
log.info("Starting new HTTPS connection (%d): %s"
% (self.num_connections, self.host))
- if not ssl: # Platform-specific: Python compiled without +ssl
+ if not ssl: # Platform-specific: Python compiled without +ssl
if not HTTPSConnection or HTTPSConnection is object:
raise SSLError("Can't connect to HTTPS URL because the SSL "
"module is not available.")
@@ -545,7 +559,9 @@ def _new_conn(self):
port=self.port,
strict=self.strict)
connection.set_cert(key_file=self.key_file, cert_file=self.cert_file,
- cert_reqs=self.cert_reqs, ca_certs=self.ca_certs)
+ cert_reqs=self.cert_reqs, ca_certs=self.ca_certs,
+ assert_hostname=self.assert_hostname,
+ assert_fingerprint=self.assert_fingerprint)
connection.ssl_version = self.ssl_version
View
44 urllib3/util.py
@@ -8,6 +8,8 @@
from base64 import b64encode
from collections import namedtuple
from socket import error as SocketError
+from hashlib import md5, sha1
+from binascii import hexlify, unhexlify
try:
from select import poll, POLLIN
@@ -23,7 +25,7 @@
HAS_SNI = False
import ssl
- from ssl import wrap_socket, CERT_NONE, SSLError, PROTOCOL_SSLv23
+ from ssl import wrap_socket, CERT_NONE, PROTOCOL_SSLv23
from ssl import SSLContext # Modern SSL?
from ssl import HAS_SNI # Has SNI?
except ImportError:
@@ -31,7 +33,7 @@
from .packages import six
-from .exceptions import LocationParseError
+from .exceptions import LocationParseError, SSLError
class Url(namedtuple('Url', ['scheme', 'auth', 'host', 'port', 'path', 'query', 'fragment'])):
@@ -302,6 +304,44 @@ def resolve_ssl_version(candidate):
return candidate
+
+def assert_fingerprint(cert, fingerprint):
+ """
+ Checks if given fingerprint matches the supplied certificate.
+
+ :param cert:
+ Certificate as bytes object.
+ :param fingerprint:
+ Fingerprint as string of hexdigits, can be interspersed by colons.
+ """
+
+ # Maps the length of a digest to a possible hash function producing
+ # this digest.
@shazow Owner
shazow added a note

Periods at the end of sentence comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ hashfunc_map = {
+ 16: md5,
+ 20: sha1
+ }
+
+ fingerprint = fingerprint.replace(':', '').lower()
+
+ digest_length, rest = divmod(len(fingerprint), 2)
+
+ if rest or digest_length not in hashfunc_map:
+ raise SSLError('Fingerprint is of invalid length.')
+
+ # We need encode() here for py32; works on py2 and p33.
+ fingerprint_bytes = unhexlify(fingerprint.encode())
+
+ hashfunc = hashfunc_map[digest_length]
+
+ cert_digest = hashfunc(cert).digest()
+
+ if not cert_digest == fingerprint_bytes:
+ raise SSLError('Fingerprints did not match. Expected "{0}", got "{1}".'
+ .format(hexlify(fingerprint_bytes),
+ hexlify(cert_digest)))
+
+
if SSLContext is not None: # Python 3.2+
def ssl_wrap_socket(sock, keyfile=None, certfile=None, cert_reqs=None,
ca_certs=None, server_hostname=None,
Something went wrong with that request. Please try again.