bpo-31399: Let OpenSSL verify hostname and IP address #3462

tiran · 2017-09-08T21:06:36Z

Replace Python's custom ssl.match_hostname() with OpenSSL 1.0.2 APIs. The hostname or IP address are now verified by OpenSSL during the TLS handshake. A failure to verify the name results in a proper handshake abort with TLS Alert bad cert.

Signed-off-by: Christian Heimes christian@python.org

https://bugs.python.org/issue31399

alex

Mostly looks good, from a first skim

alex · 2018-01-11T21:21:28Z

Modules/_ssl.c

@@ -310,6 +310,10 @@ typedef struct {
    PyObject *set_hostname;
 #endif
    int check_hostname;
+    /* OpenSSL does not provide X509_VERIFY_PARAM_get_hostflags(), hostflags
+     * are initialized as 0.


I don't understand this comment

OpenSSL has no API to get hostflags from X509_VERIFY_PARAM. We have to maintain our own copy. I'll clarify the comment in my next push.

Ahhh. Now I understand. I think "OpenSSL does not have an API for getting the hostflags from an X509_VERIFY_PARAM" would be clearer.

Also is there a bug/PR on OpenSSL for this we can link?

alex · 2018-01-11T21:24:15Z

Modules/_ssl.c

 #endif
+    self->hostflags = X509_CHECK_FLAG_NO_PARTIAL_WILDCARDS;


Do we need X509_CHECK_FLAG_SINGLE_LABEL_SUBDOMAINS as well?

tiran · 2018-01-11T22:03:45Z

Thanks @alex

I have updated the comment for hostflaggs and created openssl/openssl#5061.

tiran · 2018-01-16T15:42:36Z

Blocked by #5180

@asvetlov @1st1 The PR touches asyncio. Please review the test changes.

asvetlov

asyncio test change is correct (but please update assertion as suggested)

asvetlov · 2018-01-16T17:05:28Z

Lib/test/test_asyncio/test_events.py

                    self.loop.run_until_complete(f_c)

        # close connection
-        proto.transport.close()
+        # transport is None because TLS ALERT aborted the handshake
+        self.assertEqual(proto.transport, None)


Please use self.assertIsNone(proto.transport)

tiran · 2018-01-16T21:10:27Z

@asvetlov Thanks, I changed the assert as requested

I also forgot to remove the call to match_hostname. How should I handle the call in asyncio? If I remove it, the code will no worker be secure on 3.6.

1st1 · 2018-01-16T21:19:11Z

I also forgot to remove the call to match_hostname. How should I handle the call in asyncio? If I remove it, the code will no worker be secure on 3.6.

This change is for 3.7 only, right? If so just remove the call in the master branch. We no longer try to use asyncio CPython code with other CPython versions.

tiran · 2018-01-17T10:55:14Z

Yes, it's 3.7 only.

tiran · 2018-01-20T09:44:13Z

This branch is now based on PR #5242. Please review and ACK the other PR first.

asvetlov · 2018-01-20T16:56:24Z

Are there stoppers for the issue left?

tiran · 2018-01-20T17:08:44Z

@asvetlov No stoppers, I'm just waiting for a final review. The PR has changed a bit since @alex reviewed it. I have added more documentation and additional checks for LibreSSL.

asvetlov · 2018-01-20T17:14:52Z

Good to know.
Just for note -- LGTM.

alex

Code looks fine, I have some API design concerns though.

alex · 2018-01-22T23:19:39Z

Doc/library/ssl.rst

@@ -430,9 +431,14 @@ Certificate handling
      of the certificate, is now supported.

   .. versionchanged:: 3.7
+      The function is no longer used. Hostname matching is now performed


"no longer used by TLS connections" or something?

alex · 2018-01-22T23:20:40Z

Doc/library/ssl.rst

@@ -586,6 +592,42 @@ Constants
   be passed, either to :meth:`SSLContext.load_verify_locations` or as a
   value of the ``ca_certs`` parameter to :func:`wrap_socket`.

+.. class:: HostFlags


Do all of these need to be a public API?

I don't see a need to support things like partial wildcards or multi-label wildcards.

The previous implementation didn't support them, and I don't think expanding the public API like this is a good idea.

Partial wildcard matching is disabled by default. I have to expose the flag value so the getter can report the flag correctly:

>>> import ssl >>> ssl.create_default_context().host_flags <HostFlags.HOSTFLAG_NO_PARTIAL_WILDCARDS: 4>

I also like to add NEVER_CHECK_SUBJECT for urllib3, too.

I don't care about the other flags. I'm happy to remove them.

alex · 2018-01-22T23:21:31Z

Doc/library/ssl.rst

@@ -1730,6 +1778,14 @@ to speed up repeated connections from the same clients.
   The protocol version chosen when constructing the context.  This attribute
   is read-only.

+.. attribute:: SSLContext.host_flags


Does this need to be a public API? I feel like we can set reasonable defaults that are applicable to everything.

I exposed the flag primary for urllib3/urllib3#497. urllib3 wants to disable CN support and require certificates with SAN fields.

What if we just expose a single check_subject_hostname or something like that, instead of all these flags?

I'm ok with that, but I'm not 100% happy with the name.

check_subject_cn

check_common_name

check_subject_common_name

use_common_name

check_only_san

...

I'm going to rename host_flags to _host_flags, keep all flags in _ssl and implement the flag in pure Python. That way we don't expose the API, but still have it available to implement other flags.

Sounds good -- my desire is for us not to commit to a really broad API when we really want a tiny thing.

hostname_checks_common_name perhaps?

That's explicit, almost German. I like it :)

The feature is only available with OpenSSL 1.1.0+. How should I handle OpenSSL 1.0.2? Don't define the property at all or only implement readonly property + ssl.HAS_NEVER_CHECK_HOSTNAME?

readonly + a constant makes sense to me

Done

I'm currently at a conference and will take care of proper documentation next week.

alex · 2018-01-22T23:22:46Z

Lib/asyncio/sslproto.py

-                if (self._server_hostname and
-                        self._sslcontext.verify_mode != ssl.CERT_NONE):
-                    ssl.match_hostname(peercert, self._server_hostname)
+            # Since 3.7, the hostname is matched by OpenSSL during handshake.


Not sure this comment is needed here, since it's not acually referring to any code which exists

The ssl module now uses OpenSSL's X509_VERIFY_PARAM_set1_host() and X509_VERIFY_PARAM_set1_ip() API to verify hostname and IP addresses. Signed-off-by: Christian Heimes <christian@python.org>

Signed-off-by: Christian Heimes <christian@python.org>

libssl must provide X509_VERIFY_PARAM_set1_host() Signed-off-by: Christian Heimes <christian@python.org>

Signed-off-by: Christian Heimes <christian@python.org>

Remove all hostflags except for NO_PARTIAL_WILDCARDS and NEVER_CHECK_SUBJECT. The other flags aren't that useful at the moment. Don't support OpenSSL special mode with a leading dot, e.g. ".example.org" matches "www.example.org". It's not standard conform. Signed-off-by: Christian Heimes <christian@python.org>

Host flags are now in internal API. Public API is a new attribute hostname_checks_common_name. Signed-off-by: Christian Heimes <christian@python.org>

bedevere-bot · 2018-01-27T14:51:41Z

@tiran: Please replace # with GH- in the commit message next time. Thanks!

asottile · 2018-04-27T15:20:47Z

Running into this patch on travis-ci as ubuntu trusty ships with openssl 1.0.1 -- this patch causes python3.7 to build without _ssl support on trusty. See also:

I agree that the patch is the right move going forward 👍, figured I'd post this here for visibility though.

the-knights-who-say-ni added the CLA signed label Sep 8, 2017

bedevere-bot added the awaiting merge label Sep 8, 2017

Mariatta added needs rebase and removed needs rebase labels Oct 9, 2017

tiran force-pushed the openssl_check_hostname branch from a3fa2cf to 75d33a3 Compare January 11, 2018 20:51

tiran requested a review from a team as a code owner January 11, 2018 20:51

tiran force-pushed the openssl_check_hostname branch from 75d33a3 to 35176ba Compare January 11, 2018 21:14

tiran requested review from 1st1 and asvetlov as code owners January 11, 2018 21:14

tiran force-pushed the openssl_check_hostname branch from 35176ba to 65faab4 Compare January 11, 2018 21:18

alex reviewed Jan 11, 2018

View reviewed changes

tiran mentioned this pull request Jan 11, 2018

Feature: Remove support for common names in certificates urllib3/urllib3#497

Closed

tiran force-pushed the openssl_check_hostname branch from 65faab4 to c9f4c6f Compare January 11, 2018 22:02

tiran force-pushed the openssl_check_hostname branch 6 times, most recently from e499eca to 89a28a5 Compare January 12, 2018 14:54

tiran changed the title ~~bpo-31399: [WIP] Let OpenSSL verify hostname and IP address~~ bpo-31399: Let OpenSSL verify hostname and IP address Jan 16, 2018

asvetlov approved these changes Jan 16, 2018

View reviewed changes

1st1 approved these changes Jan 16, 2018

View reviewed changes

tiran force-pushed the openssl_check_hostname branch 2 times, most recently from c400837 to d1dd4f3 Compare January 16, 2018 21:09

tiran force-pushed the openssl_check_hostname branch from d1dd4f3 to d07134d Compare January 17, 2018 10:20

tiran force-pushed the openssl_check_hostname branch from d07134d to 7b5edd5 Compare January 20, 2018 09:43

tiran requested a review from a team as a code owner January 20, 2018 09:43

tiran force-pushed the openssl_check_hostname branch 3 times, most recently from 7c4fc70 to 197d51f Compare January 20, 2018 12:25

tiran mentioned this pull request Jan 20, 2018

bpo-31429: Define TLS cipher suite on build time #3532

Merged

tiran mentioned this pull request Jan 20, 2018

bpo-32185: Don't send IP in SNI TLS extension #4938

Closed

alex reviewed Jan 22, 2018

View reviewed changes

tiran force-pushed the openssl_check_hostname branch 2 times, most recently from 2199b99 to e819fbd Compare January 23, 2018 11:17

tiran added 7 commits January 26, 2018 15:27

bpo-31399: Let OpenSSL verify hostname and IP

f289cab

The ssl module now uses OpenSSL's X509_VERIFY_PARAM_set1_host() and X509_VERIFY_PARAM_set1_ip() API to verify hostname and IP addresses. Signed-off-by: Christian Heimes <christian@python.org>

Remove match_hostname calls

9dbf5f7

Signed-off-by: Christian Heimes <christian@python.org>

Document host flags

802807d

Signed-off-by: Christian Heimes <christian@python.org>

Check for libssl with set1_host

bb1f7aa

libssl must provide X509_VERIFY_PARAM_set1_host() Signed-off-by: Christian Heimes <christian@python.org>

bpo-31399: Add documentation for OpenSSL 1.0.2 requirement

a48ae16

Signed-off-by: Christian Heimes <christian@python.org>

Replace SSLContext.host_flags

845c149

Host flags are now in internal API. Public API is a new attribute hostname_checks_common_name. Signed-off-by: Christian Heimes <christian@python.org>

tiran force-pushed the openssl_check_hostname branch from e819fbd to 845c149 Compare January 26, 2018 15:35

alex approved these changes Jan 27, 2018

View reviewed changes

tiran merged commit 61d478c into python:master Jan 27, 2018

bedevere-bot removed the awaiting merge label Jan 27, 2018

tiran deleted the openssl_check_hostname branch January 27, 2018 14:51

awarecan mentioned this pull request Feb 26, 2019

TheThings Network Component throws SSL errors after update home-assistant/core#21367

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-31399: Let OpenSSL verify hostname and IP address #3462

bpo-31399: Let OpenSSL verify hostname and IP address #3462

tiran commented Sep 8, 2017 •

edited

Loading

alex left a comment

alex Jan 11, 2018

tiran Jan 11, 2018 •

edited

Loading

alex Jan 11, 2018

alex Jan 11, 2018

tiran commented Jan 11, 2018

tiran commented Jan 16, 2018

asvetlov left a comment

asvetlov Jan 16, 2018

tiran commented Jan 16, 2018

1st1 commented Jan 16, 2018

tiran commented Jan 17, 2018

tiran commented Jan 20, 2018

asvetlov commented Jan 20, 2018

tiran commented Jan 20, 2018

asvetlov commented Jan 20, 2018 •

edited

Loading

alex left a comment

alex Jan 22, 2018

alex Jan 22, 2018

tiran Jan 23, 2018

alex Jan 22, 2018

tiran Jan 23, 2018

alex Jan 26, 2018

tiran Jan 26, 2018

alex Jan 26, 2018

tiran Jan 26, 2018

alex Jan 26, 2018

tiran Jan 26, 2018

alex Jan 22, 2018

bedevere-bot commented Jan 27, 2018

asottile commented Apr 27, 2018

		#endif
		self->hostflags = X509_CHECK_FLAG_NO_PARTIAL_WILDCARDS;

bpo-31399: Let OpenSSL verify hostname and IP address #3462

bpo-31399: Let OpenSSL verify hostname and IP address #3462

Conversation

tiran commented Sep 8, 2017 • edited Loading

alex left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiran Jan 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiran commented Jan 11, 2018

tiran commented Jan 16, 2018

asvetlov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiran commented Jan 16, 2018

1st1 commented Jan 16, 2018

tiran commented Jan 17, 2018

tiran commented Jan 20, 2018

asvetlov commented Jan 20, 2018

tiran commented Jan 20, 2018

asvetlov commented Jan 20, 2018 • edited Loading

alex left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bedevere-bot commented Jan 27, 2018

asottile commented Apr 27, 2018

tiran commented Sep 8, 2017 •

edited

Loading

tiran Jan 11, 2018 •

edited

Loading

asvetlov commented Jan 20, 2018 •

edited

Loading