New behavior of OpenSSL hostname verification not exposed, incorrectly documented

[MojoVampire]

BPO	36868
Nosy	@tiran, @ned-deily, @MojoVampire, @miss-islington
PRs	bpo-36868: Fix what's new for SSLContext.hostname_checks_common_name #13248 [3.7] bpo-36868: Fix what's new for SSLContext.hostname_checks_common_name (GH-13248) #13784

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-04-17.19:36:19.666>
created_at = <Date 2019-05-09.18:15:47.448>
labels = ['expert-SSL', 'type-bug', '3.8', 'extension-modules', '3.7', 'library', 'docs']
title = 'New behavior of OpenSSL hostname verification not exposed, incorrectly documented'
updated_at = <Date 2021-04-17.19:36:19.666>
user = 'https://github.com/MojoVampire'

bugs.python.org fields:

activity = <Date 2021-04-17.19:36:19.666>
actor = 'christian.heimes'
assignee = 'docs@python'
closed = True
closed_date = <Date 2021-04-17.19:36:19.666>
closer = 'christian.heimes'
components = ['Documentation', 'Extension Modules', 'Library (Lib)', 'SSL']
creation = <Date 2019-05-09.18:15:47.448>
creator = 'josh.r'
dependencies = []
files = []
hgrepos = []
issue_num = 36868
keywords = ['patch', '3.6regression']
message_count = 4.0
messages = ['341990', '342202', '344460', '344465']
nosy_count = 5.0
nosy_names = ['christian.heimes', 'ned.deily', 'docs@python', 'josh.r', 'miss-islington']
pr_nums = ['13248', '13784']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36868'
versions = ['Python 3.7', 'Python 3.8']

[MojoVampire]

The What's New in 3.7 docs mention the change from bpo-31399 to use OpenSSL's built-in hostname verification ( https://docs.python.org/3/whatsnew/3.7.html#ssl ), but aside from that, information about the change is largely undiscoverable and/or wrong.

Specific problems:

1. The What's New docs repeatedly mention SSLContext.host_flags as the means of modifying behavior. The actual property is underscore prefixed though, as SSLContext._host_flags. Since SSLContext supports the creation of arbitrary names via __dict__, assigning to context.host_flags silently "works", it just fails to *do* anything (nothing ever reads it).

2. None of the flags are documented anywhere; the only way to discover them is to import _ssl (they're not exposed on ssl itself, just the internal C extension), then scan through the exposed names (they're all prefixed with HOSTFLAG_ AFAICT)

3. All of the flags are raw numeric values, but it seems like they should be IntEnums, like the other flags exposed by SSL (among other things, it would make it much easier to interpret the default _host_flags (currently it's just 4, when it could display as <HostFlags.HOSTFLAG_NO_PARTIAL_WILDCARDS: 4>)

4. Nothing about this change, _host_flags/host_flags, or the values of the flags themselves is mentioned on the ssl docs at all.

5. This unintentionally made a behavioral change (one that bit me, and may bite other folks using docker swarm, NETBIOS hostnames, etc.). Python's match_hostname implementation was fine with host names containing underscores (e.g. if the cert was wildcarded to *.example.com, it would match a_b_c.example.com just fine); they're not technically legal by the strict reading of the specs for host names (they're apparently legal for domain names, but not host names, which differ in ways I don't fully understand), but stuff like docker swarm names their services that way automatically, most (all?) browsers support visiting them, etc. It looks like OpenSSL (at least the 1.1.0g my Python 3.7.2 was built against) treats underscores as unmatchable, so any attempt to connect to such a host name in Python 3.7.2 dies with a SSLCertVerificationError, claiming a "Hostname mismatch, certificate is not valid for 'name_with_underscores.example.com'."

I discovered all this because 3.7 broke some scripts I use to connect to docker swarm services. Before I realized the issue was underscores, I was trying to figure out how to tweak the host name checks (assuming maybe something was broken with wildcard matching), and stumbled across all the other issues with the docs, the lack of flag definition exposure, etc.

For the record, I think it's reasonable to require legal host names (it was easy enough to fix for my case; I just updated our docker DNS server to provide aliases using only hyphens and changed the script to use the alias host names), but it would be nice if it was explicitly documented, and ideally, that Python itself recognize that underscores won't work and explicitly raise an exception saying why, rather than letting OpenSSL perform the rejection with a (to someone who doesn't know about the underscore issue) confusing error message.

[tiran]

The entry in whatsnew is a documentation bug. Initially I wanted to expose host_flags and wrote the whatnew entry for it. Later we decided against the flag and an only implemented the hostname_checks_common_name switch (https://docs.python.org/3/library/ssl.html#ssl.SSLContext.hostname_checks_common_name).

1. SSLContext.host_flags in whatsnew is a bug. I'm updating the text.

2/3/4) The _host_flags attribute and the HOSTFLAG_* attributes are for internal use only to provide the hostname_checks_common_name flag.

5. Underscore is not a valid character for hostnames in A, AAAA, CNAME, and similar DNS record types. It's used in e.g. SRV record types, but an application will never directly connect to a SRV record address. It looks like OpenSSL interprets RFC 6125 (https://tools.ietf.org/html/rfc6125#section-6.4.3) strictly and requires valid DNS names.

I wonder, how did you get your DNS server to accept underscores? In theory you should run into a DNS exception earlier.

[miss-islington]

New changeset 47eb223 by Miss Islington (bot) (Christian Heimes) in branch 'master':
bpo-36868: Fix what's new for SSLContext.hostname_checks_common_name (GH-13248)
47eb223

[miss-islington]

New changeset cad4ff6 by Miss Islington (bot) in branch '3.7':
bpo-36868: Fix what's new for SSLContext.hostname_checks_common_name (GH-13248)
cad4ff6