ERL-1260: hostname check failing with wildcard

[OTP-Maintainer]

Original reporter: benoitc
Affected version: OTP-23
Component: ssl
Migrated from: https://bugs.erlang.org/browse/ERL-1260

---

validation of a wildcard certificate is failing on OTP 23 with the error {{*CLIENT ALERT: Fatal - Handshake Failure, - \{bad_cert,hostname_check_failed}*.}} It is working correctly on OTP 22.1

This can be reproduced by the following code:
{code:java}
70> ssl:connect("contacts.google.com", 443, [{cacertfile, certifi:cacertfile()}, {server_name_indication, "contacts.google.com"},  {reuse_sessions, false}, {verify, verify_peer}, {depth, 99}]).
2020-05-20T10:10:03.257852+02:00 notice: TLS client: In state wait_cert_cr at ssl_handshake.erl:1840 generated CLIENT ALERT: Fatal - Handshake Failure, - {bad_cert,hostname_check_failed}
{code}
 

 

[OTP-Maintainer]

voltone said:

It fails for me also on 22.3.2 and 22.1.8.1, unless I also add:

{customize_hostname_check, [\\{match_fun, public_key:pkix_verify_hostname_match_fun(https)}]}

What changed in 23 is that the hostname check now also happens when a custom verify_fun is given. So in Hackney I guess you relying on ssl_verify_fun to do the hostname check, and now in 23 OTP does its own check before calling ssl_verify_fun. By default OTP does not permit wildcards in SAN extensions, so for the server in the example the connection fails before ssl_verify_fun is called, unless the customize_hostname_check option is added.

Mint was bitten by this as well (it uses a custom verify_fun): [https://github.com/elixir-mint/mint/pull/259]

[OTP-Maintainer]

voltone said:

 TLS Hostname Validation Not Performed when User verify_fun is Supplied ERL-1232

[OTP-Maintainer]

ingela said:

Well as pointed out this is not due to a bug, rather it behaves as expected after fixing a bug.

There are different ways to handle wildcard certs which depends on the  application using the ssl application. So if you want a HTTP way you can use the customize option as suggested.  From a security point of view wildcard certs should not be used.

[OTP-Maintainer]

JIRAUSER15301 said:

>  From a security point of view wildcard certs should not be used.

Wildcard certs are perfectly fine; this kind of dismissive comment is not. There are a myriad of ways any system can be compromised, but the best way of ensuring an attacker cannot easily forge or intercept is to use certificates with a short of a life as possible so a leaked certificate has expired. Let's not derail the focus of fixing something important that every other ecosystem handles gracefully because of a concern that a hundred other security layers have been violated and an attacker found an unprotected wildcard cert.

[OTP-Maintainer]

ingela said:

I am sorry you feel it was dismissive, that was not the intent.

This was not my personal opinion. See for instance:

[https://gist.github.com/joepie91/7e5cad8c0726fd6a5e90360a754fc568]

I  intended to make people consider  if  wildcards certs is  what they really want. We do support them for people deciding they want to use them.  We also provide a function in public_key for making it easier to apply the HTTPS rules for wildcard certs.  However due to another bug it happened that no host name check was performed under some circumstances, so then some people missed that they needed to configure this and was surprised to learn so when the bug was fixed. That can of course be seen as unfortunate, but it is not a bug. 

 

 

[OTP-Maintainer]

JIRAUSER15301 said:

Sure, but this is not new information to anyone with experience in TLS. If it was up to me everyone would be storing their keys in an HSM so the material can never leak. But that's not possible, and everyone has different threat vectors, so let's focus on fixing this in OTP so certificate verification just works out of the box for everyone.

 
----
 

I think it would be a very bad decision to require everyone needing to validate wildcard certs to write their own code for hostname verification. It will be buggy, error prone, and adds significant friction to using Erlang/OTP as an HTTPS/TLS client.

The time for Erlang/OTP to become a dominant platform is now. The internet is getting much bigger, there is a significant investment in Federated protocols, and the end result will be a new generation of software that communicates with thousands or even millions of unique servers instead of one centralized server or internal corporate system. Handling this kind of workload is something Erlang/OTP is very suitable for, so let's strive to ensure Erlang/OTP Just Works^TM reliably as a TLS client with no hidden gotchas or ever requiring custom client glue to validate.

 

>  There are different ways to handle wildcard certs which depends on the  application using the ssl application.

 

Can you explain what you mean here? I don't see how there can be more than one way to handle a wildcard cert: it either matches the hostname pattern, or it doesn't. [RFC|https://en.wikipedia.org/wiki/RFC_(identifier)] [2818|https://tools.ietf.org/html/rfc2818] clearly states how wildcard certs work.

[OTP-Maintainer]

ingela said:

Different  application layer protocols like HTTP and LDAP for instance handle them differently. RFC 2818 (is an HTTP RFC).

The user of such common protocols  does not need to write any  code of their own to handle wildcard certificates. They only need to point at the appropriate function provided by our public_key application to handle it for them. 

What defaults an HTTP client has and what defaults the ssl application has are different. The ssl application implements the TLS protocol and  can not make application layer decisions.

There will always be trade offs, that the user may need to configure.  

Our goal is also to make as it easy for the user as possible, but that does not mean being bug compatible.  

 

 

[OTP-Maintainer]

acaruth said:

As the raiser of the original and now-fixed Erlang bug (ERL-1232), I'd agree this _is_ a further bug as it seems to be an incomplete application of [https://tools.ietf.org/html/rfc6125#section-6.4.3] and very much part of the TLS protocol.

I'd hope this is corrected in future release, without the need for the {{customize_hostname_check}} workaround. In particular the following simple change to {{public_key.erl}} seems like it should be added as part of this ticket if re-opened.
{noformat}
--- otp_src_23.0/lib/public_key/src/public_key.erl      2020-05-12 21:50:47.000000000 +0100
+++ otp_src_23.0/lib/public_key/src/public_key.erl.new  2020-08-12 13:28:36.670556069 +0100
@@ -1722,8 +1722,10 @@
     not lists:member($*, FQDN);
 verify_hostname_match_default0(FQDN=[_|_], {cn,Name=[_|_]}) ->
     verify_hostname_match_wildcard(FQDN, Name);
+verify_hostname_match_default0({dns_id,R}, {dNSName,R}) ->
+    true;
 verify_hostname_match_default0({dns_id,R}, {dNSName,P}) ->
-    R==P;
+    verify_hostname_match_wildcard(R, P);
 verify_hostname_match_default0({uri_id,R}, {uniformResourceIdentifier,P}) ->
     R==P;
 verify_hostname_match_default0({ip,R}, {iPAddress,P}) when length(P) == 4 ->
{noformat}

[OTP-Maintainer]

jimdigriz said:

For reference, accessing the [Microsoft Azure ServiceBus|https://docs.microsoft.com/en-us/rest/api/servicebus/service-bus-runtime-rest] endpoints at {{.servicebus.windows.net}} fails due to this too; the workaround in https://bugs.erlang.org/browse/ERL-1260?focusedCommentId=18015&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-18015 works nicely to resolve it.

[OTP-Maintainer]

ingela said:

 

The   RFC  6125:  "Representation and Verification of Domain-Based Application Service Identity within Internet Public Key Infrastructure Using X.509 (PKIX) Certificates in the Context of Transport Layer Security (TLS)"  

The context of TLS  not  being the same thing as the TLS protocol.   So if might for instance be reasonable for a HTTP  client to default set the customization and allow wildcard certificates the "HTTP way" . The customize_hostname_check is not a workaround  it is an option  letting different types of clients behave as they want/need.

[OTP-Maintainer]

jimdigriz said:

Sorry, I should have been clearer I was using {{httpc}}:
{code}
alex@aineko:~$ erl
Erlang/OTP 23 [erts-11.1] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1]

Eshell V11.1  (abort with ^G)
1> application:ensure_all_started(ssl).
{ok,[crypto,asn1,public_key,ssl]}
2> application:ensure_all_started(inets).
{ok,[inets]}
3> httpc:request(get, {"https://dns-wingman-63f8c8af-c990-5040-ba04-9065cb9489f2.servicebus.windows.net",[]}, [{ssl,[{verify,verify_peer},{cacertfile,"/etc/ssl/certs/ca-certificates.crt"}]}], []).                       
=NOTICE REPORT==== 8-Oct-2020::07:45:57.346032 ===
TLS client: In state certify at ssl_handshake.erl:1952 generated CLIENT ALERT: Fatal - Handshake Failure
 - {bad_cert,hostname_check_failed}
{error,{failed_connect,[{to_address,{"dns-wingman-63f8c8af-c990-5040-ba04-9065cb9489f2.servicebus.windows.net",
                                     443}},
                        {inet,[inet],
                              {tls_alert,{handshake_failure,"TLS client: In state certify at ssl_handshake.erl:1952 generated CLIENT ALERT: Fatal - Handshake Failure\n {bad_cert,hostname_check_failed}"}}}]}}
{code}

Shouldn't {{httpc}} automatically use the {{customize_hostname_check}}?

[OTP-Maintainer]

ingela said:

That  would be a feature request for the inets application and not a bug report on ssl.  I think at least httpc could have its own option along the lines "allow_wild_card _certs = boolean()" that could possibly default to allowing it. That is a trade off between interoperability  and  security concerns.  But it seems reasonable that HTTP clients can have some own logic and not leave all configuration of lower level applications up to its users.

 

[OTP-Maintainer]

jimdigriz said:

Whilst I am in [there writing unit tests|https://bugs.erlang.org/browse/ERL-1321?focusedCommentId=19626&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-19626] want me to do another PR for this?

[OTP-Maintainer]

ingela said:

That is up to you. Even though we (OTP) agree such a feature is reasonable  it is not a priority to Ericsson at the moment ....