Wrong Certificate Selected When Using Multiple Virtual Host Names in Conscrypt

[mperktold]

In our project, we are using an embedded Jetty Server version 9.4.12.v20180830 with Conscrypt for SSL.
We have some custom handlers that we select based on virtual host names, which works fine so far.
However, when we try to use different certificates for those handlers, Jetty (or Conscrypt?) picks one of
them and uses that one for all handlers, so accessing one of the other handlers fails since the certificate
doesn't match the requested URI.

We are not exactly sure why this is the case, but we made some observations which might help finding
the problem.

In SslContextFactory.customize(SSLParameters), Jetty sets an AliasSNIMatcher as the only SNIMatcher.
Later, in SniX509ExtendedKeyManager.chooseEngineServerAlias(), it tries to extract that matcher from
the engine using engine.getSSLParameters().getSNIMatchers().

But here is the first problem: The Engine doesn't keep the SSLParameters instance, but only extracts
the ones it needs, and the SNIMatchers are not among them. When calling getSSLParameters() like above,
a new SSLParameters object is created with the previously extracted parameters. This means although
we set SNIMatchers, we do not receive them upon calling the getter.

We tried to fix this by patching the Engine to keep the matchers, but then we found another problem.
After extracting the SNIMatchers, Jetty passes them to another method chooseServerAlias(), which
searches an instance of AliasSNIMatcher to access its host and x507. These properties are set
when AliasSNIMatchermatches(SNIServerName) is called, but apparently that never happens, so
host and x507 are null and the method returns NO_MATCHERS.

In fact, we didn't even find any calls of that method in the source code of either Jetty or Conscrypt,
so we are not sure how this is supposed to work.
Also, as a side note: The JavaDoc of SNIMatcher states that instances are immutable. This is apparently
not the case for AliasSNIMatcher, but I guess this is intentional?

But back to the problem:
After chooseServerAlias() returns NO_MATCHERS, the call is delegated to _delegate, which does not
seem to care about multiple aliases and just returns the first one, which explains the behavior we are
experiencing.

In my opinion, there is actually a problem in the way Jetty and Conscrypt are working together to select
the right alias, but the problem could be in our usage of Jetty and virtual hosts as well.
In this regard, we use a custom main handler which selects the actual handler to be called based on the
request URI, including virtual host name information. The actual handler is also a custom handler, but usually
wrapped inside other handlers like GzipHandler or SecurityHandler.

In particular, we are not using ContextHandler. Could that be the problem?

Or could this really be an issue of Jetty or Conscrypt?

[joakime]

Can you try the jetty-9.4.x-2886-SNI-Certs branch?

This might be related to PR #2888 (from Issue #2886)

[mperktold]

I tried the branch you suggested, but the behavior is still the same.

This is not surprising, as the only difference in this branch is how to decide whether SNI is to be used or not. And in our case, we have two wildcard certificates, so both branches do use SNI.

[abarsov]

I have experienced a similar issue with jetty 9.3.24.
At least it might have the same cause as this one.

I have two java services A and B running on the same server.
Both services start embedded jetty server with TLS connector.

Services use different certificates issued for the same hostname - and this is where the problem starts from.

Service A communicate with service B.
So keystore of service A contains two aliases:

• one alias is for its own server certificate
• and another one - for trusted certificate of service B.

Server A starts successfully, but is not accessible - TLS handshake always fails.

Details:

• SslContextFactory#load() populate map _certHosts and since there are two certificates for the same hostname in keystore of service A, only one of them is registered in the map.
• TLS handshake fails if certificate of service B overrides certificate of service A in map _certHosts, in that case AliasSNIMatcher.matches(SNIServerName serverName) always resolve and set certificate of service B for server hostname,
• and its alias in keystore differs from service A certificate alias. The check a.equals(x509.getAlias()) in SniX509ExtendedKeyManager#chooseServerAlias fails, and server certificate is not resolved during handshake.

This issue has obvious work around:
SSLContextFactory can be initialized with keystore that have the only certificate for server hostname.

To fix the issue in code it would be enough to process alias _certAlias last.
It seems to be correct to prefer explicitly defined alias in case of two-certs-for-one-hostname collision .

[joakime]

@sbordet @gregw bump ... is this still valid after the 9.4.13 changes to SNI? (from issue #2886)

[laurentgo]

I also experienced a similar issue with Jetty 9.4.15.v20190215 (reproduced with 9.4.18.v20190429), on OpenJDK 8 using JSSE (not conscrypt).
Our reproduction has been with a keystore containing the same certificate twice under different aliases, but only one of the alias also contains the private key. From the logs, it seems the certificate without the private key is associated with the hostname when reading the keystore (instead of the one containing the private key).

[mperktold]

I created a minimal working example that shows our problem here:
https://github.com/mperktold/jetty-conscrypt-cert-selection

The problem exists with the Jetty version 9.4.20.v20190813, but only when using Conscrypt.

[sbordet]

@mperktold unfortunately this is a known issue in Conscrypt, in particular google/conscrypt#644.
We cannot workaround it, you want to try to comment on the Conscrypt issue instead.

[knaccc]

I think this is resolved now. I have a single jetty instance configured with two different webapps, each with its own domains and certificate. SNI was failing with Conscrypt 2.2.1 but seems to be working OK now with 2.4.0.