TLS handshake fails with BC internal error, works fine with TLS implementation shipped with OpenJDK 15.0.2+7

[toby1984]

Hi,

We're in the process of upgrading our application to run on CentOS8 and noticed that our Java server (currently running BC 1.68 but I also did test 1.69 beta which shows the same error) can no longer connect to our C++ webservices running on another machine.

The same connection still works fine if we switch to using the JDK built-in TLS implementation that comes with OpenJDK 15.0.2+7.

I've attached a PCAP file of the handshake to this ticket, feel free to contact me if you need further data/information.

The exception we're getting on the Java side is

Caused by: org.bouncycastle.tls.TlsFatalAlert: internal_error(80) at org.bouncycastle.tls.TlsProtocol.safeReadRecord(TlsProtocol.java:811) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.TlsProtocol.blockForHandshake(TlsProtocol.java:370) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.TlsClientProtocol.connect(TlsClientProtocol.java:86) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.jsse.provider.ProvSSLSocketDirect.startHandshake(ProvSSLSocketDirect.java:445) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.jsse.provider.ProvSSLSocketDirect.startHandshake(ProvSSLSocketDirect.java:426) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) ~[?:?] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1375) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1350) ~[?:?] at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:220) ~[?:?] at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:119) ~[jaxws-rt-2.3.1.jar:2.3.1] ... 20 more

Caused by: java.lang.IllegalStateException: Invalid algorithm: {sha256(4),ecdsa(3)} at org.bouncycastle.tls.crypto.impl.jcajce.JcaTlsRSASigner.generateRawSignature(JcaTlsRSASigner.java:59) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.DefaultTlsCredentialedSigner.generateRawSignature(DefaultTlsCredentialedSigner.java:53) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.TlsUtils.generateCertificateVerifyClient(TlsUtils.java:2176) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.TlsClientProtocol.handleHandshakeMessage(TlsClientProtocol.java:638) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.TlsProtocol.processHandshakeQueue(TlsProtocol.java:629) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.TlsProtocol.processRecord(TlsProtocol.java:519) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.RecordStream.readRecord(RecordStream.java:245) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.TlsProtocol.safeReadRecord(TlsProtocol.java:777) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.TlsProtocol.blockForHandshake(TlsProtocol.java:370) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.tls.TlsClientProtocol.connect(TlsClientProtocol.java:86) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.jsse.provider.ProvSSLSocketDirect.startHandshake(ProvSSLSocketDirect.java:445) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at org.bouncycastle.jsse.provider.ProvSSLSocketDirect.startHandshake(ProvSSLSocketDirect.java:426) ~[bctls-jdk15on-169b04.jar:1.68.99.0] at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:567) ~[?:?] at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1375) ~[?:?] at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1350) ~[?:?] at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:220) ~[?:?] at com.sun.xml.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:119) ~[jaxws-rt-2.3.1.jar:2.3.1] ... 20 more

broken_handshake.pcap.gz

[peterdettman]

Something appears to be wrong with the way the credentials are being loaded (it somehow gets RSA credentials when it's trying to sign SHA256withECDSA, and this is later caught by a consistency check. Do you have Java Logging enabled? It would show us some more details during the process of signature algorithm selection and KeyManager lookup.

e.g. edit your JAVA_HOME/conf/logging properties:

#java.util.logging.ConsoleHandler.level = INFO
java.util.logging.ConsoleHandler.level = FINEST
org.bouncycastle.jsse.level = FINEST

Please post the resulting logs.

One quick thing to check is that you have these security properties set (e.g. in JAVA_HOME/conf/security/java.security):

ssl.KeyManagerFactory.algorithm=PKIX
ssl.TrustManagerFactory.algorithm=PKIX

[peterdettman]

One more thing, I recall a similar issue being reported last year and in that case there was a custom X509KeyManager implementation being used (as a wrapper for the BC one). Please let me know whether that is the case here too.

[toby1984]

Hmmm.... guilty on all accounts ;) We're using JAX-WS (JDK built-in SOAP stack) with SSL client authentication so had to use a custom SSLSocketFactory to make it work.

Here's the class that I wrote (which is based on stuff I found on the internet, appropriate links are given in the header)

SSLFactorySocketGenerator.java.gz

[peterdettman]

OK, try changing AliasSelectorKeyManager.chooseClientAlias so that it first delegates to sourceKeyManager, then only return that result if it matches clientAliasId.

[toby1984]

Thanks for the swift response, really appreciate that !

Your suggested change gets rid of the exception but the connection still does not work (BC X509KeyManager#chooseClientAlias() returns NULL). IMHO the interesting question is: why is the handshake working when using the JDK default provider (useJdkSSL == true in my code) but not with BouncyCastle ? Is there a way to increase BC logging so I can see why it considers the key in the keystore 'unsuitable' ?

[toby1984]

If this is of any help, I could provide you with the certificates and keys used (as it's only an internal CI system and all certs are randomly auto-generated during the build)... I also have the output of JDK11 running with "javax.net.debug=ssl:handshake:verbose:trustmanager:keymanager but it's rather messy as there are a lot of concurrent SSL connections going on.

[peterdettman]

I described how to increase the logging in my first response above. Please send me a log. Also make absolutely sure that you are getting the BCJSSE provider's KeyManagerFactory implementation (of type "PKIX") - our SSLContext can only be used with our KMF and TMF and will not interoperate with the default JDK ones (in the general case).

[peterdettman]

Note that chooseClientAlias may be called multiple times, so we expect it to return null e.g. for certain keyTypes. I assume you meant that it is always returning null.

I have seen a case where the server sends a faulty set of issuers. Try passing null for the 'issuers' argument in the sourceKeyManager.chooseClientAlias call and see if it changes things.

[toby1984]

Your comment about the KMF was spot-on. When refactoring the initial code to also support BC JSSE I missed updating all the right places so it was still using the JDK KMF ... after fixing this it works like a charm !

Just in case somebody else encounters the same issue, these were the problems I fixed in my code:

1.) not calling the original X509KeyManager when proxying the chooseClientAlias() method
2.) mixing the JDK KMF with BouncyCastle JSSE

Sorry for the noise and thanks again for helping to resolve this so quickly !