mpRestClient-1.3 ignoring hostnameVerifier configuration

[jkjome]

Describe the bug

I have a server I'm calling where the domain doesn't exactly match the domain defined in the certificate loaded into my custom keystore. I've tried configuring a hostnameVerifier on my mpRestClient using both CDI-configured mpConfig properties from microprofile-config.properties as well as, alternatively, using a programmatically loaded client. In both cases, my custom HostnameVerifier appears to be totally ignored in favor of the Apache CXF org.apache.cxf.transport.https.httpclient.DefaultHostnameVerifier, which results in failure.

Here's how I configure when using a CDI-injected mpRestClient...

myRestClient/mp-rest/hostnameVerifier=com.acme.SSLAffirmativeHostnameVerifier

Here's how I configure when using a programmatically-configured mpRestClient...

final MyRestClient cl = RestClientBuilder.newBuilder()
    .baseUri(URI.create("https://someserver.acme.com"))
    .hostnameVerifier(new SSLAffirmativeHostnameVerifier())
    .build(MyRestClient.class);

Here's my custom hostnameVerifier class...

public class SSLAffirmativeHostnameVerifier implements HostnameVerifier {
	@Override
	public boolean verify(String hostname, SSLSession session) {
		System.out.println("SSLAffirmativeHostnameVerifier.verify() invoked");
		return true;
	}
}

Stack Dump = java.lang.RuntimeException: HostnameVerifier, socket reset for TTL
	at org.apache.cxf.transport.https.httpclient.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:98)
	at java.base/sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:640)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:581)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1337)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:246)
	at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.setupWrappedStream(URLConnectionHTTPConduit.java:312)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleHeadersTrustCaching(HTTPConduit.java:1391)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.onFirstWrite(HTTPConduit.java:1352)
	at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.onFirstWrite(URLConnectionHTTPConduit.java:345)
	at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:47)
	at org.apache.cxf.io.AbstractThresholdOutputStream.unBuffer(AbstractThresholdOutputStream.java:89)
	at org.apache.cxf.io.AbstractThresholdOutputStream.write(AbstractThresholdOutputStream.java:63)
	at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:51)
	at org.apache.cxf.io.AbstractWrappedOutputStream.write(AbstractWrappedOutputStream.java:60)
	at com.ibm.ws.jaxrs21.providers.json.JsonBProvider.writeTo(JsonBProvider.java:203)
	at org.apache.cxf.jaxrs.utils.JAXRSUtils$3.run(JAXRSUtils.java:1481)
	at org.apache.cxf.jaxrs.utils.JAXRSUtils$3.run(JAXRSUtils.java:1478)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:734)
	at org.apache.cxf.jaxrs.utils.JAXRSUtils.writeMessageBody(JAXRSUtils.java:1478)
	at org.apache.cxf.jaxrs.client.AbstractClient.writeBody(AbstractClient.java:518)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl$BodyWriter.doWriteBody(ClientProxyImpl.java:1089)
	at org.apache.cxf.jaxrs.client.AbstractClient$AbstractBodyWriter.handleMessage(AbstractClient.java:1243)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
	at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:712)
	at org.apache.cxf.microprofile.client.proxy.MicroProfileClientProxyImpl.doRunInterceptorChain(MicroProfileClientProxyImpl.java:179)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:895)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:343)
	at org.apache.cxf.microprofile.client.proxy.MicroProfileClientProxyImpl.invokeActual(MicroProfileClientProxyImpl.java:463)
	at org.apache.cxf.microprofile.client.proxy.MicroProfileClientProxyImpl.access$000(MicroProfileClientProxyImpl.java:76)
	at org.apache.cxf.microprofile.client.proxy.MicroProfileClientProxyImpl$Invoker.call(MicroProfileClientProxyImpl.java:485)
	at org.apache.cxf.microprofile.client.cdi.CDIInterceptorWrapperImpl.invoke(CDIInterceptorWrapperImpl.java:190)
	at org.apache.cxf.microprofile.client.proxy.MicroProfileClientProxyImpl.invoke(MicroProfileClientProxyImpl.java:458)
	at com.sun.proxy.$Proxy59.callService(Unknown Source)
	at com.acme.MyManager.callServiceAsynch(MyManager.java:99)
	at com.acme.MyManager.callService(MyManager.java:63)
	at com.acme.MyManager$Proxy$_$$_WeldClientProxy.callService(Unknown Source)
	at com.acme.MyResource.callService(DocuSignResource.java:33)
	at com.acme.MyResource$Proxy$_$$_WeldClientProxy.callService(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.ibm.ws.jaxrs20.server.LibertyJaxRsServerFactoryBean.performInvocation(LibertyJaxRsServerFactoryBean.java:656)
	at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.performInvocation(LibertyJaxRsInvoker.java:160)
	at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
	at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:273)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:205)
	at com.ibm.ws.jaxrs20.server.LibertyJaxRsInvoker.invoke(LibertyJaxRsInvoker.java:444)
	at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:111)
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:61)
	at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:99)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
	at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:124)
	at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:274)
	at com.ibm.ws.jaxrs20.endpoint.AbstractJaxRsWebEndpoint.invoke(AbstractJaxRsWebEndpoint.java:134)
	at com.ibm.websphere.jaxrs.server.IBMRestServlet.handleRequest(IBMRestServlet.java:146)
	at com.ibm.websphere.jaxrs.server.IBMRestServlet.doPost(IBMRestServlet.java:104)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:706)
	at com.ibm.websphere.jaxrs.server.IBMRestServlet.service(IBMRestServlet.java:96)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:729)
	at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:426)
	at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1226)
	at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:5021)
	at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.handleRequest(DynamicVirtualHost.java:314)
	at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:1007)
	at com.ibm.ws.webcontainer.osgi.DynamicVirtualHost$2.run(DynamicVirtualHost.java:279)
	at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink$TaskWrapper.run(HttpDispatcherLink.java:1134)
	at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.wrapHandlerAndExecute(HttpDispatcherLink.java:415)
	at com.ibm.ws.http.dispatcher.internal.channel.HttpDispatcherLink.ready(HttpDispatcherLink.java:374)
	at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:548)
	at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.handleNewRequest(HttpInboundLink.java:482)
	at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.processRequest(HttpInboundLink.java:347)
	at com.ibm.ws.http.channel.internal.inbound.HttpInboundLink.ready(HttpInboundLink.java:318)
	at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:167)
	at com.ibm.ws.tcpchannel.internal.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:75)
	at com.ibm.ws.tcpchannel.internal.WorkQueueManager.requestComplete(WorkQueueManager.java:504)
	at com.ibm.ws.tcpchannel.internal.WorkQueueManager.attemptIO(WorkQueueManager.java:574)
	at com.ibm.ws.tcpchannel.internal.WorkQueueManager.workerRun(WorkQueueManager.java:958)
	at com.ibm.ws.tcpchannel.internal.WorkQueueManager$Worker.run(WorkQueueManager.java:1047)
	at com.ibm.ws.threading.internal.ExecutorServiceImpl$RunnableWrapper.run(ExecutorServiceImpl.java:239)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:831)

Steps to Reproduce
Run a CDI-configured or programmatically-configured mpRestClient (while using a custom keystore - because that might be an important aspect of this issue), using configuration like I provided further above, and call a server that accepts the certificate but where the DNS name does not exactly match that defined in the certificate.

One can verify that the precondition is in place by enabling the transportSecurity-1.0 feature and setting verifyHostname="true" on the defaultSSLConfig (and ensure keyStoreRef is valued with your custom keystore Id). In such case, you should see something like...

CWPKI0824E: SSL HANDSHAKE FAILURE: Host name verification error while connecting to host [someserver.acme.com]. The host name used to access the server does not match the server certificate's SubjectDN or Subject Alternative Name information. The extended error message from the SSL handshake exception is: [No subject alternative DNS name matching someserver.acme.com found.].

Expected behavior
Hostname verification should pass and I should see the following written to the message.log file...

SSLAffirmativeHostnameVerifier.verify() invoked

Instead, hostname verification fails and I find no evidence that my custom SSLAffirmativeHostnameVerifier is ever invoked. I also get the stacktrace above, which clearly shows that org.apache.cxf.transport.https.httpclient.DefaultHostnameVerifier is being used for hostname verification instead of my custom hostname verifier class.

Diagnostic information:
product = Open Liberty 20.0.0.2 (wlp-1.0.37.cl200220200204-1746)
java.version = 11.0.5
java.runtime = OpenJDK Runtime Environment (11.0.5+10)
os = Windows 10 (10.0; amd64) (en_US)

server.xml...

<server description="Default Liberty server config">
    <featureManager>
        <feature>jaxrs-2.1</feature>
        <feature>jsonp-1.1</feature>
        <feature>cdi-2.0</feature>
        <feature>mpRestClient-1.3</feature>
        <feature>mpConfig-1.3</feature>
        <feature>jsonb-1.0</feature>
    </featureManager>

    <httpEndpoint host="*" httpPort="${endpoint_http_port}" httpsPort="${endpoint_https_port}" id="defaultHttpEndpoint"/>

    <webApplication location="${application_location}.war" contextRoot="/"/>
    
    <variable name="endpoint_http_port" defaultValue="9080"/>
    <variable name="endpoint_https_port" defaultValue="9443"/>
</server>

configDropins/defaults/keystore.xml...

<server description="Default keystore configuration">
    <featureManager>
    	<feature>appSecurity-2.0</feature>
    </featureManager>
    <keyStore id="defaultKeyStore" location="key.jks" password="changeit" type="jks"/>
</server>

jvm.options...

-Dsun.net.inetaddr.ttl=60
-Djava.security.properties=file:./custom.security

custom.security...

crypto.policy=unlimited
jdk.tls.disabledAlgorithms=TLSv1, TLSv1.1, SSLv3, RC4, MD5withRSA, DH keySize < 768,TLSv1,TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_MD5,TLS_RSA_WITH_RC4_128_SHA

Additional context
Note that I am using the openliberty-microProfile3 package run via...

mvn liberty:dev -Ddebug=false -DskipTests=true

Note that I use a custom keystore. I have already opened another issue (#11103) regarding unnecessarily having to use the appSecurity-2.0 (or 3.0) feature, instead of just ssl-1.0 or transportSecurity-1.0, in order for my custom keystore to be utilized. Whether that has any bearing to this issue is unknown to me, but I thought I'd mention it.

[jkjome]

A couple things I've noticed....

First, a side-note. Apparently one can set hostname verification at the SSL config level without applying the transportSecurity-1.0 feature, as I thought was implied by the Open Liberty 19.0.0.6 announcement. Not super important, but interesting to note.

Second, There appear to be two, somewhat redundant, hostname verification levels implemented in Open Liberty....

One level is the server SSL config, where one can optionally set verifyHostname=“true” (default is false) where hostname verification is controlled by com.ibm.ws.ssl.core.WSX509TrustManager. When enabled, it takes precedence...

Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching someserver.acme.com found.
	at com.ibm.ws.ssl.core.WSX509TrustManager.processServerTrustError(WSX509TrustManager.java:938)
	at com.ibm.ws.ssl.core.WSX509TrustManager.checkServerTrusted(WSX509TrustManager.java:729)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:625)
	... 95 more

A second level of hostname verification appears to be applied by the Apache CXF (the Impl for mpRestClients in Open Liberty) when verifyHostname=“false” (or isn't set at all) via org.apache.cxf.transport.https.httpclient.DefaultHostnameVerifier. So, while one can enable hostname verification at a higher level, if no earlier hostname verification is performed then it is, eventually, performed by Apache CXF. This begs the question of what use is the SSL config when there is already fallback hostname verification? And, more importantly, why don't mpRestClients respect the hostnameVerifier config?

For the time being, hostname verification in Open Liberty appears to be redundantly enabled, but with no way to disable.

[jkjome]

After testing with a older-style JAX-RS-2.1 client, I've got incontrovertible proof that this issue isn't due to something unique about my own appserver environment. The mpRestClient-1.3 hostnameVerifier functionality is definitely broken. I can confidently state this because an equivalent JAX-RS-2.1 client that I wrote, which makes use of the same custom SSLAffirmativeHostnameVerifier, actually passed hostname verification! The debugging line in my custom hostname verifier was actually printed, proving it was invoked. I was finally able to obtain a proper response from the server using the following JAX-RS-2.1 client code...

final Client cl = ClientBuilder.newBuilder()
	.hostnameVerifier(new SSLAffirmativeHostnameVerifier())
	.build();
final WebTarget wt = cl.target("https://someserver.acme.com");
return wt.path("/somepath").request("application/json").post(Entity.entity(reqEntityObj, "application/json"), ResEntityObj.class));

Please fix ASAP, as I really need to be able to make use of the mpConfig aspects of my mpRestClient.