TLS mutual Authentication with keystore configuration on Linux OS fails to establish a successful connection

[ragaur-tibco]

Search before asking

• I searched in the issues and found nothing similar.

Version

Java Client(2.11.0)-->broker(2.11.0)
Broker instance OS: Amazon Linux 2023
Java App running OS : RHEL and Ubuntu

Minimal reproduce step

Configure the Pulsar instance for mTLS authentication using Keystore. Then create the Pulsar client using the following sample code.

PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar+ssl://broker.example.com:6651/")
    .useKeyStoreTls(true)
    .tlsTrustStorePath("/var/private/tls/client.truststore.jks")
    .tlsTrustStorePassword("clientpw")
    .allowTlsInsecureConnection(false)
    .enableTlsHostnameVerification(false)
    .authentication(
            "org.apache.pulsar.client.impl.auth.AuthenticationKeyStoreTls",
            "keyStoreType:JKS,keyStorePath:/var/private/tls/client.keystore.jks,keyStorePassword:clientpw")
    .build();

What did you expect to see?

It should get succeeded with the correct keystore path configuration for Linux.

What did you see instead?

trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
keyStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.keystore.jks]
trustStore path ============================================[/home/jnagar/Tibco/BW_Workspace/bw681_pulsar/Pulsar_twossl/certs/client.truststore.jks]
2023-05-03T11:28:07,645 DEBUG [CM Configuration Updater (Update: pid={http://ns.tibco.com/bw/sharedresource/pulsar}PulsarSharedResource.f019e58e-1c01-4874-9375-a3fffcc020b5)] org.apache.pulsar.shade.io.netty.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
2023-05-03T11:28:07,646 DEBUG [CM Configuration Updater (Update: pid={http://ns.tibco.com/bw/sharedresource/pulsar}PulsarSharedResource.f019e58e-1c01-4874-9375-a3fffcc020b5)] org.apache.pulsar.shade.io.netty.util.internal.InternalThreadLocalMap - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024

2023-05-03T11:28:08,252 DEBUG [pulsar-client-io-1-1] org.apache.pulsar.common.util.SecurityUtility - Already instantiated Bouncy Castle provider BCFIPS
2023-05-03T11:28:08,299 ERROR [pulsar-client-io-1-1] org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Exception while trying to refresh ssl Context null (No such file or directory)
java.io.FileNotFoundException: null (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
	at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createSSLContext(KeyStoreSSLContext.java:145)
	at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createClientKeyStoreSslContext(KeyStoreSSLContext.java:230)
	at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:130)
	at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:32)
	at org.apache.pulsar.common.util.SslContextAutoRefreshBuilder.get(SslContextAutoRefreshBuilder.java:79)
	at org.apache.pulsar.client.impl.PulsarChannelInitializer.lambda$initTls$1(PulsarChannelInitializer.java:175)
	at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
	at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
2023-05-03T11:28:08,304 WARN  [pulsar-client-io-1-1] org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to ec2-3-110-90-23.ap-south-1.compute.amazonaws.com:6651 : java.lang.NullPointerException
2023-05-03T11:28:08,407 WARN  [pulsar-client-scheduled-5-1] org.apache.pulsar.client.impl.PulsarClientImpl - [topic: persistent://public/default/test-topic] Could not get connection while getPartitionedTopicMetadata -- Will try again in 100 ms
2023-05-03T11:28:08,407 DEBUG [pulsar-client-scheduled-5-1] org.apache.pulsar.client.impl.ConnectionPool - Connection for ec2-3-110-90-23.ap-south-1.compute.amazonaws.com:6651 not found in cache
2023-05-03T11:28:08,410 ERROR [pulsar-client-io-1-1] org.apache.pulsar.common.util.SslContextAutoRefreshBuilder - Exception while trying to refresh ssl Context null (No such file or directory)
java.io.FileNotFoundException: null (No such file or directory)
	at java.base/java.io.FileInputStream.open0(Native Method)
	at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112)
	at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createSSLContext(KeyStoreSSLContext.java:145)
	at org.apache.pulsar.common.util.keystoretls.KeyStoreSSLContext.createClientKeyStoreSslContext(KeyStoreSSLContext.java:230)
	at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:130)
	at org.apache.pulsar.common.util.keystoretls.NettySSLContextAutoRefreshBuilder.update(NettySSLContextAutoRefreshBuilder.java:32)
	at org.apache.pulsar.common.util.SslContextAutoRefreshBuilder.get(SslContextAutoRefreshBuilder.java:79)
	at org.apache.pulsar.client.impl.PulsarChannelInitializer.lambda$initTls$1(PulsarChannelInitializer.java:175)
	at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
	at org.apache.pulsar.shade.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at org.apache.pulsar.shade.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
	at org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)

Anything else?

As you can observe the debug logs, the keystore path we are providing is not taking into effect and it is being set to null and throwing the file not found error.

Note: Same source code is working fine with windows machine.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[github-actions]

The issue had no activity for 30 days, mark with Stale label.

[tisonkun]

This can be related to #19483. You can try out 2.11.1 or 3.0.0.

cc @nodece

[nodece]

I cannot reproduce this issue, could you check your keyStorePath?

[github-actions]

The issue had no activity for 30 days, mark with Stale label.