jetcd cannot reload renewed client certificate automatically

[guyoulan]

Hi,

I'm using jetcd as java client to connect etcd server with tls, but face problem when the client certificate expires, it can not automatically reload the cert and failed to connect server.

   private static SslContext getOpenSslContext(EtcdClientConfig config) throws SSLException {
        File trustManagerFile = new File(config.getCaCert());
        File keyCertChainFile = new File(config.getCliCert());
        File KeyFile = new File(config.getCliKey());

        ApplicationProtocolConfig alpn = new ApplicationProtocolConfig(ApplicationProtocolConfig.Protocol.ALPN,
              ApplicationProtocolConfig.SelectorFailureBehavior.NO_ADVERTISE,
              ApplicationProtocolConfig.SelectedListenerFailureBehavior.ACCEPT,
              ApplicationProtocolNames.HTTP_2);
        sslContext = SslContextBuilder
              .forClient()
              .applicationProtocolConfig(alpn)
              .sslProvider(SslProvider.OPENSSL)
              .trustManager(trustManagerFile)
              .keyManager(keyCertChainFile, KeyFile)
              .build();
        return sslContext;
    }

        ClientBuilder builder = Client.builder();
        builder.endpoints(config.getEndPoints()).sslContext(getOpenSslContext(config));
        Client client = builder.build();

When the client cert expires, the following error appears when connect:
06:04:56.301 [pool-4-thread-1] ERROR com.ericsson.cces.extensionmanager.in.JaxrsExceptionMapper - creating a 500 response
io.etcd.jetcd.common.exception.EtcdException: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
at io.etcd.jetcd.common.exception.EtcdExceptionFactory.newEtcdException(EtcdExceptionFactory.java:34)
at io.etcd.jetcd.common.exception.EtcdExceptionFactory.fromStatus(EtcdExceptionFactory.java:82)
at io.etcd.jetcd.common.exception.EtcdExceptionFactory.toEtcdException(EtcdExceptionFactory.java:78)
at io.etcd.jetcd.common.exception.EtcdExceptionFactory.toEtcdException(EtcdExceptionFactory.java:73)
at io.etcd.jetcd.ClientConnectionManager.generateToken(ClientConnectionManager.java:303)
at io.etcd.jetcd.ClientConnectionManager.getToken(ClientConnectionManager.java:135)
at io.etcd.jetcd.ClientConnectionManager.access$200(ClientConnectionManager.java:71)
at io.etcd.jetcd.ClientConnectionManager$AuthTokenInterceptor$1.start(ClientConnectionManager.java:380)
at io.grpc.ForwardingClientCall.start(ForwardingClientCall.java:32)
at io.etcd.jetcd.ClientConnectionManager$1$1.start(ClientConnectionManager.java:275)
at io.grpc.stub.ClientCalls.startCall(ClientCalls.java:332)
at io.grpc.stub.ClientCalls.asyncUnaryRequestCall(ClientCalls.java:306)
at io.grpc.stub.ClientCalls.futureUnaryCall(ClientCalls.java:218)
at io.etcd.jetcd.api.KVGrpc$KVFutureStub.range(KVGrpc.java:494)
at io.etcd.jetcd.KVImpl.lambda$get$3(KVImpl.java:112)
at io.etcd.jetcd.ClientConnectionManager.lambda$execute$4(ClientConnectionManager.java:349)
at net.jodah.failsafe.Functions$1.get(Functions.java:134)
at net.jodah.failsafe.Functions$1.get(Functions.java:129)
at net.jodah.failsafe.Functions$2.get(Functions.java:226)
at net.jodah.failsafe.Functions$2.get(Functions.java:216)
at net.jodah.failsafe.internal.util.DelegatingScheduler.lambda$schedule$0(DelegatingScheduler.java:141)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1069)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1359)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1308)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1384)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1427)
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:208)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1358)
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1253)
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1300)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted

Form server side,
2021-09-27 06:05:20.045583 I | embed: rejected connection from "192.168.7.8:54170" (error "tls: failed to verify client's certificate: x509: certificate has expired or is not yet valid", ServerName "")

What's the problem? Does jetcd support authentication refresh in its tls connection?

[lburgazzoli]

Well, if the certificates has expired, you'd probably need to provide the new one and restart the application. jetcd does not watch for the files read by the trustManager and keyManager for changes.

[guyoulan]

Hi @lburgazzoli

The new one already renew and placed in the same location which already provided to jetcd client.
Then what does this code test for? Seems it says can refresh token even if the ca cert expired.
https://github.com/etcd-io/jetcd/blob/master/jetcd-core/src/test/java/io/etcd/jetcd/WatchTokenExpireTest.java

[guyoulan]

If need to restart the application, all the watch client, lease keepalive connection are shutting down....

[lburgazzoli]

cleont certificate expiration and token refresh are two different things, how is your etcd started ?

[guyoulan]

I use etcd cluster in kubernetes environment, it enables tls authentication.
Well, so jetcd currently does not support client cert refresh in its connection, right?
This should be a common question, yet may not jetcd issue. I just wander how others handle this kind of scenario.

[guyoulan]

Or is there any way to set ssl socket for this scenario in jetcd client but not close the client and create another client?

[lburgazzoli]

Again there are two things:

1. token refresh which should be supported as per the test you linked
2. hot reload of a client cert file which is not supported and (not sure if that is even possible but you should check the underlying grpc-java library as the stack if provided by such library)

[lburgazzoli]

Have a look at: grpc/grpc-java#5335

[guyoulan]

Thanks, I'm looking into that link.

[lburgazzoli]

@guyoulan it is ok if I close this issue ? It does not seems something directly related to jetcd and if you find a way to make it easy, I'm happy to incorporate a PR

[guyoulan]

@lburgazzoli Sorry for late reply, you can close this issue. Agree that it's not directly related to jetcd.

[Hakky54]

@guyoulan it is ok if I close this issue ? It does not seems something directly related to jetcd and if you find a way to make it easy, I'm happy to incorporate a PR

Hello! I noticed this issue from here: grpc/grpc-java#5335

I wanted to share a solution which already works with gRPC java. Maybe it would be usefull for the jetcd project. See here for an example setup:

SSLFactory baseSslFactory = SSLFactory.builder()
        .withDummyIdentityMaterial()
        .withDummyTrustMaterial()
        .withSwappableIdentityMaterial()
        .withSwappableTrustMaterial()
        .build();

Runnable sslUpdater = () -> {
    SSLFactory updatedSslFactory = SSLFactory.builder()
            .withIdentityMaterial(Paths.get("/path/to/your/identity.jks"), "password".toCharArray())
            .withTrustMaterial(Paths.get("/path/to/your/truststore.jks"), "password".toCharArray())
            .build();

    SSLFactoryUtils.reload(baseSslFactory, updatedSslFactory);
};

// initial update of ssl material to replace the dummies
sslUpdater.run();

// update ssl material every hour
Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(sslUpdater, 1, 1, TimeUnit.HOURS);

SslContext sslContext = GrpcSslContexts.configure(NettySslUtils.forClient(baseSslFactory)).build();

Client client = Client.builder()
    .endpoints("http://etcd0:2379", "http://etcd1:2379", "http://etcd2:2379")
    .sslContext(sslContext);
    .build();