Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pulsar-broker] Broker auto refresh bk-client certs to avoid cnx failure after cert refresh #12107

Merged
merged 2 commits into from
Sep 22, 2021
Merged

[pulsar-broker] Broker auto refresh bk-client certs to avoid cnx failure after cert refresh #12107

merged 2 commits into from
Sep 22, 2021

Conversation

rdhabalia
Copy link
Contributor

Motivation

Broker is not able to auto-cert refresh. Broker should support a refresh duration and perform auto-cert refresh.

00:12:59.613 [bookkeeper-io-12-34] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - TLS handshake failed
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000415:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) ~[netty-codec-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-transport-native-epoll-4.1.60.Final-linux-x86_64.jar:4.1.60.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-transport-native-epoll-4.1.60.Final-linux-x86_64.jar:4.1.60.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.60.Final-linux-x86_64.jar:4.1.60.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.60.Final.jar:4.1.60.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: error:10000415:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.shutdownWithError(ReferenceCountedOpenSslEngine.java:1069) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1359) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1308) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1384) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1427) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:207) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1388) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1295) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1332) ~[netty-handler-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.60.Final.jar:4.1.60.Final]
        ... 15 more
00:12:59.613 [bookkeeper-io-12-34] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Unexpected exception caught by bookie client channel handler
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000415:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) ~[netty-codec-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795) [netty-transport-native-epoll-4.1.60.Final-linux-x86_64.jar:4.1.60.Final]
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-transport-native-epoll-4.1.60.Final-linux-x86_64.jar:4.1.60.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.60.Final-linux-x86_64.jar:4.1.60.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.60.Final.jar:4.1.60.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.60.Final.jar:4.1.60.Final]
        at java.lang.Thread.run(Thread.java:834) [?:?]

@rdhabalia rdhabalia added this to the 2.9.0 milestone Sep 20, 2021
@rdhabalia rdhabalia self-assigned this Sep 20, 2021
merlimat
merlimat reviewed Sep 21, 2021
View reviewed changes
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
@@ -1352,6 +1352,9 @@
@FieldContext(category = CATEGORY_STORAGE_BK, doc = "Path for the trusted TLS certificate file")
private String bookkeeperTLSTrustCertsFilePath;

@FieldContext(category = CATEGORY_STORAGE_BK, doc = "Tls cert refresh duration at bookKeeper-client in seconds (0 to disable check)")
private int bookkeeperTlsCertFilesRefreshDurationSeconds = 300;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also add the ref on conf/broker.conf and site2/docs/reference-configuration.md

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. done.

@merlimat merlimat merged commit 7e7dc32 into apache:master Sep 22, 2021
@rdhabalia rdhabalia deleted the bk_cert branch September 22, 2021 02:39
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
@rdhabalia @ciaocloud
[pulsar-broker] Broker auto refresh bk-client certs to avoid cnx fail…
eb195bf 
…ure after cert refresh (apache#12107)

* [pulsar-broker] Broker auto refresh bk-client certs to avoid cnx failure after cert refresh

* fix readme
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@merlimat merlimat merlimat approved these changes

Assignees

@rdhabalia rdhabalia

Labels
area/broker release/2.9.0
Projects
None yet
Milestone
2.9.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@rdhabalia @merlimat @eolivelli