Broker errors when handshake with other nodes

[soullkk]

Broker errors when handshake with other nodes

2023-04-02 04:35:24,831 WARN [qtp1319386445-190[groupBy_[xxxx]_7a5c4d33-5a19-41d3-b39c-0d1f4458e68c]][][org.apache.druid.java.util.http.client.NettyHttpClient] Netty faulty channel failed!!!
org.jboss.netty.channel.ChannelException: Failed to handshake with host[https://x.x.x.x:26204]
at org.apache.druid.java.util.http.client.pool.ChannelResourceFactory$2$1.operationComplete(ChannelResourceFactory.java:157) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:409) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:395) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:362) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1461) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.ssl.SslHandler.access$200(SslHandler.java:180) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.ssl.SslHandler$1.run(SslHandler.java:372) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: javax.net.ssl.SSLException: Handshake did not complete within 10000ms
... 6 more
2023-04-02 04:35:24,875 WARN [ForkJoinPool-1-worker-14][][org.apache.druid.client.JsonParserIterator] Query [7a5c4d33-5a19-41d3-b39c-0d1f4458e68c] to host [x.x.x.x:26204] interrupted
org.jboss.netty.channel.ChannelException: Faulty channel in resource pool
at org.apache.druid.java.util.http.client.NettyHttpClient.go(NettyHttpClient.java:132) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.client.DirectDruidClient.run(DirectDruidClient.java:462) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.getSimpleServerResults(CachingClusteredClient.java:723) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.lambda$addSequencesFromServer$9(CachingClusteredClient.java:685) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at java.util.TreeMap.forEach(TreeMap.java:1005) ~[?:1.8.0_362]
at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.addSequencesFromServer(CachingClusteredClient.java:669) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.client.CachingClusteredClient$SpecificQueryRunnable.lambda$run$2(CachingClusteredClient.java:397) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.LazySequence.toYielder(LazySequence.java:46) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.MergeSequence.lambda$toYielder$1(MergeSequence.java:65) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.MergeSequence.toYielder(MergeSequence.java:62) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.query.RetryQueryRunner$1.toYielder(RetryQueryRunner.java:134) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.common.guava.CombiningSequence.toYielder(CombiningSequence.java:78) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.MappedSequence.toYielder(MappedSequence.java:49) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:88) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence$2.get(WrappingSequence.java:84) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:78) ~[druid-processing-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.WrappingSequence.toYielder(WrappingSequence.java:83) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.java.util.common.guava.Yielders.each(Yielders.java:32) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.server.QueryResource.doPost(QueryResource.java:230) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at sun.reflect.GeneratedMethodAccessor115.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.3.jar:1.19.3]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.3.jar:1.19.3]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.3.jar:1.19.3]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.3.jar:1.19.3]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) ~[guice-servlet-4.1.0.jar:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:82) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:75) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.apache.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:84) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77) ~[druid-server-0.21.1-h0.gdd.sop.r49.jar:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.48.v20220622.jar:9.4.48.v20220622]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.jboss.netty.channel.ChannelException: Failed to handshake with host[https://x.x.x.x:26204]
at org.apache.druid.java.util.http.client.pool.ChannelResourceFactory$2$1.operationComplete(ChannelResourceFactory.java:157) ~[druid-core-0.21.1-h0.gdd.sop.r49.jar:?]
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:409) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:395) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:362) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1461) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.ssl.SslHandler.access$200(SslHandler.java:180) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.handler.ssl.SslHandler$1.run(SslHandler.java:372) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?]
... 1 more
Caused by: javax.net.ssl.SSLException: Handshake did not complete within 10000ms
at org.jboss.netty.handler.ssl.SslHandler$1.run(SslHandler.java:372) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369) ~[netty-3.10.6.Final.jar:?]
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) ~[netty-3.10.6.Final.jar:?]

Affected Version

druid 0.21.1 version

Description

Please include as much detailed information about the problem as possible.

• total 7 nodes in cluster

This exception occurs frequently.
I found that the exception occurred in the SslHandler.handshake() when the ChannelFuture is null or not done.
The source code in netty is as follows：

        try {
            engine.beginHandshake();
            runDelegatedTasks();
            handshakeFuture = this.handshakeFuture = future(channel);
            if (handshakeTimeoutInMillis > 0) {
                handshakeTimeout = timer.newTimeout(new TimerTask() {
                        public void run(Timeout timeout) throws Exception {
                        ChannelFuture future = SslHandler.this.handshakeFuture;
                        if (future != null && future.isDone()) {
                            return;
                        }
                        setHandshakeFailure(channel, new SSLException("Handshake did not complete within " +
                                        handshakeTimeoutInMillis + "ms"));
                    }
                    }, handshakeTimeoutInMillis, TimeUnit.MILLISECONDS);
            }
        } catch (Exception e) {
            handshakeFuture = this.handshakeFuture = failedFuture(channel, e);
            exception = e;
        }

I checked the server's configuration for druid. server. http. numThreads as follows：
overlord=50
coordinator=50
indexer=25
historical=25
broker=12

When there is a exception connecting to other nodes(coodinator,historical,overlord,indexer), the metric of "jetty/numOpenConnections" are as follows:
{"feed":"metrics","timestamp":"2023-05-05T10:00:26.959Z","service":"druid/coordinator","host":"x.x.x.x:26200","version":"","metric":"jetty/numOpenConnections","value":141}
{"feed":"metrics","timestamp":"2023-05-06T10:12:44.278Z","service":"druid/historical","host":"x.x.x.x:26202","version":"","metric":"jetty/numOpenConnections","value":49}
{"feed":"metrics","timestamp":"2023-05-06T12:26:32.252Z","service":"druid/overlord","host":"x.x.x.x:26203","version":"","metric":"jetty/numOpenConnections","value":158}
{"feed":"metrics","timestamp":"2023-05-06T18:10:34.614Z","service":"druid/indexer","host":"x.x.x.x:26204","version":"","metric":"jetty/numOpenConnections","value":44}

[soullkk]

@techdocsmith @abhishekagarwal87 Do you have any ideas or suggestions to help me continue analyzing this issue？I would greatly appreciate it.

[abhishekagarwal87]

How many brokers do you have? This seems like a sizing issue. I will suggest going through https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#sizing-the-connection-pool-for-queries so you can make sure that indexers/peons have the right number of threads.

[soullkk]

The broker has 6 nodes with the following IP addresses ：xxx.8, xxx.11, xxx.9, xxx.12, xxx.5, xxx.10
historical has 7 nodes with the following IP addresses ：xxx.9, xxx.7, xxx.5, xxx.11, xxx.8, xxx.10, xxx.12
The parameter 'druid. server. http. numThreads' value for the broker is 12
The parameter 'druid. server. http. numThreads' value for the historical is 25

Follow the guide of https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html#sizing-the-connection-pool-for-queries, the value of parameter 'druid. server. http. numThreads' for historical should be set to 82?

[soullkk]

For testing purposes, I attempted to set the value of parameter 'druid. server. http. numThreads' for historical to 1，but the issue did not recur

[soullkk]

It should be noted that this issue not only occurs when connecting to port 26204, but also when connecting to port 26200，26202，26203

[abhishekagarwal87]

The threads on historicals and peons = (number of outgoing connections from broker) * (number of brokers)
so if you have 20 connections on broker and 6 brokers, you should ideally have 120 threads on data nodes.

[soullkk]

Okay, I will follow your suggestions to set the values for historicals and peons, but how to set the overlord and coordinator? This issue may also occur occasionally when the broker connects to them.

[abhishekagarwal87]

We have this setting druid.global.http.eagerInitialization=false that you can set in runtime properties. It will disable eager initialization of connections to master nodes. You don't need eager initialization in the path that doesn't involve querying. In querying path, we create connections eagerly so the query performance is high from the get go and doesn't suffer from connection initialization latency.

[soullkk]

I tried to grab the handshake timeout packets and found that the client closed the connection after the Change Cipher Spec. client port is 36631, server port is 26202. Do you have any suggestions to analyze the occurrence of this situation? I am not very familiar with TLS handshake.

[soullkk]

Thanks very much!

[soullkk]

Do Druid have any plans to upgrade to the Netty version？I submitted this issue to Netty, but they no longer support 3.10.6. They suggest upgrading to at least 4.1.@abhishekagarwal87