Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reactor Netty intermittently throws IllegalStateException: Only one connection receive subscriber is allowed #13501

Closed
kushagraThapar opened this issue Jul 25, 2020 · 0 comments · Fixed by #13493
Closed

Reactor Netty intermittently throws IllegalStateException: Only one connection receive subscriber is allowed #13501

kushagraThapar opened this issue Jul 25, 2020 · 0 comments · Fixed by #13493
Assignees
@kushagraThapar
Labels
Client This issue points to a problem in the data-plane of the library. cosmos:v4-item Indicates this feature will be shipped as part of V4 release train Cosmos
Projects
Cosmos DB SDK team
Milestone
Backlog

Comments

@kushagraThapar
Copy link
Member

Describe the bug
This is happening after we fixed SSLException: SSLHandler is closed already.

Exception or Stack Trace

java.lang.IllegalStateException: Only one connection receive subscriber allowed.
2020-07-24T23:09:51.4300513Z 	at reactor.netty.channel.FluxReceive.subscribe(FluxReceive.java:175) [reactor-netty-0.9.10.RELEASE.jar:0.9.10.RELEASE]
2020-07-24T23:09:51.4302101Z 	at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4303637Z 	at reactor.netty.ByteBufFlux.subscribe(ByteBufFlux.java:327) [reactor-netty-0.9.10.RELEASE.jar:0.9.10.RELEASE]
2020-07-24T23:09:51.4305169Z 	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4307004Z 	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4308614Z 	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4310161Z 	at reactor.netty.ByteBufMono.subscribe(ByteBufMono.java:134) [reactor-netty-0.9.10.RELEASE.jar:0.9.10.RELEASE]
2020-07-24T23:09:51.4311700Z 	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4313273Z 	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4314914Z 	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4316643Z 	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:173) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4318554Z 	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4320282Z 	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:203) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4378086Z 	at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1782) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4390343Z 	at reactor.core.publisher.MonoSingle$SingleSubscriber.onComplete(MonoSingle.java:171) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4392170Z 	at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4393797Z 	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2346) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4395779Z 	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onSubscribeInner(MonoFlatMapMany.java:143) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4397557Z 	at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:182) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4399181Z 	at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4400839Z 	at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:162) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4409136Z 	at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156) [reactor-core-3.3.8.RELEASE.jar:3.3.8.RELEASE]
2020-07-24T23:09:51.4420478Z 	at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:431) [reactor-netty-0.9.10.RELEASE.jar:0.9.10.RELEASE]
2020-07-24T23:09:51.4422418Z 	at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:514) [reactor-netty-0.9.10.RELEASE.jar:0.9.10.RELEASE]
2020-07-24T23:09:51.4442330Z 	at reactor.netty.resources.PooledConnectionProvider$DisposableAcquire.onStateChange(PooledConnectionProvider.java:540) [reactor-netty-0.9.10.RELEASE.jar:0.9.10.RELEASE]
2020-07-24T23:09:51.4444400Z 	at reactor.netty.resources.PooledConnectionProvider$PooledConnection.onStateChange(PooledConnectionProvider.java:427) [reactor-netty-0.9.10.RELEASE.jar:0.9.10.RELEASE]
2020-07-24T23:09:51.4449504Z 	at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:574) [reactor-netty-0.9.10.RELEASE.jar:0.9.10.RELEASE]
2020-07-24T23:09:51.4459412Z 	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96) [reactor-netty-0.9.10.RELEASE.jar:0.9.10.RELEASE]
2020-07-24T23:09:51.4497459Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4503407Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4509224Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4515592Z 	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4521510Z 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4527697Z 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:311) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4541939Z 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:425) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4543871Z 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4548635Z 	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4550762Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4555815Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4558459Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4563847Z 	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1526) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4565805Z 	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1275) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4567418Z 	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1322) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4575662Z 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4577631Z 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4599379Z 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4603835Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4608182Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4612721Z 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4615117Z 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4619563Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4623954Z 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4625899Z 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4630957Z 	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
2020-07-24T23:09:51.4633614Z 	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
2020-07-24T23:09:51.4638198Z 	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
2020-07-24T23:09:51.4643746Z 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4645384Z 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4658286Z 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.51.Final.jar:4.1.51.Final]
2020-07-24T23:09:51.4659876Z 	at java.lang.Thread.run(Thread.java:834) [?:?]

To Reproduce
It can be reproduced in HTTP E2E CI in cosmos - java - tests

Code Snippet
I believe this method is being subscribed twice - which could be causing this issue, however this is just a guess based on the stack trace.

        private ByteBufFlux bodyIntern() {
            return reactorNettyConnection.inbound().receive();
        }

Expected behavior
No errors at all.

Setup (please complete the following information):

  • OS: [e.g. iOS] - Ubuntu
  • IDE : [e.g. IntelliJ]
  • Cosmos SDK - 4.3.0-beta.1
@kushagraThapar kushagraThapar added Cosmos Client This issue points to a problem in the data-plane of the library. cosmos:v4-item Indicates this feature will be shipped as part of V4 release train labels Jul 25, 2020
@kushagraThapar kushagraThapar added this to the Backlog milestone Jul 25, 2020
@kushagraThapar kushagraThapar self-assigned this Jul 25, 2020
@kushagraThapar kushagraThapar added this to To do in Cosmos DB SDK team via automation Jul 25, 2020
@kushagraThapar kushagraThapar linked a pull request Jul 25, 2020 that will close this issue
Consume/drain the inbound in case a cancellation is received before subscription #13493
Merged
Cosmos DB SDK team automation moved this from To do to Done Jul 27, 2020
@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kushagraThapar kushagraThapar

Labels
Client This issue points to a problem in the data-plane of the library. cosmos:v4-item Indicates this feature will be shipped as part of V4 release train Cosmos
Projects
Cosmos DB SDK team
  
Done
Milestone
Backlog
1 participant
@kushagraThapar