Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Intermittent java.lang.AssertionError: null appearing in calls to Pyroscope #70

Open
taliatrilling opened this issue Nov 4, 2022 · 6 comments
Open

Intermittent java.lang.AssertionError: null appearing in calls to Pyroscope #70

taliatrilling opened this issue Nov 4, 2022 · 6 comments
Assignees
@korniltsev

Comments

@taliatrilling
Copy link

When using pyroscope in our production environment, we noticed a number of these error messages, albeit intermittently.

java.lang.AssertionError: null at io.otel.pyroscope.shadow.labels.ScopedContext.assertAlive(ScopedContext.java:94) at io.otel.pyroscope.shadow.labels.ScopedContext.<init>(ScopedContext.java:28) at io.otel.pyroscope.PyroscopeOtelSpanProcessor.onStart(PyroscopeOtelSpanProcessor.java:62) at io.opentelemetry.sdk.trace.MultiSpanProcessor.onStart(MultiSpanProcessor.java:40) at io.opentelemetry.sdk.trace.SdkSpan.startSpan(SdkSpan.java:192) at io.opentelemetry.sdk.trace.SdkSpanBuilder.startSpan(SdkSpanBuilder.java:218) at io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.Instrumenter.start(Instrumenter.java:192) at io.opentelemetry.javaagent.shaded.instrumentation.awssdk.v2_2.TracingExecutionInterceptor.afterMarshalling(TracingExecutionInterceptor.java:68) at io.opentelemetry.javaagent.shaded.instrumentation.awssdk.v2_2.autoconfigure.TracingExecutionInterceptor.afterMarshalling(TracingExecutionInterceptor.java:61) at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.lambda$afterMarshalling$4(ExecutionInterceptorChain.java:81) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.afterMarshalling(ExecutionInterceptorChain.java:81) at software.amazon.awssdk.core.internal.handler.BaseClientHandler.runAfterMarshallingInterceptors(BaseClientHandler.java:151) at software.amazon.awssdk.core.internal.handler.BaseClientHandler.finalizeSdkHttpFullRequest(BaseClientHandler.java:82) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:151) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:82) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.putItem(DefaultDynamoDbClient.java:4200) at com.modeanalytics.xxxxxx(xxxxxx.java:148) at com.modeanalytics.xxxxxx(xxxxxx.java:110) at com.modeanalytics.xxxxxx(xxxxxx.java:74) at jdk.internal.reflect.GeneratedMethodAccessor65.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:244) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:47) at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:41) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:319) at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:273) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at com.modeanalytics.xxxxxx(xxxxxx.java:21) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at com.modeanalytics.xxxxxx(xxxxxx.java:27) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at com.modeanalytics.xxxxxx(xxxxxx.java:32) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:284) at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:722) at io.dropwizard.jetty.ZipExceptionHandlingGzipHandler.handle(ZipExceptionHandlingGzipHandler.java:26) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:829)

I tried to walk through the pyroscope code to understand the issue, but was not very successful.
@petethepig
Copy link
Member

Thanks for the report @taliatrilling

We'll look into this soon!

@korniltsev
Copy link
Collaborator

@taliatrilling I s there any chance you could run a custom version of pyroscope-otel with extra asserts, not in production, just in local/dev env? That might help.
https://s01.oss.sonatype.org/content/repositories/snapshots/io/pyroscope/otel/0.10.1.5-SNAPSHOT/otel-0.10.1.5-20221107.165005-2.jar
Built from this branch: https://github.com/pyroscope-io/pyroscope-java/pull/71/files

Also do you run with otel.pyroscope.root.span.only=false?

Also can you share which libraries(Otel instrumentations) are used in the app?(except jetty and dynamodb)

@taliatrilling
Copy link
Author

Hi @korniltsev -

We run with whatever the default is for otel.pyroscope.root.span.only, which looking here looks like is true.

We have been using pyroscope-otel-profiling-0.10.1.4 as I was under the impression from this thread in the honeycomb slack that we were required to be pinned to that version in order to use the Honeycomb Java Agent with Pyroscope.

In addition to the libraries you mentioned, we are using Dropwizard and the AWS S3 SDK.

@korniltsev
Copy link
Collaborator

as I was under the impression from this thread in the honeycomb slack that we were required to be pinned to that version

I don't think there is a requirement to pin to that specific version exactly.

0.10.1.3 was the first version to implement pyroscope as an Otel extension.
And 0.10.1.4 was a tiny fix for compatibility with not so fresh Otel versions. So the requirement is >= 0.10.1.4, not == 0.10.1.4.

We run with whatever the default is for otel.pyroscope.root.span.only, which looking here looks like is true.

That is weird. In that case I'd expect to have a root span of "HTTP request" or smth from jetty/jersy and dynamodb span be a child of it. And with otel.pyroscope.root.span.only=true periscope should only process the first one. Do you usually have an "HTTP request" span as the top span in the trace or it is common for dynamodb span to be the root span?

@korniltsev
Copy link
Collaborator

I will try to reproduce with Dropwizard, s3 as well

@korniltsev
Copy link
Collaborator 

28-Apr-2023 09:52:48.891 WARNING [http-nio-0.0.0.0-8080-exec-1] org.apache.catalina.core.AsyncContextImpl.timeout onTimeout() call failed for listener of type [org.apache.cataina.core.AsyncListenerWrapper]
        java.lang.AssertionError: Thread check failure: Thread[http-nio-0.0.0.0-8080-exec-11,5,main] 3742 != Thread[http-nio-0.0.0.0-8080-exec-1,5,main] 2684
                at io.otel.pyroscope.shadow.labels.ScopedContext.close(ScopedContext.java:76)
                at io.otel.pyroscope.PyroscopeOtelSpanProcessor.onEnd(PyroscopeOtelSpanProcessor.java:111)
                at io.opentelemetry.sdk.trace.MultiSpanProcessor.onEnd(MultiSpanProcessor.java:52)
                at io.opentelemetry.sdk.trace.SdkSpan.endInternal(SdkSpan.java:445)
                at io.opentelemetry.sdk.trace.SdkSpan.end(SdkSpan.java:425)
                at io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.Instrumenter.doEnd(Instrumenter.java:243)
                at io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.Instrumenter.end(Instrumenter.java:144)
                at io.opentelemetry.javaagent.instrumentation.servlet.AsyncRequestCompletionListener.onTimeout(AsyncRequestCompletionListener.java:48)
                at io.opentelemetry.javaagent.instrumentation.servlet.v3_0.Servlet3Accessor$Listener.onTimeout(Servlet3Accessor.java:87)
                at org.apache.catalina.core.AsyncListenerWrapper.fireOnTimeout(AsyncListenerWrapper.java:44)
                at org.apache.catalina.core.AsyncContextImpl.timeout(AsyncContextImpl.java:136)
                at org.apache.catalina.connector.CoyoteAdapter.asyncDispatch(CoyoteAdapter.java:146)
                at org.apache.coyote.AbstractProcessor.dispatch(AbstractProcessor.java:241)
                at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)
                at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
                at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)
                at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
                at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
                at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.base/java.lang.Thread.run(Thread.java:833)

APP: 2023-04-28 08:45:38,789 [Thread-1317][, ] INFO  package.hidden.ClassHidden - ERROR MESSAGE 
Exception in thread "Thread-1317" org.springframework.web.reactive.function.client.WebClientRequestException: Thread check failure: Thread[Thread-1317,5,main] 3444 != Thread[re
actor-http-epoll-3,5,main] 3427; nested exception is java.lang.AssertionError: Thread check failure: Thread[Thread-1317,5,main] 3444 != Thread[reactor-http-epoll-3,5,main] 3427
        at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)
        Suppressed: The stacktrace has been enhanced by Reactor, refer to additional information below:
Error has been observed at the following site(s):
        *__checkpoint ⇢ Request to POST https://URL/ENDPOINT?QUERY=VALUE [DefaultWebClient]
Original Stack Trace:
                at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.lambda$wrapException$9(ExchangeFunctions.java:141)
                at reactor.core.publisher.MonoErrorSupplied.subscribe(MonoErrorSupplied.java:55)
                at reactor.core.publisher.Mono.subscribe(Mono.java:4490)
                at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:103)
                at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
                at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
                at reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:222)
                at reactor.core.publisher.MonoNext$NextSubscriber.onError(MonoNext.java:93)
                at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onError(MonoFlatMapMany.java:255)
                at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onError(FluxContextWrite.java:121)
                at reactor.core.publisher.Operators.reportThrowInSubscribe(Operators.java:232)
                at reactor.core.publisher.Flux.subscribe(Flux.java:8645)
                at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:195)
                at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
                at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)
                at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:172)
                at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:431)
                at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:707)
                at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onStateChange(DefaultPooledConnectionProvider.java:193)
                at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnection.onStateChange(DefaultPooledConnectionProvider.java:454)
                at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:647)
                at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
                at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
                at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
                at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
                at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
                at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
                at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
                at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
                at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
                at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
                at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
                at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
                at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
                at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
                at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
                at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
                at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
                at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
                at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
                at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
                at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
                at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
                at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
                at java.base/java.lang.Thread.run(Thread.java:833)
        Suppressed: java.lang.Exception: #block terminated with an error
                at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
                at reactor.core.publisher.Mono.block(Mono.java:1742)
                at package.hidden.ClassHidden.methodHidden(ClassHidden.java:96)
                at package.hidden.ClassHidden.methodHidden(ClassHidden.java:59)
                at package.hidden.ClassHidden.methodHidden(ClassHidden.java:28)
                at package.hidden.ClassHidden.methodHidden(ClassHidden.java:32)
                at package.hidden.ClassHidden.methodHidden(ClassHidden.java:34)
                at package.hidden.ClassHidden.methodHidden(ClassHidden.java:30)
                at package.hidden.ClassHidden.methodHidden(ClassHidden.java:29)
                at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.AssertionError: Thread check failure: Thread[Thread-1317,5,main] 3444 != Thread[reactor-http-epoll-3,5,main] 3427
        at io.otel.pyroscope.shadow.labels.ScopedContext.close(ScopedContext.java:76)
        at io.otel.pyroscope.PyroscopeOtelSpanProcessor.onEnd(PyroscopeOtelSpanProcessor.java:111)
        at io.opentelemetry.sdk.trace.MultiSpanProcessor.onEnd(MultiSpanProcessor.java:52)
        at io.opentelemetry.sdk.trace.SdkSpan.endInternal(SdkSpan.java:445)
        at io.opentelemetry.sdk.trace.SdkSpan.end(SdkSpan.java:425)
        at io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.Instrumenter.doEnd(Instrumenter.java:243)
        at io.opentelemetry.javaagent.shaded.instrumentation.api.instrumenter.Instrumenter.end(Instrumenter.java:144)
        at io.opentelemetry.javaagent.shaded.instrumentation.spring.webflux.v5_3.internal.TraceWebClientSubscriber.onNext(TraceWebClientSubscriber.java:50)
        at io.opentelemetry.javaagent.shaded.instrumentation.spring.webflux.v5_3.internal.TraceWebClientSubscriber.onNext(TraceWebClientSubscriber.java:20)
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200)
        at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122)
        at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200)
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200)
        at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200)
        at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onNext(MonoFlatMapMany.java:250)
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
        at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2400)
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.request(FluxContextWrite.java:136)
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onSubscribeInner(MonoFlatMapMany.java:150)
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner.onSubscribe(MonoFlatMapMany.java:245)
        at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onSubscribe(FluxContextWrite.java:101)
        at reactor.core.publisher.FluxJust.subscribe(FluxJust.java:68)
        at reactor.core.publisher.Flux.subscribe(Flux.java:8642)
        at reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain.onNext(MonoFlatMapMany.java:195)
        at reactor.core.publisher.SerializedSubscriber.onNext(SerializedSubscriber.java:99)
        at reactor.core.publisher.FluxRetryWhen$RetryWhenMainSubscriber.onNext(FluxRetryWhen.java:174)
        at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:172)
        at reactor.netty.http.client.HttpClientConnect$HttpIOHandlerObserver.onStateChange(HttpClientConnect.java:431)
        at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:707)
        at reactor.netty.resources.DefaultPooledConnectionProvider$DisposableAcquire.onStateChange(DefaultPooledConnectionProvider.java:193)
        at reactor.netty.resources.DefaultPooledConnectionProvider$PooledConnection.onStateChange(DefaultPooledConnectionProvider.java:454)
        at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:647)
        at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:113)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
        at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)

Thanks to Adam Kaczmarski https://pyroscope.slack.com/archives/C01FJRYENPQ/p1682676466647379?thread_ts=1682600996.218059&cid=C01FJRYENPQ

@korniltsev korniltsev mentioned this issue Sep 4, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@korniltsev korniltsev

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@korniltsev @petethepig @taliatrilling