HttpExporter encapsulating StreamResetException as IllegalStateException unhandled by the HttpSenders

[ricardo-mestre]

Describe the bug
The HttpExporter is catching StreamResetException (coming from OkHttp) and converting it to IllegalStateException which is non retryable and not catched by the senders. This ends up propagating and killing threads on the threadpool generated for the dispatcher.

Steps to reproduce
While debugging I stopped at HttpExporter.java line 72 and simulated a closed connection, or just throw a StreamResetException. This will propagate back to the threadpool and ignore any retry policy or safely interrupting the thread.

java.lang.IllegalStateException: java.io.IOException: okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
	at io.opentelemetry.exporter.internal.http.HttpExporter.lambda$export$0(HttpExporter.java:75) ~[opentelemetry-exporter-common-1.29.0.jar:1.29.0]
	at io.opentelemetry.exporter.sender.okhttp.internal.OkHttpHttpSender$RawRequestBody.writeTo(OkHttpHttpSender.java:175) ~[opentelemetry-exporter-sender-okhttp-1.29.0.jar:1.29.0]
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:69) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:43) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:94) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:88) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:142) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:117) ~[okhttp-3.14.9.jar:?]
	at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:229) ~[okhttp-3.14.9.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:172) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[okhttp-3.14.9.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.IOException: okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
	at io.opentelemetry.exporter.internal.marshal.ProtoSerializer.close(ProtoSerializer.java:198) ~[opentelemetry-exporter-common-1.29.0.jar:1.29.0]
	at io.opentelemetry.exporter.internal.marshal.Marshaler.writeBinaryTo(Marshaler.java:24) ~[opentelemetry-exporter-common-1.29.0.jar:1.29.0]
	at io.opentelemetry.exporter.internal.http.HttpExporter.lambda$export$0(HttpExporter.java:72) ~[opentelemetry-exporter-common-1.29.0.jar:1.29.0]
	... 20 more
Caused by: okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
	at okhttp3.internal.http2.Http2Stream.checkOutNotClosed(Http2Stream.java:650) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http2.Http2Stream$FramingSink.emitFrame(Http2Stream.java:575) ~[okhttp-3.14.9.jar:?]
	at okhttp3.internal.http2.Http2Stream$FramingSink.write(Http2Stream.java:555) ~[okhttp-3.14.9.jar:?]
	at okio.ForwardingSink.write(ForwardingSink.kt:29) ~[okio-2.10.0.jar:?]
	at okhttp3.internal.connection.Exchange$RequestBodySink.write(Exchange.java:231) ~[okhttp-3.14.9.jar:?]
	at okio.RealBufferedSink.emitCompleteSegments(RealBufferedSink.kt:255) ~[okio-2.10.0.jar:?]
	at okio.RealBufferedSink$outputStream$1.write(RealBufferedSink.kt:117) ~[okio-2.10.0.jar:?]
	at io.opentelemetry.exporter.internal.marshal.CodedOutputStream$OutputStreamEncoder.doFlush(CodedOutputStream.java:581) ~[opentelemetry-exporter-common-1.29.0.jar:1.29.0]
	at io.opentelemetry.exporter.internal.marshal.CodedOutputStream$OutputStreamEncoder.flush(CodedOutputStream.java:537) ~[opentelemetry-exporter-common-1.29.0.jar:1.29.0]
	at io.opentelemetry.exporter.internal.marshal.ProtoSerializer.close(ProtoSerializer.java:191) ~[opentelemetry-exporter-common-1.29.0.jar:1.29.0]
	at io.opentelemetry.exporter.internal.marshal.Marshaler.writeBinaryTo(Marshaler.java:24) ~[opentelemetry-exporter-common-1.29.0.jar:1.29.0]
	at io.opentelemetry.exporter.internal.http.HttpExporter.lambda$export$0(HttpExporter.java:72) ~[opentelemetry-exporter-common-1.29.0.jar:1.29.0]
	... 20 more

What did you expect to see?
Same as OkHttp did, making StreamResetException retryable, and this kind of IllegalStateException should be properly handled by OkHttpSender and JDKSender.

What did you see instead?
An unretryable exception, non means to change that behaviour and dying threads on the original threadpool.

What version and what artifacts are you using?
opentelemetry-sdk-common v1.29.0
io.opentelemetry:opentelemetry-exporter-sender-okhttp v1.29.0

Environment
Compiler: openjdk version "17.0.6"
OS: Mac OS Ventura - 13.3.1

[jack-berg]

Thanks for the report. Will take a look at this and try to resolve 👍

[jack-berg]

Resolved in #6082.