Retry HTTP/2 REFUSED_STREAM errors automatically

[swankjesse]

The specs suggest it's safe to do so

• The REFUSED_STREAM error code can be included in a RST_STREAM frame to indicate that the stream is being closed prior to any processing having occurred. Any request that was sent on the reset stream can be safely retried.

https://httpwg.org/specs/rfc7540.html#Reliability

Nginx can be configured to send this on old-enough connections:

Sets the maximum number of requests that can be served through one keep-alive connection. After the maximum number of requests are made, the connection is closed.

Closing connections periodically is necessary to free per-connection memory allocations. Therefore, using too high maximum number of requests could result in excessive memory usage and not recommended.

http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_requests

And our users are otherwise going to do it manually.

Is there an option (we may have missed) so OkHttp takes care of this transparently to the application?

https://stackoverflow.com/questions/67883227/dealing-with-okhttp-http-2-refused-stream-errors

We need a limit on how many automatic retries to attempt; maybe 1. Such retries should be on a different connection.

[ginkel]

We're seeing the following nginx-HTTP2-related exceptions:

• okhttp3.internal.http2.StreamResetException
• okhttp3.internal.http2.ConnectionShutdownException

okhttp3.internal.http2.ConnectionShutdownException: null
	at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.kt:246)
	at okhttp3.internal.http2.Http2Connection.newStream(Http2Connection.kt:225)
	at okhttp3.internal.http2.Http2ExchangeCodec.writeRequestHeaders(Http2ExchangeCodec.kt:76)
	at okhttp3.internal.connection.Exchange.writeRequestHeaders(Exchange.kt:59)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:36)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	... 43 common frames omitted

At least for the ConnectionShutdownException, retrying from the interceptor does not seem to help and throws another exception:

java.io.IOException: exhausted all routes
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:132)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	[...]

[MFAshby]

I've got a repro of this, working on making it minimal to isolate exactly what setting is breaking okhttp. It's definitely the keepalive_requests directive.

https://github.com/patientsknowbest/stream_refused_repro

[freeleeq]

[自动回复] 邮件我已收到！                          ——葛立青

[ajrice6713]

Seeing this issue as well i believe - Ive tested the same API with multiple different languages + clients, and only with OkHTTP do i get a REFUSED_STREAM error, leading me to believe that the issue is with the client and not our web server.

Exception in thread 4: java.util.concurrent.CompletionException: okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
java.util.concurrent.CompletionException: okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
	at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368)
	at java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377)
	at java.base/java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1097)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2194)
	at com.bandwidth.http.client.OkClient.publishResponse(OkClient.java:197)
	at com.bandwidth.http.client.OkClient.access$000(OkClient.java:35)
	at com.bandwidth.http.client.OkClient$1.onFailure(OkClient.java:133)
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:525)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: okhttp3.internal.http2.StreamResetException: stream was reset: REFUSED_STREAM
	at okhttp3.internal.http2.Http2Stream.takeHeaders(Http2Stream.kt:148)
	at okhttp3.internal.http2.Http2ExchangeCodec.readResponseHeaders(Http2ExchangeCodec.kt:96)
	at okhttp3.internal.connection.Exchange.readResponseHeaders(Exchange.kt:106)
	at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.kt:79)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:34)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at com.bandwidth.http.client.HttpRedirectInterceptor.intercept(HttpRedirectInterceptor.java:38)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)

Any insight into this issue or a fix would be super helpful!

[yschimke]

I'll take a look at this one

[freeleeq]

[自动回复] 邮件我已收到！                          ——葛立青