DispatchFailure: HTTP/2 REFUSED_STREAM should be retried

[FSMaxB]

Describe the bug

We regularly get an HTTP/2 REFUSED_STREAM error in production when deleting S3 blobs. This error means that a request should be retried, so I'm opening an issue based on the following warning that was emitted:

aws_smithy_client::hyper_ext
unrecognized error from Hyper. If this error should be retried, please file an issue.

{err: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic (hyper::Error(Http2, Error { kind: Reset(StreamId(243), REFUSED_STREAM, Remote) }))}

See https://datatracker.ietf.org/doc/html/rfc9113#REFUSED_STREAM and https://datatracker.ietf.org/doc/html/rfc9113#Reliability saying that such requests can be safely retried.

Expected Behavior

When calling DispatchFailure::other, it should have returned Some. Unclear which ErrorKind exactly, but probably ErrorKind::ServerError.

Current Behavior

When calling DispatchFailure::other, it returns None.

Reproduction Steps

I haven't found a way to reproduce this yet, I can only see it in production sometimes when using aws-sdk-s3 against the google cloud storage API.

Possible Solution

Treat the REFUSED_STREAM error as aws_smithy_types::retry::ErrorKind::ServerError

Additional Information/Context

No response

Version

$ cargo tree | grep aws -
├── aws-sdk-s3 v0.28.0
│   ├── aws-credential-types v0.55.3
│   │   ├── aws-smithy-async v0.55.3
│   │   ├── aws-smithy-types v0.55.3
│   ├── aws-endpoint v0.55.3
│   │   ├── aws-smithy-http v0.55.3
│   │   │   ├── aws-smithy-eventstream v0.55.3
│   │   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-types v0.55.3
│   │   │   ├── aws-credential-types v0.55.3 (*)
│   │   │   ├── aws-smithy-async v0.55.3 (*)
│   │   │   ├── aws-smithy-client v0.55.3
│   │   │   │   ├── aws-smithy-async v0.55.3 (*)
│   │   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   │   ├── aws-smithy-http-tower v0.55.3
│   │   │   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   │   ├── aws-smithy-types v0.55.3 (*)
│   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   │   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-http v0.55.3
│   │   ├── aws-credential-types v0.55.3 (*)
│   │   ├── aws-smithy-http v0.55.3 (*)
│   │   ├── aws-smithy-types v0.55.3 (*)
│   │   ├── aws-types v0.55.3 (*)
│   ├── aws-sig-auth v0.55.3
│   │   ├── aws-credential-types v0.55.3 (*)
│   │   ├── aws-sigv4 v0.55.3
│   │   │   ├── aws-smithy-eventstream v0.55.3 (*)
│   │   │   ├── aws-smithy-http v0.55.3 (*)
│   │   ├── aws-smithy-eventstream v0.55.3 (*)
│   │   ├── aws-smithy-http v0.55.3 (*)
│   │   ├── aws-types v0.55.3 (*)
│   ├── aws-sigv4 v0.55.3 (*)
│   ├── aws-smithy-async v0.55.3 (*)
│   ├── aws-smithy-checksums v0.55.3
│   │   ├── aws-smithy-http v0.55.3 (*)
│   │   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-smithy-client v0.55.3 (*)
│   ├── aws-smithy-eventstream v0.55.3 (*)
│   ├── aws-smithy-http v0.55.3 (*)
│   ├── aws-smithy-http-tower v0.55.3 (*)
│   ├── aws-smithy-json v0.55.3
│   │   └── aws-smithy-types v0.55.3 (*)
│   ├── aws-smithy-types v0.55.3 (*)
│   ├── aws-smithy-xml v0.55.3
│   ├── aws-types v0.55.3 (*)
├── aws-smithy-async v0.55.3 (*)
├── aws-smithy-http v0.55.3 (*)
├── aws-smithy-types v0.55.3 (*)
├── aws-sdk-s3 v0.28.0 (*)

Environment details (OS name and version, etc.)

Docker on Linux (not really relevant)

Logs

As shown above:

aws_smithy_client::hyper_ext
unrecognized error from Hyper. If this error should be retried, please file an issue.

{err: http2 error: stream error received: refused stream before processing any application logic: stream error received: refused stream before processing any application logic (hyper::Error(Http2, Error { kind: Reset(StreamId(243), REFUSED_STREAM, Remote) }))}

[Velfi]

Hey @FSMaxB, thanks for submitting this issue. We'll add it to our backlog.

[FSMaxB]

Might also be this issue: hyperium/h2#707

I'll report back wether that already fixes it once I have that version running in production.

[FSMaxB]

The new h2 release actually fixes this for my use-case. There might still be other scenarios that trigger this though.

[jdisanti]

Built-in retry was added for this error in the December 5, 2023 release.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.