[DynamoDB] RequestError: send request failed

[lsimunec]

Describe the bug

On random occasions we receive connection reset by peer when querying Dynamo database.

The error message "connection reset by peer" typically occurs in network-related communication, and it often indicates that the connection between two systems has been unexpectedly closed by the remote end, which, in this case, is the DynamoDB service.

Expected Behavior

Expected to not happen or at least be handled by dynamodb.ErrCodeInternalServerError.
If already handled by some error, please document in which one.

Current Behavior

RequestError: send request failed
caused by: Post "https://dynamodb.eu-central-1.amazonaws.com/": read tcp 100.117.202.148:56260->35.71.105.0:443: read: connection reset by peer

Reproduction Steps

Cannot reproduce, appears to be random on all types of queries.

Example
output, err := e.client.QueryWithContext(ctx, input)

Possible Solution

No response

Additional Information/Context

No response

SDK version used

v1.45.11

Environment details (Version of Go (go version)? OS name and version, etc.)

Go 1.20 and Go 1.21

[RanVaknin]

Hi @MikaelSmith,

Expected to not happen or at least be handled by dynamodb.ErrCodeInternalServerError.

The error you're seeing originates directly from the networking layer. This occurs before the request has made a full round trip to and from the DynamoDB service, disrupting the process before a response is received.
Because of this disruption, the SDK doesn't have service data required to deserialize it into a service-specific, concrete error, like dynamodb.ErrCodeInternalServerError.

Instead, a common strategy of http clients is to retry on transient error (connection reset, connection timeout etc). The Go SDK is already equipped to deal with this specific error in a form of retry. connection reset error.

Since this is not reproducible, caused by networking stack, and we have a mitigation strategy (retry) this is not actionable by the Go SDK team. You can enable logging to make sure that retries are taking effect.

If you have any other questions please let us know.

Thanks,
Ran~

[github-actions]

This issue has not received a response in 1 week. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled.

[wojtekwanczyk]

Refreshing the issue as I see the same read: connection reset by peer error not being retried by DynamoDB.

I checked the function referenced by @RanVaknin and it seems that this specific error string (read: connection reset) is excluded from being Connection Reset Error in this commit by @jasdel. Why is this excluded from retry process?

SDK version used

v1.42.25

Environment details (Version of Go (go version)? OS name and version, etc.)

Go 1.21

[bensie]

@wojtekwanczyk Been seeing this issue as well. While it's not particularly well-documented in that commit, a read: connection reset here means that the request was already handled by the server and is being interrupted on the way back to the client. Since this could have been a write request (PutItem, etc), it's not inherently safe to retry. It should be safe to retry in Query or GetItem requests, but since this is configured at the client level and not the operation level, the default needs to assume it's unsafe for retry.

aws-sdk-go-v2 supports modifying the retryer per-operation:

	getItemOutput, err := DynamoDB.GetItem(ctx, input, func(o *dynamodb.Options) {
		o.Retryer = ...
	})

Haven't found a way to do this yet in v1 - @RanVaknin can you speak to whether or not that's possible?