Skip to content

RetryOnException in RetryStrategy does not retry on UncheckedIOException #6578

@carlosbaumont

Description

@carlosbaumont

Describe the bug

Customizing retryable exceptions as described in the documentation by using the retryOnException function in RetryStrategy class does not trigger retries when an UncheckedIOException is configured.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

The retry strategy should recognise UncheckedIOException as an additional retryable exception and automatically retry operations when this exception is thrown.

Current Behavior

Stack trace from the local test using TestContainers:

java.io.UncheckedIOException: java.io.IOException
	at com.qmetric.testcontainersutils.TestHttpClient$1.call(S3ClientTest.java:60)
	at com.qmetric.testcontainersutils.TestHttpClient$1.call(S3ClientTest.java:50)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:103)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest(MakeHttpRequestStage.java:88)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:64)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute(MakeHttpRequestStage.java:46)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.executeRequest(RetryableStage.java:93)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:56)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
	at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:12535)
	at com.example.S3ClientTest.testS3Client(S3ClientTest.java:32)

Stack trace from the issue in production:

software.amazon.awssdk.utils.FunctionalUtils.asRuntimeException (FunctionalUtils.java:180)
software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4 (FunctionalUtils.java:110)
software.amazon.awssdk.utils.FunctionalUtils.invokeSafely (FunctionalUtils.java:136)
software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.lambda$tryGetOutputStream$0 (UrlConnectionHttpClient.java:331)
software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.getAndHandle100Bug (UrlConnectionHttpClient.java:367)
software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.tryGetOutputStream (UrlConnectionHttpClient.java:331)
software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.call (UrlConnectionHttpClient.java:308)
software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.call (UrlConnectionHttpClient.java:281)
software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe (MetricUtils.java:103)
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.executeHttpRequest (MakeHttpRequestStage.java:88)
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute (MakeHttpRequestStage.java:64)
software.amazon.awssdk.core.internal.http.pipeline.stages.MakeHttpRequestStage.execute (MakeHttpRequestStage.java:46)
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute (RequestPipelineBuilder.java:206)
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute (RequestPipelineBuilder.java:206)
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute (RequestPipelineBuilder.java:206)
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute (RequestPipelineBuilder.java:206)
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute (ApiCallAttemptTimeoutTrackingStage.java:74)
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute (ApiCallAttemptTimeoutTrackingStage.java:43)
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute (TimeoutExceptionHandlingStage.java:79)
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute (TimeoutExceptionHandlingStage.java:41)
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute (ApiCallAttemptMetricCollectionStage.java:55)
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute (ApiCallAttemptMetricCollectionStage.java:39)
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.executeRequest (RetryableStage.java:93)
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute (RetryableStage.java:56)
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute (RetryableStage.java:36)
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute (RequestPipelineBuilder.java:206)
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute (StreamManagingStage.java:53)
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute (StreamManagingStage.java:35)
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer (ApiCallTimeoutTrackingStage.java:82)
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute (ApiCallTimeoutTrackingStage.java:62)
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute (ApiCallTimeoutTrackingStage.java:43)
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute (ApiCallMetricCollectionStage.java:50)
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute (ApiCallMetricCollectionStage.java:32)
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute (RequestPipelineBuilder.java:206)
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute (RequestPipelineBuilder.java:206)
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute (ExecutionFailureExceptionReportingStage.java:37)
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute (ExecutionFailureExceptionReportingStage.java:26)
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute (AmazonSyncHttpClient.java:210)
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke (BaseSyncClientHandler.java:103)
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute (BaseSyncClientHandler.java:173)
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1 (BaseSyncClientHandler.java:80)
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess (BaseSyncClientHandler.java:182)
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute (BaseSyncClientHandler.java:74)
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute (SdkSyncClientHandler.java:45)
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute (AwsSyncClientHandler.java:53)
software.amazon.awssdk.services.s3.DefaultS3Client.putObject (DefaultS3Client.java:11883)
com.example.S3ObjectStoreKt$s3$1.write (S3ObjectStore.kt:26)
com.example.S3Store.store (S3Store.java:50)
com.example.S3Store.lambda$requestProcessed$0 (S3Store.java:42)
io.opentelemetry.javaagent.bootstrap.executors.ContextPropagatingRunnable.run (ContextPropagatingRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
java.lang.Thread.run (Thread.java:1583)

Reproduction Steps

import org.junit.Assert;
import org.junit.Test;
import org.testcontainers.containers.localstack.LocalStackContainer;
import org.testcontainers.utility.DockerImageName;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.http.ExecutableHttpRequest;
import software.amazon.awssdk.http.HttpExecuteRequest;
import software.amazon.awssdk.http.HttpExecuteResponse;
import software.amazon.awssdk.http.SdkHttpClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import java.io.IOException;
import java.io.UncheckedIOException;

public class S3ClientTest {
    @Test
    public void testS3Client() {
        TestHttpClient httpClient = new TestHttpClient();
        try (LocalStackContainer localStack = new LocalStackContainer(DockerImageName.parse("localstack/localstack"))
                .withServices(LocalStackContainer.Service.S3)) {
            localStack.start();
            try (S3Client s3Client = S3Client.builder()
                    .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create(localStack.getAccessKey(), localStack.getSecretKey())))
                    .endpointOverride(localStack.getEndpointOverride(LocalStackContainer.Service.S3))
                    .httpClient(httpClient)
                    .overrideConfiguration(o -> o.retryStrategy(b -> b.retryOnException(UncheckedIOException.class)))
                    .build()) {
                s3Client.putObject(
                        PutObjectRequest.builder().bucket("test-bucket").key("test-key").build(),
                        RequestBody.fromString("test-content")
                );
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        Assert.assertEquals(4, httpClient.count);
    }
}

class TestHttpClient implements SdkHttpClient {

    public int count = 0;

    @Override
    public ExecutableHttpRequest prepareRequest(HttpExecuteRequest httpExecuteRequest) {
        return new ExecutableHttpRequest() {

            @Override
            public void abort() {
            }

            @Override
            public HttpExecuteResponse call() throws IOException {
                count++;
                throw new UncheckedIOException(new IOException());
            }
        };
    }

    @Override
    public void close() {
    }
}

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.37.3 and 2.39.0

JDK version used

openjdk 21.0.3 2024-04-16 LTS

Operating System and version

alpine 3.19

Metadata

Metadata

Assignees

Labels

bugThis issue is a bug.needs-reviewThis issue or PR needs review from the team.p2This is a standard priority issue

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions