S3 CRT Client GetObject Request with Invalid Range crashes JVM with SIGSEGV

[dwragge]

Describe the bug

When using the CRT S3 Client, and providing a byte range for a GetObject request, if that byte range is invalid (e.g. start > end), the JVM crashes with a SIGSEGV, with the stack frame inside libjvm, making tracing down the bug more challenging than it needs to be.

Expected Behavior

No segfault, some kind of error returned / thrown back to the caller.

Current Behavior

The JVM segfaults with the following logs:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00000001067ab034, pid=23410, tid=10243
#
# JRE version: OpenJDK Runtime Environment Zulu17.32+13-CA (17.0.2+8) (build 17.0.2+8-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.32+13-CA (17.0.2+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# V  [libjvm.dylib+0x32b034]  AccessInternal::PostRuntimeDispatch<G1BarrierSet::AccessBarrier<548964ull, G1BarrierSet>, (AccessInternal::BarrierType)0, 548964ull>::oop_access_barrier(void*, oopDesc*)+0x14
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

Enabling CRT logging, the issue becomes clearer:

[DEBUG] [2024-06-10T15:35:32Z] [000000016f0db000] [task-scheduler] - id=0x1386ead08: Scheduling gather_statistics task for future execution at time 424558118885916
[INFO] [2024-06-10T15:35:32Z] [000000016b767000] [S3Client] - id=0x10f844f50 Initiating making of meta request
[ERROR] [2024-06-10T15:35:32Z] [000000016b767000] [S3MetaRequest] - id=0x1386f6280 Could not parse Range header for Auto-Ranged-Get Meta Request.
[DEBUG] [2024-06-10T15:35:32Z] [000000016b767000] [S3MetaRequest] - id=0x1386f6280 Cleaning up meta request
[ERROR] [2024-06-10T15:35:32Z] [000000016b767000] [S3Client] - id=0x10f844f50: Could not create new meta request.

Full hs_err thread stack trace:

---------------  T H R E A D  ---------------

Current thread (0x0000000124008a00):  JavaThread "main" [_thread_in_vm, id=10243, stack(0x000000016af28000,0x000000016b12b000)]

Stack: [0x000000016af28000,0x000000016b12b000],  sp=0x000000016b128e40,  free space=2051k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.dylib+0x32b034]  AccessInternal::PostRuntimeDispatch<G1BarrierSet::AccessBarrier<548964ull, G1BarrierSet>, (AccessInternal::BarrierType)0, 548964ull>::oop_access_barrier(void*, oopDesc*)+0x14
V  [libjvm.dylib+0x509dc8]  JNIHandles::destroy_global(_jobject*)+0x24
V  [libjvm.dylib+0x4c60d0]  jni_DeleteGlobalRef+0xd8
C  [AWSCRT_15933358566040015791libaws-crt-jni.dylib+0x22ec4]  Java_software_amazon_awssdk_crt_s3_S3Client_s3ClientMakeMetaRequest+0x628
j  software.amazon.awssdk.crt.s3.S3Client.s3ClientMakeMetaRequest(JLsoftware/amazon/awssdk/crt/s3/S3MetaRequest;[BIIIZ[I[BLsoftware/amazon/awssdk/crt/http/HttpRequestBodyStream;[BLsoftware/amazon/awssdk/crt/auth/signing/AwsSigningConfig;Lsoftware/amazon/awssdk/crt/s3/S3MetaRequestResponseHandlerNativeAdapter;[BLsoftware/amazon/awssdk/crt/s3/ResumeToken;)J+0
j  software.amazon.awssdk.crt.s3.S3Client.makeMetaRequest(Lsoftware/amazon/awssdk/crt/s3/S3MetaRequestOptions;)Lsoftware/amazon/awssdk/crt/s3/S3MetaRequest;+295
j  software.amazon.awssdk.services.s3.internal.crt.S3CrtAsyncHttpClient.execute(Lsoftware/amazon/awssdk/http/async/AsyncExecuteRequest;)Ljava/util/concurrent/CompletableFuture;+215
j  software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.doExecuteHttpRequest(Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;Lsoftware/amazon/awssdk/http/async/AsyncExecuteRequest$Builder;Lsoftware/amazon/awssdk/core/internal/http/TransformingAsyncResponseHandler;)Ljava/util/concurrent/CompletableFuture;+45
j  software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.executeHttpRequest(Lsoftware/amazon/awssdk/http/SdkHttpFullRequest;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/util/concurrent/CompletableFuture;+147
j  software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$1(Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;Ljava/util/concurrent/CompletableFuture;Lsoftware/amazon/awssdk/http/SdkHttpFullRequest;)V+3
j  software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$$Lambda$443+0x0000000800d4e238.accept(Ljava/lang/Object;)V+16
j  java.util.concurrent.CompletableFuture.uniAcceptNow(Ljava/lang/Object;Ljava/util/concurrent/Executor;Ljava/util/function/Consumer;)Ljava/util/concurrent/CompletableFuture;+73 java.base@17.0.2
j  java.util.concurrent.CompletableFuture.uniAcceptStage(Ljava/util/concurrent/Executor;Ljava/util/function/Consumer;)Ljava/util/concurrent/CompletableFuture;+25 java.base@17.0.2
j  java.util.concurrent.CompletableFuture.thenAccept(Ljava/util/function/Consumer;)Ljava/util/concurrent/CompletableFuture;+3 java.base@17.0.2
j  software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(Ljava/util/concurrent/CompletableFuture;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/util/concurrent/CompletableFuture;+29
j  software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+6
j  software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+16
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallAttemptMetricCollectionStage.execute(Lsoftware/amazon/awssdk/http/SdkHttpFullRequest;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/util/concurrent/CompletableFuture;+26
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallAttemptMetricCollectionStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+6
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.attemptExecute(Ljava/util/concurrent/CompletableFuture;)V+25
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(Ljava/util/concurrent/CompletableFuture;)V+181
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.execute()Ljava/util/concurrent/CompletableFuture;+10
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(Lsoftware/amazon/awssdk/http/SdkHttpFullRequest;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/util/concurrent/CompletableFuture;+11
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+6
j  software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+10
j  software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+10
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(Lsoftware/amazon/awssdk/http/SdkHttpFullRequest;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/util/concurrent/CompletableFuture;+6
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+6
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(Lsoftware/amazon/awssdk/http/SdkHttpFullRequest;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/util/concurrent/CompletableFuture;+67
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+6
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(Lsoftware/amazon/awssdk/http/SdkHttpFullRequest;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/util/concurrent/CompletableFuture;+33
j  software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+6
j  software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(Ljava/lang/Object;Lsoftware/amazon/awssdk/core/internal/http/RequestExecutionContext;)Ljava/lang/Object;+16
j  software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(Lsoftware/amazon/awssdk/core/internal/http/TransformingAsyncResponseHandler;)Ljava/util/concurrent/CompletableFuture;+202
j  software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.invoke(Lsoftware/amazon/awssdk/core/client/config/SdkClientConfiguration;Lsoftware/amazon/awssdk/http/SdkHttpFullRequest;Lsoftware/amazon/awssdk/core/async/AsyncRequestBody;Lsoftware/amazon/awssdk/core/SdkRequest;Lsoftware/amazon/awssdk/core/http/ExecutionContext;Lsoftware/amazon/awssdk/core/internal/http/TransformingAsyncResponseHandler;)Ljava/util/concurrent/CompletableFuture;+46
j  software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(Lsoftware/amazon/awssdk/core/client/handler/ClientExecutionParams;Lsoftware/amazon/awssdk/core/http/ExecutionContext;Lsoftware/amazon/awssdk/core/internal/http/TransformingAsyncResponseHandler;)Ljava/util/concurrent/CompletableFuture;+146
j  software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$3(Lsoftware/amazon/awssdk/core/client/handler/ClientExecutionParams;Lsoftware/amazon/awssdk/core/async/AsyncResponseTransformer;)Ljava/util/concurrent/CompletableFuture;+131
j  software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler$$Lambda$290+0x0000000800d18448.get()Ljava/lang/Object;+12
j  software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(Lsoftware/amazon/awssdk/core/client/handler/ClientExecutionParams;Ljava/util/function/Supplier;)Ljava/util/concurrent/CompletableFuture;+1
j  software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(Lsoftware/amazon/awssdk/core/client/handler/ClientExecutionParams;Lsoftware/amazon/awssdk/core/async/AsyncResponseTransformer;)Ljava/util/concurrent/CompletableFuture;+10
j  software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(Lsoftware/amazon/awssdk/core/client/handler/ClientExecutionParams;Lsoftware/amazon/awssdk/core/async/AsyncResponseTransformer;)Ljava/util/concurrent/CompletableFuture;+3
j  software.amazon.awssdk.services.s3.DefaultS3AsyncClient.getObject(Lsoftware/amazon/awssdk/services/s3/model/GetObjectRequest;Lsoftware/amazon/awssdk/core/async/AsyncResponseTransformer;)Ljava/util/concurrent/CompletableFuture;+253
j  software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.lambda$getObject$43(Lsoftware/amazon/awssdk/core/async/AsyncResponseTransformer;Lsoftware/amazon/awssdk/services/s3/model/GetObjectRequest;)Ljava/util/concurrent/CompletableFuture;+6
j  software.amazon.awssdk.services.s3.DelegatingS3AsyncClient$$Lambda$286+0x0000000800d14538.apply(Ljava/lang/Object;)Ljava/lang/Object;+12
j  software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.invokeOperation(Lsoftware/amazon/awssdk/services/s3/model/S3Request;Ljava/util/function/Function;)Ljava/util/concurrent/CompletableFuture;+2
j  software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.getObject(Lsoftware/amazon/awssdk/services/s3/model/GetObjectRequest;Lsoftware/amazon/awssdk/core/async/AsyncResponseTransformer;)Ljava/util/concurrent/CompletableFuture;+9
j  com.github.dwragge.Main.main([Ljava/lang/String;)V+72
v  ~StubRoutines::call_stub
V  [libjvm.dylib+0x46b270]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x38c
V  [libjvm.dylib+0x4cfa64]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, JavaThread*)+0x12c
V  [libjvm.dylib+0x4d30f8]  jni_CallStaticVoidMethod+0x130
C  [libjli.dylib+0x5378]  JavaMain+0x9d4
C  [libjli.dylib+0x76e8]  ThreadJavaMain+0xc
C  [libsystem_pthread.dylib+0x7034]  _pthread_start+0x88

Reproduction Steps

This can be trivially reproduced with

S3AsyncClient s3 = S3AsyncClient.crtBuilder().build();

CompletableFuture<ResponseBytes<GetObjectResponse>> fut = s3.getObject(GetObjectRequest.builder()
        .bucket(bucket)
        .key(key)
        .range("bytes=100-99")
        .build(), AsyncResponseTransformer.toBytes());

Possible Solution

No response

Additional Information/Context

aws-crt version 0.29.19

AWS Java SDK version used

2.25.69

JDK version used

17.0.2, 21.0.3

Operating System and version

Host: "MacBookPro18,2" arm64 1 MHz, 10 cores, 64G, Darwin 23.1.0, macOS 14.1.1 (23B81)

[debora-ito]

Thank you for the report @dwragge, we'll take a look.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[debora-ito]

@dwragge the crash was fixed in the AWS CRT version 0.30.0, the upgrade will be included in the next Java SDK release.