S3Client.headObject does not throw NoSuchBucketException/NoSucheKeyException when bucket doesn't exist

[disordered]

Describe the bug

S3Client.headObject(request) will throw NoSuchKeyException when key doesn't exist, but will result in the following when the bucket doesn't exist:

software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 301, Request ID: null, Extended Request ID: frf1/X60l1EYDa23MUX8Xh3QH4k41SWwPZcYolsrZ4P3WtXbzNphkJC4opC+t4ZhdmSxwEo2Pyk=)

Expected Behavior

It should either throw NoSuchKeyException or NoSuchBucketException like S3Client.headBucket(request) does.

Current Behavior

Missing bucket results in generic exception:

software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 301, Request ID: null, Extended Request ID: frf1/X60l1EYDa23MUX8Xh3QH4k41SWwPZcYolsrZ4P3WtXbzNphkJC4opC+t4ZhdmSxwEo2Pyk=)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106)
	at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84)
  | => cat software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94)
	at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$5(BaseClientHandler.java:229)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
	at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162)
	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91)
	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
	at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:4716)

Steps to Reproduce

request = HeadObjectRequest.builder().bucket("bucket-that-does-not-exist").key("does-not-matter").build()
S3Client.create().headObject(request)

Possible Solution

Context

I'm trying to implement exist checks, so better solution would be to provide objectExists/bucketExists.

Your Environment

• AWS Java SDK version used: Tested with 2.13.71
• JDK version used: OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_265-b01)
• Operating System and version: OSX 10.15.5.

[debora-ito]

Hi @disordered, the exception shows HTTP status code 301 - Moved Permanently, which is a more specific response case. It's returned when the bucket do exists but in a different region than the region you're using to send the requests. And I think you'll only see this response code when you use S3 path-style requests.

When I run the sample code you provided with a bucket that does not exists, using virtual hosted-style, I get:

software.amazon.awssdk.services.s3.model.NoSuchKeyException: The specified key does not exist. (Service: S3, Status Code: 404, Request ID: xxx, Extended Request ID: null)

Do you think that could be the case why you're seeing 301?

[disordered]

From what I can tell the default configuration disables path-style requests and I don't explicitly enable it.
Hmm, turns out in my test the bucket name I was actually using no-such-bucket instead of bucket-that-does-not-exist. Both buckets don't exist in either of the regions, but I get 301 with no-such-bucket and I do get NoSuchKeyException for the other one. CLI, however, returns 403 for no-such-bucket:

aws s3api head-bucket --bucket no-such-bucket 

An error occurred (403) when calling the HeadBucket operation: Forbidden

aws s3api head-object --bucket no-such-bucket --key no-such-key

An error occurred (403) when calling the HeadObject operation: Forbidden

I guess I accidentally picked a wrong bucket name to test against.
Thanks for getting back.

[debora-ito]

Did some investigation: I ran the CLI command aws s3api head-bucket --bucket no-such-bucket in debug mode and this popped up in the CLI logs:

MainThread - botocore.utils - DEBUG - S3 client configured for region us-west-2 but the bucket no-such-bucket is in region us-east-1; Please configure the proper region to avoid multiple unnecessary redirects and signing attempts.

Then, CLI automatically updated the host to https://no-such-bucket.s3.us-east-1.amazonaws.com using the us-east-1 region and sent a new request, and the response was 403 Forbidden.

Bucket names are globally unique, meaning that after a bucket is created the name of that bucket cannot be used by another AWS account in any AWS Region until the bucket is deleted. You are seeing 403 Forbidden in CLI because the no-such-bucket bucket actually exists, and you don't have permissions to access it.

Let me know if you have follow up questions.