s3 head object, erroneously modified object name

[luoyanpeng-bytedancer]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

s3 head object：If the object name starts with"/+ bucket name", the SDK will remove the bucket name from the object name in the path of the URL, causing the request to return 404.

code :
s3@v1.73.2 internal/customizations/update_endpoint.go
func moveBucketNameToHost(u *url.URL, bucket string) { u.Host = bucket + "." + u.Host removeBucketFromPath(u, bucket) }

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Object names should not be modified

Current Behavior

If the object name starts with"/+ bucket name", the SDK will remove the bucket name from the object name in the path of the URL

Reproduction Steps

Head An object prefixed with "/+ bucket name"

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.33.0
github.com/aws/aws-sdk-go-v2/config v1.5.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.73.2
github.com/aws/smithy-go v1.22.1

Compiler and Version used

go version go1.21.7 darwin/amd64

Operating System and version

macOS 14.0 (23A344)

[adev-code]

Hello @luoyanpeng-bytedancer, thanks for reaching out. For doing head object operations on S3 key names that contains characters such as "/" and "+", you would need to encode those characters as they are identified as special characters for S3 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html). So for the request, it would need to have the encoded version of the characters. If you have any questions, please do let me know. Thanks.

[luoyanpeng-bytedancer]

Hello @luoyanpeng-bytedancer, thanks for reaching out. For doing head object operations on S3 key names that contains characters such as "/" and "+", you would need to encode those characters as they are identified as special characters for S3 (https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html). So for the request, it would need to have the encoded version of the characters. If you have any questions, please do let me know. Thanks.

Why boto3 can not encode "/", and why only "/+ bucket name" has this problem

[adev-code]

The way I have replicated on my side is that:

1. Uploaded a sample object inside a prefix. From local machine, it looked like this: "/ +bucket-name/sample.png". So, the name of the folder actually has "/" then followed by a space then a "+".
2. Then uploaded it using the AWS Go SDK to S3. The S3 URI appeared as follows: s3://bucket-name// +bucket-name/sample.png . As shown, the is a double slash added.
3. Then I ran the Head Object operation and got 200OK. I did not get the 404. Not that I have a / and a space and a + on the name of the object key and it worked fine.

My snippet of code for Head Object:

package main

import (
    "context"
    "fmt"
    "log"
    "os"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

const (
    bucketName = "bucket-name"               
    objectKey  = "/ +bucket-name/sample.png" 
    region     = "us-east-1"                                     
)

func main() {
    ctx := context.Background()

    logFile, err := os.OpenFile("s3_head_object_go.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0644)
    if err != nil {
        log.Fatalf("Failed to open log file: %v", err)
    }
    log.SetOutput(logFile) 

    cfg, err := config.LoadDefaultConfig(ctx,
        config.WithRegion(region),
        config.WithClientLogMode(aws.LogRequest|aws.LogResponse),
    )
    if err != nil {
        log.Fatalf("failed to load AWS config: %v", err)
    }

    client := s3.NewFromConfig(cfg)

    resp, err := client.HeadObject(ctx, &s3.HeadObjectInput{
        Bucket: aws.String(bucketName),
        Key:    aws.String(objectKey),
    })
    if err != nil {
        log.Printf("Error occurred: %v\n", err)
        return
    }

    fmt.Println("Metadata:", resp.Metadata)
    fmt.Println("Content Length:", resp.ContentLength, "bytes")
    fmt.Println("Last Modified:", resp.LastModified)
    fmt.Println("ETag:", *resp.ETag)
    fmt.Println("Content Type:", *resp.ContentType)

    log.Printf("HeadObject Response: %+v\n", resp)
}

From the above,
A) How are you uploading the object with the prefix ? Is it straight to the console via drag and drop or via the SDK?
B) When you upload the object on S3, wanted to clarify if the prefix actually have the forward slash in its name?
C) On your code, does the object key match to the S3 URI that is shown on your console?

If you have any other questions, please let me know. Thanks.