Error with Custom Endpoints and S3 Client

[nevil2]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Upgrading to v3.183 and beyond makes createPresignedPost return wrong URL. e.g. https://ams3.digitaloceanspaces.com rather than correct url (as per v3.154) of https://ams3.digitaloceanspaces.com/{bucket}.

With v3.154, I have previously raised issue (#3885 still open) that the S3client gets corrupted and starts adding the bucket twice. e.g. https://{bucket}.ams3.digitaloceanspaces.com/{bucket}/filename

I tested yesterday uploading file to Digital Ocean spaces with both POST (with details from createPresignedPost) and PUT (with URL from getSignedUrl with PutObjectCommand) yesterday.

With v3.183, Digital Ocean rejected POST but upload with PUT worked correctly.
With v3.154, Digital Ocean upload with POST worked correctly, but with PUT, the file ended up within in the bucket in {bucket}/{directory}/{filename}.

Digital Ocean have confirmed that no changes have been made which might affect this behaviour.

It is clear that changes to the way endpoints are handled have been made to @AWS-SDK libraries between v3.154 and v3.183. I do not know whether the 'double' bucket issue is solved with v3.183 (as it appears randomly, sometimes after several weeks), but it clear that a new problem has been introduced.

SDK version number

@AWS-SDK @3.183.0+, client-s3, s3-presigned-post, s3-request-presigner

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

Node v18

Reproduction Steps

Create Client

const s3main = new S3Client({
  endpoint: 'https://ams3.digitaloceanspaces.com',
  region: 'ams3',
  credentials: {
    accessKeyId: CloudStorage.accessKeyId,
    secretAccessKey: CloudStorage.secretAccessKey
  }
})

Get URL and details for POST upload

const { url, fields } = await createPresignedPost(s3main, {
  Bucket: bucket,
  Key: filename,
  Expires: signedUrlExpireSeconds,
  Fields: { acl },
  Conditions: [
    { success_action_status: '201' },
    { 'Content-Type': contentType }
  ]
})

Get URL for PUT upload

const command = new PutObjectCommand({ Bucket, Key, ContentType })
const url = await getSignedUrl(s3main, command, { expiresIn: signedUrlExpireSeconds })

Observed Behavior

with v3.183.0 Get URL and details for POST upload:
url = 'https://ams3.digitaloceanspaces.com'

with v3.154 Get URL for PUT upload:
url = https://{bucket}.ams3.digitaloceanspaces.com/{bucket}/filename

Expected Behavior

with v3.154.0 Get URL and details for POST upload:
url = 'https://ams3.digitaloceanspaces.com/{bucket}'

with v3.183 Get URL for PUT upload:
url = https://{bucket}.ams3.digitaloceanspaces.com/filename

Note that because v3.183 Get URL for PUT upload worked correctly once, it does not guarantee that error will not develop over time.

Possible Solution

Building workaround (recreate client, use PUT vs POST, hard code endpoint for POST, etc) into my code is pointless if behaviour keeps changing with new versions and hence code keeps failing.

Additional Information/Context

No response

[yenfryherrerafeliz]

Hi @nevil2, thanks for opening this issue. I can confirm that there is a problem when using custom endpoints along with createPresignedPost. Basically, I could find one condition that would make the bucket to be added at the custom url, but it will never be met because it validates that the endpoint is not a custom one, here, which is not our case here.

if (endpoint && !client.config.isCustomEndpoint && !isEndpointV2) {
endpoint.path = /${Bucket};
}

I will mark this issue to be reviewed so we can address it further.

I also want to mention that is latest version of the SDK you need to pass an extract parameter "forcePathStyle", that will decide how the url will be formed, when using custom endpoints.
For example, when forcePathStyle is set to true, the url is formed in the following way:

https://customendpoint.com/bucketname

And when set to false it will be:

https://bucketname.customendpoint.com

This parameter is given when constructing the client:

const s3main = new S3Client({
  endpoint: 'https://ams3.digitaloceanspaces.com',
  forcePathStyle: true | false,
  region: 'ams3',
  credentials: {
    accessKeyId: CloudStorage.accessKeyId,
    secretAccessKey: CloudStorage.secretAccessKey
  }
})

Steps to reproduce:

• Create an empty project npm init
• Install the dependencies:
  • npm i @aws-sdk/client-s3
    -npm i @aws-sdk/s3-presigned-post
• Create a js file, example: src/index.js
• Copy and paste the following code:

import {
    S3Client
} from "@aws-sdk/client-s3";
import {
    createPresignedPost
} from "@aws-sdk/s3-presigned-post";


const client = new S3Client({
    region: 'us-east-2',
    endpoint: 'https://customendpoint.com/',
    forcePathStyle: true
});
const signedUrl = await createPresignedPost(client, {
    Bucket: process.env.TEST_BUCKET,
    Key: process.env.TEST_KEY,
    Expires: 600,
    Fields: {
        field1: 'Value1'
    }
})
console.log(signedUrl)

• Run the code node src/index.js
• And in the output you will see that the bucket is not appended to the specified url as endpoint.

Thanks!

[nevil2]

Great you've found issue.
I stopped using createPresignedPost in my applications and used getSignedUrl with PUT instead.