Encountered AWS_ERROR_NO_PERMISSION when I tried to use CRT in AWS CLI to sync files from local to S3

[jacky96623]

Describe the bug

I am trying to use aws s3 sync to backup files from my servers to S3. To utilize the bandwidth available, I configure the CLI tool to with preferred_transfer_client = crt. From initial testing, the transfer speed can go up to 600MB/s so I decide to adopt this client for the actual backup job.

I ran some command similar to the following:

aws s3 sync <some-readonly-file> s3://backup-store

However, I saw the following error:

upload failed: <some-readonly-file> to s3://backup-store/<some-readonly-file> 43 (AWS_ERROR_NO_PERMISSION): User does not have permission to perform the requested action.

I then re-run the cmd with debug mode on and saw the following detailed error log:

[ERROR] [2023-07-04T03:23:54Z] [00007f2a05976b80] [common-io] - static: Failed to open file. path:'/data/<some-readonly-file>' mode:'r+b' errno:13 aws-error:43(AWS_ERROR_NO_PERMISSION)

I suspect it is due to the codes in https://github.com/awslabs/aws-c-io/blob/1f9a085027c5e0f3e0c0e2a9fbe685f12af8105d/source/stream.c#L290-L293 and the permission of my file, which looks like the following:

ll <some-readonly-file>
-r--r--r-- 1 root root 11236 Jul  2 01:56 <some-readonly-file>

Expected Behavior

I expect the file to be able to upload as if I use the default transfer client, even though I have only read access to the files.

Current Behavior

Error AWS_ERROR_NO_PERMISSION is encountered.

Reproduction Steps

• This reproduction steps assume a default AWS profile with suitable S3 access has been configured.

touch readonly-empty-file
chmod 0444 readonly-empty-file

aws configure set default.s3.preferred_transfer_client crt
aws s3 cp readonly-empty-file s3://<some writable bucket>

Possible Solution

Open the file to be uploaded with rb access instead of r+b access.

Additional Information/Context

I hope that I do not open the issue in the wrong repository.

aws-crt-cpp version used

Unknown

Compiler and version used

Unknown

Operating System and version

Ubuntu 22.04.2 LTS

[yasminetalby]

Hello @jacky96623 ,

Thank you very much for your submission.
Is the behavior successful using the default client?

In general for issues using the AWS CLI, I would recommend to submitting an issue to the aws-cli repository.

Best regards,

Yasmine

[jacky96623]

Hi @yasminetalby , thanks for the reply. Actually I tested the same file with the default client and the upload is successful.

[TingDaoK]

We update the underlying C code and updated crt-python to pick up the code.
https://github.com/awslabs/aws-crt-python/releases/tag/v0.16.24

The new release for awscrt python should be available soon. We will also contact aws-cli to pick up the latest change. Thanks for reporting the issue!

[TingDaoK]

aws/aws-cli#8029

[yasminetalby]

Hello,

I am closing this issue as there are no further action from the CRT team on this issue. Let me know if you have any further question/inquiries.

Best regards,

Yasmine