CLI re-uploads random unchanged files

[KristofMorva]

Describe the bug

Hey, I've seen several tickets for this issue, but none of the solutions seemed to be fitting, so my hope is that it's a regression in a recent version.

While syncing from a local folder to a remote S3-compatible server (Cloudflare R2), some seemingly random files keep getting re-uploaded.

• The files are always the same
• They are re-uploaded even if I start the next sync right after the previous one
• I've tried size-only check, didn't help
• Nothing unique about the files, average file name length, average extension, no special characters, average size
• I've also tried syncing a small sub-folder, same files are getting uploaded
• I've tried removing the files from the bucket, and then re-upload
• I've verified that the files do not change (modification date is old locally)

Expected Behavior

When I sync right after syncing with the same command, I expect sync to do nothing.

Current Behavior

It instead uploads some of the files again and again.

For example, Binaries\DotNET\IOS\DeploymentInterface.dll is re-uploaded, while Binaries\DotNET\IOS\DeploymentInterface.pdb is perfectly fine (same folder, file name besides the extension, etc).

Please note that the whole repository is 150.000+ files, dll extension is very common, and only 16 specific files are re-uploaded.
What is common about the files:

• They share a common parent (Engine\Binaries), but not a common sub-folder (and there are loads of other files of the same type in this folder, so it's not directly a problem with the whole folder)
• They are all binaries (exe, dll, or extension-less linux binaries) (but there are a lot of the same types elsewhere)

The debug log says file does not exist at destination. Is there any way I can debug further, i.e. why it thinks that it does not exist? Like some Super Duper Deep Debug Mode? It is clearly in the XML it received.

Here's a log excerpt (deliberately didn't obfuscate any file names):

2024-01-15 20:43:00,230 - MainThread - botocore.hooks - DEBUG - Event request-created.s3.ListObjectsV2: calling handler <function signal_transferring at 0x000001A81BAC7060>
2024-01-15 20:43:00,230 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=GET, url=https://my_account_id.r2.cloudflarestorage.com/bucket-name?list-type=2&prefix=Binaries%2FDotNET%2FIOS%2F&encoding-type=url, headers={'User-Agent': b'aws-cli/2.15.10 Python/3.11.6 Windows/10 exe/AMD64 prompt/off command/s3.sync', 'X-Amz-Date': b'20240115T194300Z', 'X-Amz-Content-SHA256': b'hash', 'Authorization': b'AWS4-HMAC-SHA256 Credential=hash/20240115/auto/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=hash'}>
2024-01-15 20:43:00,230 - MainThread - botocore.httpsession - DEBUG - Certificate path: C:\Program Files\Amazon\AWSCLIV2\awscli\botocore\cacert.pem
2024-01-15 20:43:00,231 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): my_client_id.r2.cloudflarestorage.com:443
2024-01-15 20:43:00,668 - MainThread - urllib3.connectionpool - DEBUG - https://my_client_id.r2.cloudflarestorage.com:443 "GET /bucket-name?list-type=2&prefix=Binaries%2FDotNET%2FIOS%2F&encoding-type=url HTTP/1.1" 200 4500
2024-01-15 20:43:00,670 - MainThread - botocore.parsers - DEBUG - Response headers: {'Date': 'Mon, 15 Jan 2024 19:43:00 GMT', 'Content-Type': 'application/xml', 'Content-Length': '4500', 'Connection': 'keep-alive', 'Vary': 'Accept-Encoding', 'Server': 'cloudflare', 'CF-RAY': '84609aebad913492-WAW'}
2024-01-15 20:43:00,675 - MainThread - botocore.parsers - DEBUG - Response body:
b'<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bucket-name</Name><Contents><Key>Binaries%2FDotNET%2FIOS%2FDeploymentInterface.dll.config</Key><Size>266</Size><LastModified>2024-01-14T14:38:32.185Z</LastModified><ETag>&quot;264d460f706ac9c712df1e35eaf914b1&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDeploymentInterface.dll</Key><Size>5120</Size><LastModified>2024-01-15T19:41:29.735Z</LastModified><ETag>&quot;b7e6a5e7034be8a1398f8685d8d2e522&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDeploymentInterface.pdb</Key><Size>15872</Size><LastModified>2024-01-14T14:38:32.284Z</LastModified><ETag>&quot;179fff9da907e9266c1ecbcaa0e7d79b&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDeploymentServer.exe.config</Key><Size>271</Size><LastModified>2024-01-14T14:38:32.720Z</LastModified><ETag>&quot;d55e1b974bdae70e577d89fa272d782e&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDeploymentServer.exe</Key><Size>38912</Size><LastModified>2024-01-15T19:41:30.174Z</LastModified><ETag>&quot;63c1ed8d967c7fa8d5c55eb416d3427a&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDeploymentServer.pdb</Key><Size>69120</Size><LastModified>2024-01-14T14:38:32.957Z</LastModified><ETag>&quot;3db47764d6cb744dcd2290bb94c7fe47&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDeploymentServerLauncher.exe.config</Key><Size>474</Size><LastModified>2024-01-14T14:38:32.761Z</LastModified><ETag>&quot;2088a9eac8c92c66da8e9852a7743566&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDeploymentServerLauncher.exe</Key><Size>6656</Size><LastModified>2024-01-15T19:41:30.178Z</LastModified><ETag>&quot;d3a54c0dba9efa0d8e94e69165926ec7&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDeploymentServerLauncher.pdb</Key><Size>15872</Size><LastModified>2024-01-14T14:38:32.734Z</LastModified><ETag>&quot;0d0a32d802adb7ac28ad481e0d6dbcab&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDotNETUtilities.dll</Key><Size>187904</Size><LastModified>2024-01-14T14:38:33.127Z</LastModified><ETag>&quot;a4f6a9f85cabee4b79295ebc80662816&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FDotNETUtilities.pdb</Key><Size>560640</Size><LastModified>2024-01-14T14:38:34.621Z</LastModified><ETag>&quot;f155a858ff616b6547237f11ff10f729&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FIPhonePackager.exe.config</Key><Size>274</Size><LastModified>2024-01-14T14:38:33.178Z</LastModified><ETag>&quot;674f78ebbbcda8d78059c9d8b170831d&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FIPhonePackager.exe</Key><Size>254464</Size><LastModified>2024-01-15T19:41:30.346Z</LastModified><ETag>&quot;e500d124f21c84e7dd1dccdd18638cdb&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FIPhonePackager.pdb</Key><Size>355840</Size><LastModified>2024-01-14T14:38:33.528Z</LastModified><ETag>&quot;9d6b589e420864b13a186674fe440e0d&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FMobileDeviceInterface.dll.config</Key><Size>278</Size><LastModified>2024-01-14T14:38:33.399Z</LastModified><ETag>&quot;8089043b8cbc71f695e563cb3c850311&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FMobileDeviceInterface.dll</Key><Size>76800</Size><LastModified>2024-01-15T19:41:30.237Z</LastModified><ETag>&quot;0d5a98253b9a640c9d76c08b691f8a2e&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>Binaries%2FDotNET%2FIOS%2FMobileDeviceInterface.pdb</Key><Size>187904</Size><LastModified>2024-01-14T14:38:33.587Z</LastModified><ETag>&quot;f7de8908d188a2b46f3c0ad90ae80067&quot;</ETag><StorageClass>STANDARD</StorageClass></Contents><IsTruncated>false</IsTruncated><Prefix>Binaries%2FDotNET%2FIOS%2F</Prefix><MaxKeys>1000</MaxKeys><KeyCount>17</KeyCount><EncodingType>url</EncodingType></ListBucketResult>'
2024-01-15 20:43:00,681 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x000001A81D76EF50>>
2024-01-15 20:43:00,681 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2024-01-15 20:43:00,682 - MainThread - botocore.hooks - DEBUG - Event needs-retry.s3.ListObjectsV2: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x000001A81D76E8D0>>
2024-01-15 20:43:00,686 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function decode_list_object_v2 at 0x000001A81A78F1A0>
2024-01-15 20:43:00,686 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <function enhance_error_msg at 0x000001A81BDF2700>
2024-01-15 20:43:00,688 - MainThread - botocore.hooks - DEBUG - Event after-call.s3.ListObjectsV2: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x000001A81D76EA10>>
2024-01-15 20:43:00,689 - MainThread - awscli.customizations.s3.syncstrategy.base - DEBUG - syncing: C:\Users\krist\Documents\RapaxEngine\Build\Windows\Engine\Binaries\DotNET\IOS\DeploymentInterface.dll -> bucket-name/Binaries/DotNET/IOS/DeploymentInterface.dll, file does not exist at destination

Reproduction Steps

The command I run is:

aws s3 sync 'C:\Path\Without\Spaces\Or\Anything\Engine' "s3://bucket-name/" --size-only --debug

Possible Solution

Maybe some more detailed debugging logs could shed some light as to why it fails to find these files in the XML?

Additional Information/Context

While testing my script, I've been terminating AWS CLI very frequently, and the mentioned Engine\Binaries folder is on the topmost of the root folder lexically. Although the problematic files do not follow each other alphabetically, is it possible for something to be corrupted due to the interruptions? Or am I being blind and the problem is obvious?

CLI version used

2.15.10

Environment details (OS name and version, etc.)

Windows 10.0.19045

[KristofMorva]

Just to give one more insight, I have tried with --dryrun and it seems that there are more files it wants to upload, the AWS CLI just seems to exit after some upload: with Shutdown request received in result processing thread, shutting down result thread.. There are no errors preceding it, so I'm a little confused as to why it quits, and why it tries to re-upload the files that are already in the S3 bucket. I have attached the sanitized log in case somebody can see what I can't: log.txt

Also, for reference, rclone sync C:\Same\Path remote:bucket-name --size-only --fast-list --progress --config="" correctly does not try to upload anything (as everything is up to date on the server).

[RyanFitzSimmonsAK]

Hi @KristofMorva, thanks for reaching out. I'm looking into this issue, can you confirm whether the S3 bucket you're using is a directory bucket or not?

[KristofMorva]

Hi @KristofMorva, thanks for reaching out. I'm looking into this issue, can you confirm whether the S3 bucket you're using is a directory bucket or not?

Hey! Sorry, my S3 lingo is pretty bad (it's the first time I'm using it), but my bucket does have folders (not all files are dumped in the root) if that's what you mean :)

[RyanFitzSimmonsAK]

No worries! By directory bucket, I'm referring to these. They're listed separately in the S3 console, and all objects in those buckets use the S3 Express One Zone storage class.

[KristofMorva]

No worries! By directory bucket, I'm referring to these. They're listed separately in the S3 console, and all objects in those buckets use the S3 Express One Zone storage class.

Ah okay! As mentioned in my post, I'm using Cloudflare R2, and sadly I'm unsure what's the underlying architecture there. It does not seem to have different options, it has a single-trier pricing (storage + read/write transactions).

[RyanFitzSimmonsAK]

Although the problematic files do not follow each other alphabetically

If you use aws s3api list-objects-v2 on the bucket, is the output in alphabetical order? I suspect this might be the same issue as #8470, which I'm working on.

[KristofMorva]

Although the problematic files do not follow each other alphabetically

If you use aws s3api list-objects-v2 on the bucket, is the output in alphabetical order? I suspect this might be the same issue as #8470, which I'm working on.

Yes, they do appear to be in alphabetical order.
Thank you for the quick follow-up! :)

[kellertk]

Hi there! I'm glad you were able to get this working, but as a reminder, the AWS SDKs and the AWS CLI are designed for and intended for use with AWS. We cannot support the CLI or the SDK in non-AWS environments.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.