Rclone does not generate the expected md5chksum header

[audouts]

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/trying-to-force-single-part-with-s3-upload-cutoff/42950/2

What is the problem you are having with rclone?

When uploading a large file (1000M) to S3, I expected rclone to send it as a single-part, based on the --s3-upload-cutoff=4000M flag. However, the log shows that rclone started a multipart upload. Despite this, rclone does not generate the expected x-amz-meta-md5chksum header and md5sum reports an empty value.

If I use the default s3-upload-cutoff size (or set it below 1000M), rclone uses the same multipart process but correctly adds the x-amz-meta-md5chksum header.

What is your rclone version (output from rclone version)

rclone v1.64.2
- os/version: debian 11.1 (64 bit)
- os/kernel: 5.10.0-9-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

debian 11.1 (64 bit)

Which cloud storage system are you using? (e.g. Google Drive)

CDN77 (S3)

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone --s3-upload-cutoff=4000M copy largefile.rar remote:files

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

2023/11/19 02:20:12 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "-vv" "--s3-upload-cutoff=4000M" "copy" "largefile.rar" "remote:files"]
2023/11/19 02:20:12 DEBUG : Creating backend with remote "largefile.rar"
2023/11/19 02:20:12 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2023/11/19 02:20:12 DEBUG : fs cache: adding new entry for parent of "largefile.rar", "/home/user/files"
2023/11/19 02:20:12 DEBUG : Creating backend with remote "remote:files"
2023/11/19 02:20:12 DEBUG : remote: detected overridden config - adding "{J8TjX}" suffix to name
2023/11/19 02:20:12 DEBUG : Resolving service "s3" region "us-east-1"
2023/11/19 02:20:12 DEBUG : fs cache: renaming cache item "remote:files" to be canonical "remote{J8TjX}:files"
2023/11/19 02:20:12 DEBUG : largefile.rar: Need to transfer - File not found at Destination
2023/11/19 02:20:12 INFO  : S3 bucket files: Bucket "files" created with ACL "private"
2023/11/19 02:20:20 DEBUG : largefile.rar: open chunk writer: started multipart upload: 2~qG7BsI0hqHPphx3wupRhH9MCLN9RbGi
2023/11/19 02:20:20 DEBUG : largefile.rar: multi-thread copy: using backend concurrency of 4 instead of --multi-thread-streams 4
2023/11/19 02:20:20 DEBUG : largefile.rar: Starting multi-thread copy with 200 chunks of size 5Mi with 4 parallel streams
2023/11/19 02:20:20 DEBUG : largefile.rar: multi-thread copy: chunk 4/200 (15728640-20971520) size 5Mi starting
2023/11/19 02:20:20 DEBUG : largefile.rar: multi-thread copy: chunk 2/200 (5242880-10485760) size 5Mi starting
2023/11/19 02:20:20 DEBUG : largefile.rar: multi-thread copy: chunk 1/200 (0-5242880) size 5Mi starting
2023/11/19 02:20:20 DEBUG : largefile.rar: multi-thread copy: chunk 3/200 (10485760-15728640) size 5Mi starting
2023/11/19 02:21:09 DEBUG : largefile.rar: multipart upload wrote chunk 4 with 5242880 bytes and etag "6d36dd88386e0d014e76e5015ad43389"
2023/11/19 02:21:09 DEBUG : largefile.rar: multi-thread copy: chunk 4/200 (15728640-20971520) size 5Mi finished
2023/11/19 02:21:09 DEBUG : largefile.rar: multi-thread copy: chunk 5/200 (20971520-26214400) size 5Mi starting
2023/11/19 02:22:57 DEBUG : largefile.rar: multipart upload wrote chunk 3 with 5242880 bytes and etag "8bdb52b050fae024204a8c070cbc1901"
2023/11/19 02:22:57 DEBUG : largefile.rar: multi-thread copy: chunk 3/200 (10485760-15728640) size 5Mi finished
2023/11/19 02:22:57 DEBUG : largefile.rar: multi-thread copy: chunk 6/200 (26214400-31457280) size 5Mi starting

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[ncw]

I attempted to reproduce the md5chksum metadata being missing.

$ rclone test makefile 10M 10M
$ rclone copyto -vv 10M s3:rclone/10M.singlepart
$ rclone copyto -vv 10M --s3-upload-cutoff 0 s3:rclone/10M.multipart
...
2023/11/20 15:44:35 DEBUG : 10M.multipart: multipart upload: starting chunk 0 size 5Mi offset 0/10Mi
...
$ rclone copyto -vv 10M --s3-upload-cutoff 0 --multi-thread-cutoff 0 s3:rclone/10M.multithread
...
2023/11/20 15:44:56 DEBUG : 10M: Starting multi-thread copy with 2 chunks of size 5Mi with 2 parallel streams
...

And I can see here that all the uploads have md5sum

$ rclone md5sum s3:rclone
518ad6a602364206bd47e38eceda2e7c  10M.singlepart
518ad6a602364206bd47e38eceda2e7c  10M.multipart
518ad6a602364206bd47e38eceda2e7c  10M.multithread

And with this I can see that all the uploads have the correct metadata

rclone md5sum s3:rclone -vv --dump bodies

2023/11/20 15:50:58 DEBUG : HEAD /10M.multipart HTTP/1.1
Host: rclone.s3.eu-west-2.amazonaws.com
User-Agent: rclone/v1.65.0-beta.7507.64ec5709f
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231120T155058Z

2023/11/20 15:50:58 DEBUG : HTTP/1.1 200 OK
Content-Length: 10485760
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Mon, 20 Nov 2023 15:50:59 GMT
Etag: "7c5df926e31f6de31650b06c3ce237a8-2"
Last-Modified: Mon, 20 Nov 2023 15:44:36 GMT
Server: AmazonS3
X-Amz-Id-2: 8wmys/Ho9q7RqYCtH4Oug6Y2jwGqbizjj+919SkWgYF8FfgCbBIXpiV0YDYvClcoOZ+rCslkoRY=
X-Amz-Meta-Md5chksum: UYrWpgI2Qga9R+OOztoufA==
X-Amz-Meta-Mtime: 1700494997.548750577
X-Amz-Request-Id: ZPHM2W3ZDNMTCJDZ
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: null

2023/11/20 15:50:58 DEBUG : HEAD /10M.multithread HTTP/1.1
Host: rclone.s3.eu-west-2.amazonaws.com
User-Agent: rclone/v1.65.0-beta.7507.64ec5709f
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231120T155058Z

2023/11/20 15:50:58 DEBUG : HTTP/1.1 200 OK
Content-Length: 10485760
Accept-Ranges: bytes
Content-Type: application/octet-stream
Date: Mon, 20 Nov 2023 15:50:59 GMT
Etag: "7c5df926e31f6de31650b06c3ce237a8-2"
Last-Modified: Mon, 20 Nov 2023 15:44:57 GMT
Server: AmazonS3
X-Amz-Id-2: 3Rg/bzeUVpxG4P6GbdjVp7PyR5JR75UPtjN6XbUFn15hlZJCMuBsmuVHYJcswHTZaR2W6ecYCI8=
X-Amz-Meta-Md5chksum: UYrWpgI2Qga9R+OOztoufA==
X-Amz-Meta-Mtime: 1700494997.548750577
X-Amz-Request-Id: ZPHW316ECNMZ2J1G
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: null

So I've failed to reproduce the problem :-(

Can you give me a set of commands to reproduce the problem please?