Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

moveto hangs on big files with s3 (provider: minio) #7144

Closed
pasztorl opened this issue Jul 17, 2023 · 3 comments
Closed

moveto hangs on big files with s3 (provider: minio) #7144

pasztorl opened this issue Jul 17, 2023 · 3 comments

Comments

@pasztorl
Copy link

The associated forum post URL from https://forum.rclone.org

What is the problem you are having with rclone?

I can use moveto with small files, but with larger (10G+) files rclone hangs.

What is your rclone version (output from rclone version)

I've tried the 1.50.0 and the latest 1.63.0

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

linux, amd64

Which cloud storage system are you using? (e.g. Google Drive)

s3 - minio

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone --config=rclone.conf --s3-chunk-size=100Mi -P -vv moveto s3:test/testfile s3:test/testfile.new

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

2023/07/17 14:01:57 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "--config=rclone.conf" "--s3-chunk-size=100Mi" "-P" "-vv" "moveto" "s3:test/testfile" "s3:test/testfile.new"]
2023/07/17 14:01:57 DEBUG : Creating backend with remote "s3:test/testfile"
2023/07/17 14:01:57 DEBUG : Using config file from "/root/rclone.conf"
2023/07/17 14:01:57 DEBUG : s3: detected overridden config - adding "{HvT2Y}" suffix to name
2023/07/17 14:01:57 DEBUG : name = "s3{HvT2Y}", root = "test/testfile", opt = &s3.Options{Provider:"Minio", EnvAuth:true, AccessKeyID:"yyy", SecretAccessKey:"xxx", Region:"eu-west-1", Endpoint:"https://localhost", STSEndpoint:"", LocationConstraint:"", ACL:"bucket-owner-full-control", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:104857600, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:true, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false}, UsePresignedRequest:false, Versions:false, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2023/07/17 14:01:57 DEBUG : Resolving service "s3" region "eu-west-1"
2023/07/17 14:01:57 DEBUG : fs cache: adding new entry for parent of "s3:test/testfile", "s3{HvT2Y}:test"
2023/07/17 14:01:57 DEBUG : Creating backend with remote "s3:test/"
2023/07/17 14:01:57 DEBUG : s3: detected overridden config - adding "{HvT2Y}" suffix to name
2023/07/17 14:01:57 DEBUG : name = "s3{HvT2Y}", root = "test/", opt = &s3.Options{Provider:"Minio", EnvAuth:true, AccessKeyID:"yyy", SecretAccessKey:"xxx", Region:"eu-west-1", Endpoint:"https://localhost", STSEndpoint:"", LocationConstraint:"", ACL:"bucket-owner-full-control", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:104857600, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:true, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false}, UsePresignedRequest:false, Versions:false, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2023/07/17 14:01:57 DEBUG : Resolving service "s3" region "eu-west-1"
2023/07/17 14:01:57 DEBUG : fs cache: renaming cache item "s3:test/" to be canonical "s3{HvT2Y}:test"
2023-07-17 14:01:57 DEBUG : testfile: Need to transfer - File not found at Destination
2023-07-17 14:01:57 DEBUG : testfile: Starting  multipart copy with 3 parts
2023-07-17 14:22:46 DEBUG : testfile: Cancelling multipart copy
2023-07-17 14:22:55 ERROR : testfile: Failed to copy: SerializationError: failed to unmarshal error message
	status code: 504, request id: , host id:
caused by: UnmarshalError: failed to unmarshal error message
	00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|
00000010  69 74 6c 65 3e 35 30 34  20 47 61 74 65 77 61 79  |itle>504 Gateway|
00000020  20 54 69 6d 65 2d 6f 75  74 3c 2f 74 69 74 6c 65  | Time-out</title|
00000030  3e 3c 2f 68 65 61 64 3e  0d 0a 3c 62 6f 64 79 3e  |></head>..<body>|
00000040  0d 0a 3c 63 65 6e 74 65  72 3e 3c 68 31 3e 35 30  |..<center><h1>50|
00000050  34 20 47 61 74 65 77 61  79 20 54 69 6d 65 2d 6f  |4 Gateway Time-o|
00000060  75 74 3c 2f 68 31 3e 3c  2f 63 65 6e 74 65 72 3e  |ut</h1></center>|
00000070  0d 0a 3c 68 72 3e 3c 63  65 6e 74 65 72 3e 6e 67  |..<hr><center>ng|
00000080  69 6e 78 2f 31 2e 31 38  2e 30 20 28 55 62 75 6e  |inx/1.18.0 (Ubun|
00000090  74 75 29 3c 2f 63 65 6e  74 65 72 3e 0d 0a 3c 2f  |tu)</center>..</|
000000a0  62 6f 64 79 3e 0d 0a 3c  2f 68 74 6d 6c 3e 0d 0a  |body>..</html>..|

caused by: expected element type <Error> but have <html>
2023-07-17 14:22:55 ERROR : testfile: Not deleting source as copy failed: SerializationError: failed to unmarshal error message
	status code: 504, request id: , host id:
caused by: UnmarshalError: failed to unmarshal error message
	00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|
00000010  69 74 6c 65 3e 35 30 34  20 47 61 74 65 77 61 79  |itle>504 Gateway|
00000020  20 54 69 6d 65 2d 6f 75  74 3c 2f 74 69 74 6c 65  | Time-out</title|
00000030  3e 3c 2f 68 65 61 64 3e  0d 0a 3c 62 6f 64 79 3e  |></head>..<body>|
00000040  0d 0a 3c 63 65 6e 74 65  72 3e 3c 68 31 3e 35 30  |..<center><h1>50|
00000050  34 20 47 61 74 65 77 61  79 20 54 69 6d 65 2d 6f  |4 Gateway Time-o|
00000060  75 74 3c 2f 68 31 3e 3c  2f 63 65 6e 74 65 72 3e  |ut</h1></center>|
00000070  0d 0a 3c 68 72 3e 3c 63  65 6e 74 65 72 3e 6e 67  |..<hr><center>ng|
00000080  69 6e 78 2f 31 2e 31 38  2e 30 20 28 55 62 75 6e  |inx/1.18.0 (Ubun|
00000090  74 75 29 3c 2f 63 65 6e  74 65 72 3e 0d 0a 3c 2f  |tu)</center>..</|
000000a0  62 6f 64 79 3e 0d 0a 3c  2f 68 74 6d 6c 3e 0d 0a  |body>..</html>..|

caused by: expected element type <Error> but have <html>
2023-07-17 14:22:55 ERROR : Attempt 1/3 failed with 1 errors and: SerializationError: failed to unmarshal error message
	status code: 504, request id: , host id:
caused by: UnmarshalError: failed to unmarshal error message
	00000000  3c 68 74 6d 6c 3e 0d 0a  3c 68 65 61 64 3e 3c 74  |<html>..<head><t|
00000010  69 74 6c 65 3e 35 30 34  20 47 61 74 65 77 61 79  |itle>504 Gateway|
00000020  20 54 69 6d 65 2d 6f 75  74 3c 2f 74 69 74 6c 65  | Time-out</title|
00000030  3e 3c 2f 68 65 61 64 3e  0d 0a 3c 62 6f 64 79 3e  |></head>..<body>|
00000040  0d 0a 3c 63 65 6e 74 65  72 3e 3c 68 31 3e 35 30  |..<center><h1>50|
00000050  34 20 47 61 74 65 77 61  79 20 54 69 6d 65 2d 6f  |4 Gateway Time-o|
00000060  75 74 3c 2f 68 31 3e 3c  2f 63 65 6e 74 65 72 3e  |ut</h1></center>|
00000070  0d 0a 3c 68 72 3e 3c 63  65 6e 74 65 72 3e 6e 67  |..<hr><center>ng|
00000080  69 6e 78 2f 31 2e 31 38  2e 30 20 28 55 62 75 6e  |inx/1.18.0 (Ubun|
00000090  74 75 29 3c 2f 63 65 6e  74 65 72 3e 0d 0a 3c 2f  |tu)</center>..</|
000000a0  62 6f 64 79 3e 0d 0a 3c  2f 68 74 6d 6c 3e 0d 0a  |body>..</html>..|

caused by: expected element type <Error> but have <html>
2023-07-17 14:22:55 DEBUG : testfile: Need to transfer - File not found at Destination
2023-07-17 14:22:55 DEBUG : testfile: Starting  multipart copy with 3 parts
Transferred:   	          0 B / 10 GiB, 0%, 0 B/s, ETA -
Checks:                 1 / 2, 50%
Transferred:            0 / 1, 0%
Elapsed time:     23m58.0s
Checking:

Transferring:
 *                                      testfile:  0% /10Gi, 0/s, -

When the test file is 1GB it works well:

2023/07/17 14:30:40 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "--config=rclone.conf" "--s3-chunk-size=100Mi" "-P" "-vv" "moveto" "s3:test/testfile" "s3:test/testfile.new"]
2023/07/17 14:30:40 DEBUG : Creating backend with remote "s3:test/testfile"
2023/07/17 14:30:40 DEBUG : Using config file from "/root/rclone.conf"
2023/07/17 14:30:40 DEBUG : s3: detected overridden config - adding "{HvT2Y}" suffix to name
2023/07/17 14:30:40 DEBUG : name = "s3{HvT2Y}", root = "test/testfile", opt = &s3.Options{Provider:"Minio", EnvAuth:true, AccessKeyID:"cnRio0qdV2eAZWoM", SecretAccessKey:"HHIbZjeQDGNNNCy38oeb6jRrIGnqsa3Y", Region:"eu-west-1", Endpoint:"https://backup1.live1.de1.eu.nuaxiasystem.com", STSEndpoint:"", LocationConstraint:"", ACL:"bucket-owner-full-control", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:104857600, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:true, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false}, UsePresignedRequest:false, Versions:false, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2023/07/17 14:30:40 DEBUG : Resolving service "s3" region "eu-west-1"
2023/07/17 14:30:40 DEBUG : fs cache: adding new entry for parent of "s3:test/testfile", "s3{HvT2Y}:test"
2023/07/17 14:30:40 DEBUG : Creating backend with remote "s3:test/"
2023/07/17 14:30:40 DEBUG : s3: detected overridden config - adding "{HvT2Y}" suffix to name
2023/07/17 14:30:40 DEBUG : name = "s3{HvT2Y}", root = "test/", opt = &s3.Options{Provider:"Minio", EnvAuth:true, AccessKeyID:"cnRio0qdV2eAZWoM", SecretAccessKey:"HHIbZjeQDGNNNCy38oeb6jRrIGnqsa3Y", Region:"eu-west-1", Endpoint:"https://backup1.live1.de1.eu.nuaxiasystem.com", STSEndpoint:"", LocationConstraint:"", ACL:"bucket-owner-full-control", BucketACL:"", RequesterPays:false, ServerSideEncryption:"", SSEKMSKeyID:"", SSECustomerAlgorithm:"", SSECustomerKey:"", SSECustomerKeyBase64:"", SSECustomerKeyMD5:"", StorageClass:"", UploadCutoff:209715200, CopyCutoff:4999341932, ChunkSize:104857600, MaxUploadParts:10000, DisableChecksum:false, SharedCredentialsFile:"", Profile:"", SessionToken:"", UploadConcurrency:4, ForcePathStyle:true, V2Auth:false, UseAccelerateEndpoint:false, LeavePartsOnError:false, ListChunk:1000, ListVersion:0, ListURLEncode:fs.Tristate{Value:false, Valid:false}, NoCheckBucket:true, NoHead:false, NoHeadObject:false, Enc:0x3000002, MemoryPoolFlushTime:60000000000, MemoryPoolUseMmap:false, DisableHTTP2:false, DownloadURL:"", DirectoryMarkers:false, UseMultipartEtag:fs.Tristate{Value:false, Valid:false}, UsePresignedRequest:false, Versions:false, VersionAt:fs.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}, Decompress:false, MightGzip:fs.Tristate{Value:false, Valid:false}, UseAcceptEncodingGzip:fs.Tristate{Value:false, Valid:false}, NoSystemMetadata:false}
2023/07/17 14:30:40 DEBUG : Resolving service "s3" region "eu-west-1"
2023/07/17 14:30:40 DEBUG : fs cache: renaming cache item "s3:test/" to be canonical "s3{HvT2Y}:test"
2023-07-17 14:30:40 DEBUG : testfile: Need to transfer - File not found at Destination
2023-07-17 14:31:03 DEBUG : testfile: md5 = cd573cfaace07e7949bc0c46028904ff OK
2023-07-17 14:31:03 INFO  : testfile: Copied (server-side copy) to: testfile.new
2023-07-17 14:31:03 INFO  : testfile: Deleted
Transferred:   	        1 GiB / 1 GiB, 100%, 0 B/s, ETA -
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:        22.5s
2023/07/17 14:31:03 INFO  :
Transferred:   	        1 GiB / 1 GiB, 100%, 0 B/s, ETA -
Checks:                 2 / 2, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Transferred:            1 / 1, 100%
Elapsed time:        22.5s

2023/07/17 14:31:03 DEBUG : 7 go routines active

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
@pasztorl
Copy link
Author

If I use --disable Copy it works, so the problem related to server-side copy feature?

@ncw
Copy link
Member

ncw commented Jul 17, 2023

It looks like minio is timing out as the server side copy operations are taking too long.

Try setting this value to much smaller 512M say - this should fix the problem if 1GB file copies work OK.

  --s3-copy-cutoff SizeSuffix   Cutoff for switching to multipart copy (default 4.656Gi)

@pasztorl
Copy link
Author

It worked, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ncw @pasztorl