Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

serve s3: multipart server side copy does not work #7454

Open
ncw opened this issue Nov 24, 2023 · 0 comments
Open

serve s3: multipart server side copy does not work #7454

ncw opened this issue Nov 24, 2023 · 0 comments
Labels
bug serve s3

Comments

@ncw
Copy link
Member

ncw commented Nov 24, 2023

Multipart server side copies do not appear to work properly.

These can be tested with the integration test TestIntegration/FsMkdir/FsCopyChunked

The server appears to ignore the X-Amz-Copy-Source header and does not return the expected

<CopyPartResult>
   <ETag>string</ETag>
   <LastModified>timestamp</LastModified>
   <ChecksumCRC32>string</ChecksumCRC32>
   <ChecksumCRC32C>string</ChecksumCRC32C>
   <ChecksumSHA1>string</ChecksumSHA1>
   <ChecksumSHA256>string</ChecksumSHA256>
</CopyPartResult>

Which causes the AWS SDK to retry many times and it eventually fails.

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
@ncw ncw added the serve s3 label Nov 24, 2023
ncw added a commit that referenced this issue Nov 24, 2023
@ncw
serve s3: document multipart copy doesn't work #7454
d8855b2 
This puts in a workaround for the tests also
@ncw ncw added the bug label Nov 24, 2023
miku added a commit to internetarchive/rclone that referenced this issue Nov 30, 2023
@miku
Merge branch 'master' into ia-wt-1168
45c6105 
* master: (41 commits)
  Start v1.66.0-DEV development
  Version v1.65.0
  onedrive: add --onedrive-delta flag to enable ListR
  smb: fix modtime of multithread uploads by setting PartialUploads
  smb: fix about size wrong by switching to github.com/cloudsoda/go-smb2/ fork
  serve s3: fix overwrite of files with 0 length file
  serve s3: fix error handling for listing non-existent prefix - fixes rclone#7455
  test_all: make integration test for serve s3
  Add Abhinav Dhiman to contributors
  Add 你知道未来吗 to contributors
  imagekit: Added ImageKit backend
  fs/fshttp: fix --contimeout being ignored
  s3: ensure we can set upload cutoff that we use for Rclone provider
  serve s3: document multipart copy doesn't work rclone#7454
  b2: fix streaming chunked files an exact multiple of chunk size
  fstest: factor chunked streaming tests from b2 and use in all backends
  b2: fix server side chunked copy when file size was exactly --b2-copy-cutoff
  fstest: factor chunked copy tests from b2 and use them in s3 and oos
  operations: fix overwrite of destination when multi-thread transfer fails
  random: stop using deprecated rand.Seed in go1.20 and later
  ...
@saw-jan saw-jan mentioned this issue Jul 17, 2024
Merged
5 tasks
ncw pushed a commit that referenced this issue Jul 17, 2024
@saw-jan
serve s3: implement --auth-proxy
9de485f 
This implements --auth-proxy for serve s3. In addition it:

* add listbuckets tests with and without authProxy
* use auth proxy test framework
* servetest: implement workaround for #7454
* update github.com/rclone/gofakes3 to fix race condition
Fornax96 pushed a commit to Fornaxian/rclone that referenced this issue Jul 30, 2024
@ncw @Fornax96
serve s3: document multipart copy doesn't work rclone#7454
912115c 
This puts in a workaround for the tests also
Fornax96 pushed a commit to Fornaxian/rclone that referenced this issue Jul 30, 2024
@saw-jan @Fornax96
serve s3: implement --auth-proxy
49f020e 
This implements --auth-proxy for serve s3. In addition it:

* add listbuckets tests with and without authProxy
* use auth proxy test framework
* servetest: implement workaround for rclone#7454
* update github.com/rclone/gofakes3 to fix race condition
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug serve s3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ncw