Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 /trash moves fail > 5GB #73

Open
vacri opened this issue Nov 5, 2020 · 0 comments
Open

S3 /trash moves fail > 5GB #73

vacri opened this issue Nov 5, 2020 · 0 comments

Comments

@vacri
Copy link

vacri commented Nov 5, 2020

version: buttersink 0.6.9 via pip

Buttersink runs into the 5GB chunk limit when 'trashing' files on s3. I have a list of old snapshots on s3 which float around 5GB - all the ones larger than 5GB haven't been 'trashed', but the ones below 5GB move into /trash as expected.

I'm using buttersink in a backup script for prometheus, and I get these sorts of errors when it runs

   InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120
   Trash: 0539...104e //prometheus_backup from None (17.45 GiB)
   InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120
   Trash: 60f2...6410 //prometheus_backup from None (5.341 GiB)
   InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120
   Trash: 73de...22cb //prometheus_backup from None (5.528 GiB)
   InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120
   Trash: 81f7...d2ff //prometheus_backup from None (5.082 GiB)
   InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120
   Trash: 4f96...b6a7 //prometheus_backup from None (5.406 GiB)

S3 needs all operations above 5GB to be broken into 5GB chunks (or less). Note that I have no problem in syncing the original large files from the server to s3 - this issue is just in moving the snapshot to the /trash folder

Cheers,

vacri

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant