New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
aws s3 sync --delete: delete files _after_ uploading #1417
Comments
What's the reasoning for waiting to delete until after all uploads are complete? If we did that, then we'd have to keep every delete request in memory, which could consume a large amount of memory for large buckets. |
@mtdowling all the reasoning that’s behind rsync’s For example, if the transfer fails while syncing, let’s say right after the delete, I’ve deleted the old files, but haven’t added any of the new, which could be a problem. |
Interesting. I suppose we could also evaluate adding other delete strategies similar to the delete options offered by rsync. I'll mark this as a feature request, and we'll add it to our backlog. |
A quick workaround is to run the |
Good Morning! We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI. This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports. As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions. We’ve imported existing feature requests from GitHub - Search for this issue there! And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue. GitHub will remain the channel for reporting bugs. Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface -The AWS SDKs & Tools Team |
Based on community feedback, we have decided to return feature requests to GitHub issues. |
Any updates on this? the feature delete_after is important IMHO in order to clean up source systems that have been synced |
@mehditlili use rclone instead. |
Run your command twice, second time with |
That's a nice workaround! However it is slower than if the command was run once with the deletes at the end. |
Thank you for sharing this holy grail! This is exactly what I needed - rclone delete-after. For deploying a static website to S3, I can safely say that storing the website's content in memory due to Thanks again! |
Hello,
With
awscli==1.7.37
, when usings3 sync --delete
, is there a way to have the files be deleted after the rest is uploaded? It seems like they’re always deleted before uploading.The text was updated successfully, but these errors were encountered: