Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aws s3 sync --existing #1449

Open
ChrisSLT opened this issue Aug 15, 2015 · 7 comments
Open

aws s3 sync --existing #1449

ChrisSLT opened this issue Aug 15, 2015 · 7 comments
Labels
feature-request A feature should be added or improved. p3 This is a minor priority issue s3sync s3syncstrategy s3

Comments

@ChrisSLT
Copy link

Could you implement a feature like RYNC's --existing so that you can tell AWS to "skip creating files that do not exist yet on the destination"

Or is there a way to do this already?

The use case is you have just synced from machine 1 up to S3 and you have a huge dataset in S3. Now you want to sync down from S3 to machine 2 so you're working with up to date files on machine 2. But you only want to update files that are already on machine 2, you don't want to copy the huge dataset in S3 down.

Make sense?
@jamesls
Copy link
Member

jamesls commented Aug 15, 2015

Not currently possible to do with the CLI. The closest thing you could is either:

a) only sync up to a specific prefix (s3://bucket/specific/dataset/)
b) only sync with --exclude/--include filters: `--exclude '' --include '.txt' s3://bucket/specific/prefix)

This could end up being a little tricky to implement efficiently, we may have to switch our internals to use HeadObject instead of ListObjects if --existing is specified.

Marking as a feature request.

@jamesls jamesls added the feature-request A feature should be added or improved. label Aug 15, 2015
@ASayre
Copy link
Contributor

ASayre commented Feb 6, 2018

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

@ASayre ASayre closed this as completed Feb 6, 2018
@salmanwaheed
Copy link

salmanwaheed commented Feb 6, 2018 via email

@kenorb
Copy link

kenorb commented Feb 13, 2018 •
edited
Loading

Related:

This was referenced Feb 13, 2018
Open
Open
@jamesls jamesls reopened this Apr 6, 2018
@jamesls
Copy link
Member

jamesls commented Apr 6, 2018

Based on community feedback, we have decided to return feature requests to GitHub issues.

@landsman
Copy link

Based on community feedback, we have decided to return feature requests to GitHub issues.

Very good choise

@lachesis
Copy link

lachesis commented Dec 27, 2022 •
edited
Loading

So... glad to hear discussion is back on Github. Is this feature going to be implemented at some point? What would need to happen to move it forward?

I'm about to write this myself (and I'm sure I'm not the first). My case is easy because I have no "directories" (or paths with / in s3). My script will be roughly:

aws s3 ls s3://$bucket | sort > remote
find $local_path | sort > local
comm -23 local remote | parallel aws s3 cp $local_path/{} s3://$bucket/{}

@tim-finnigan tim-finnigan added the p3 This is a minor priority issue label Mar 8, 2023
@tim-finnigan tim-finnigan mentioned this issue Apr 6, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature-request A feature should be added or improved. p3 This is a minor priority issue s3sync s3syncstrategy s3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@kenorb @lachesis @jamesls @landsman @kyleknap @ASayre @ChrisSLT @salmanwaheed @tim-finnigan