Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
aws s3 sync --exclude does't work consistently #541
On OSX 10.8.5, I'm trying to exclude .DS_Store but --exclude doesn't seem to work as expected.
Why is ../.DS_Store getting uploaded?
Instead of --exclude "*.DS_Store", I tried --exclude .DS_Store, --exclude *.DS_Store, ".DS_Store" but all produce the same results.
==EDITED by author===
I believe that --include does not work and ends up including even if the filter does not work. I have not read the spec yet but if you are trying to sync from S3 down to a local file system and use --include, it does not behave as expected. Notice in the examples below how the "does not match" is identified, but the "should_include:" is set to true.
2013-12-11 18:32:15,643 - awscli.customizations.s3.filters - DEBUG - atlas-config.ec2.arbor.net/logs/feed-web-1/access-20131119T061701 did not match include filter: atlas-config.ec2.arbor.net/20131210T
2013-12-11 18:49:08,615 - awscli.customizations.s3.filters - DEBUG - atlas-config.ec2.arbor.net/logs/feed-web-1/access-20131122T071702 did not match include filter: atlas-config.ec2.arbor.net/20131210T
For the good of others - this is actually by code design. The conditional in filters.py only checkes for matched and then --include and then elseif --exclude, with a default of True. As a result, you MUST use --exclude="_" --include ="_pattern*" in order to just match the files you want.
otherwise, filters.py would have to be forked/modified to handle explicit use cases of --include w/o --exclude.
As to why --exclude does not work in the above, what happens if you put --debug after sync in your line? It will show the printout of what filters.py is doing.
Here's the problem:
A specific path to .DS_Store is being used so any .DS_Store that doesn't match that path will be included. How do I exclude any files that match .DS_Store?
yes, pathing is important, to prefix of * may be needed if it's not in the root of the bucket/container.
aws s3 sync ./ s3://sajee-sync/ --debug --exclude "*/.DS_Store" --dryrun
I'm not sure why *.DS_Store didn't work for you - but I'd like to see what the above shows. Also, can you aws --version for us too.