Skip to content

Fix bug when filtering s3 locations #531

Merged
merged 2 commits into from Dec 5, 2013

2 participants

@jamesls
Amazon Web Services member
jamesls commented Dec 5, 2013

The pattern is evaluated against the entire bucket. This doesn't
mattern for suffix searches, but for a prefix search, you'd have to
include the bucket name. This is inconsistent with filtering local
files. Now they're both consistent. Given:

  --exclude 'foo*'

This will filter any full path that startsw with foo. So locally:

  rootdir/
    foo.txt      # yes
    foo1.txt     # yes
    foo/bar.txt  # yes
    bar.txt      # no

And on s3, we now have the same results:

  bucket/
    foo.txt      # yes
    foo1.txt     # yes
    foo/bar.txt  # yes
    bar.txt      # no

I also added debug logs to help customers troubleshoot why their
filters aren't working the way they expect. For example:

s3.filters - DEBUG - /private/tmp/syncme/level-1/file-7 matched exclude filter: /private/tmp/syncme/*
s3.filters - DEBUG - /private/tmp/syncme/level-1/file-7 did not match include filter: /private/tmp/syncme/f*
s3.filters - DEBUG - /private/tmp/syncme/level-1/file-7 matched include filter: /private/tmp/syncme/level*
s3.filters - DEBUG - =/private/tmp/syncme/level-1/file-7 final filtered status, should_include: True
jamesls added some commits Dec 4, 2013
@jamesls jamesls Clean up existing filter unit tests
This makes the unit tests more concise and easier to
follow.  Removes a lot of the duplication.  Also makes it
easier to write new filter tests.
8c2941a
@jamesls jamesls Fix bug when filtering s3 locations
The pattern is evaluated against the entire bucket.  This doesn't
mattern for suffix searches, but for a prefix search, you'd have to
include the bucket name.  This is inconsistent with filtering local
files.  Now they're both consistent.  Given:

  --exclude 'foo*'

This will filter any full path that startsw with foo.  So locally:

  rootdir/
    foo.txt      # yes
    foo1.txt     # yes
    foo/bar.txt  # yes
    bar.txt      # no

And on s3, we now have the same results:

  bucket/
    foo.txt      # yes
    foo1.txt     # yes
    foo/bar.txt  # yes
    bar.txt      # no

I also added debug logs to help customers troubleshoot why their
filters aren't working the way they expect.
5063e56
@garnaat
garnaat commented Dec 5, 2013

LGTM

@jamesls jamesls merged commit 5063e56 into aws:develop Dec 5, 2013

1 check passed

Details default The Travis CI build passed
@jamesls jamesls deleted the jamesls:s3-fnmatch-bucket branch Jun 23, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.