Skip to content
Browse files

Add a prefix to b.list() so s3put/s3multiput don't list the whole bucket

Listing the whole bucket to avoid overwriting keys is extremely
inefficient (especially for buckets will millions or billions of
objects). Use the path provided by the user to add a prefix to b.list()
so that the keyspace is reduced to just the area we're trying to sync.
  • Loading branch information...
1 parent 0f16d46 commit 9289efed729546df092ea33e48d9726dee61126f @psa psa committed Feb 10, 2012
Showing with 2 additions and 2 deletions.
  1. +1 −1 bin/s3multiput
  2. +1 −1 bin/s3put
View
2 bin/s3multiput
@@ -256,7 +256,7 @@ def main():
if not quiet:
print 'Getting list of existing keys to check against'
keys = []
- for key in b.list():
+ for key in b.list(get_key_name(path, prefix)):
keys.append(key.name)
for root, dirs, files in os.walk(path):
for ignore in ignore_dirs:
View
2 bin/s3put
@@ -162,7 +162,7 @@ def main():
if not quiet:
print 'Getting list of existing keys to check against'
keys = []
- for key in b.list():
+ for key in b.list(get_key_name(path, prefix)):
keys.append(key.name)
for root, dirs, files in os.walk(path):
for ignore in ignore_dirs:

0 comments on commit 9289efe

Please sign in to comment.
Something went wrong with that request. Please try again.