Skip to content
/ s3copy Public
forked from lathanh/s3nukem

A Ruby script to copy an Amazon S3 bucket to another bucket without downloading the file and reuploading. Uses multiple threads to speed things up dramatically. This script is derived from https://github.com/lathanh/s3nukem.

License

Notifications You must be signed in to change notification settings

fredngo/s3copy

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

S3NUKEM

s3nukem is a slightly improved version of s3nuke, a Ruby script by Steve Eley that relatively quickly deletes an Amazon Web Services (AWS) Simple Storage Service (S3) bucket with many objects (millions) by using multiple threads to retrieve and delete the individual objects.

In my use case, s3cmd, which deletes with a single thread, deleted objects at a rate of about 1,800/minute (2.5 million / day). s3nukem, with 10 delete threads deleted objects at a rate of about 9,000/minute (13 million / day). My task of deleting 99 million objects went from 40 days to 7.6 days. More threads and Ruby 1.9 (I was using 1.8.5) would have probably completed the job even more quickly.

Improvements

  • The key retrieval thread will pause when the queue contains 1000 * thread_count items. The original script's queue would grow unabated, eating up memory unnecessarily.

  • All output is automatically flushed, which ensures you can keep an eye on progress in real-time

  • Added the number of seconds elapsed since the start of the script to output so you can calculate the rate at which items are being deleted.

Installation

You'll need:

  • Ruby Ruby 1.9 should work faster because of the native thread implementation (on the other hand, network/S3 latency may be your biggest bottleneck).

  • right_aws gem; dmarkow's version if you're running Ruby >= 1.9

      # Ruby < 1.9
      sudo gem install right_aws
    
      # Ruby >= 1.9
      sudo gem install dmarkow-right_aws --source http://gems.github.com
    

Download and make executable; e.g.,

# download
wget https://raw.github.com/lathanh/s3nukem/master/s3nukem
# or
curl -O https://raw.github.com/lathanh/s3nukem/master/s3nukem

# make executable
chmod 755 s3nukem

Obvious Warning

This script is intended to delete all of the items in an S3 bucket very quickly. You will not be prompted to ask you if you're sure. There is no undo.

Do not taunt Happy Fun Script.

License

This script is released under the Apache License, Version 2.0. I really don't care what you do with it, so long as "sue me" is not on the agenda.

Credits

Original script by Steve Eley.

Improvements by Robert LaThanh.

About

A Ruby script to copy an Amazon S3 bucket to another bucket without downloading the file and reuploading. Uses multiple threads to speed things up dramatically. This script is derived from https://github.com/lathanh/s3nukem.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Ruby 100.0%