Skip to content

S3 multipart upload #43

Open
wants to merge 1 commit into from

4 participants

@rgeyer
RightScale-Engineering member
rgeyer commented Jan 8, 2011

Hey all,

I found myself needing a multithreaded multi part upload to S3 for some big files I was moving around. I found that only Fog had any existing support, and I didn't want yet another gem to install with my ServerTemplates so I decided to add the functionality to right_aws.

I wrote a few tests which do not fully cover my method, but I have used the method pretty extensively already and it's reasonably stable. It also appears as though the existing tests don't cover all of the existing code. Besides, I am a bit of a Ruby n00b and I couldn't get the entire suite of existing tests to pass, so I sorta gave up. ;-)

I posted in the RightScale forums about this as well; https://forums.rightscale.com/showthread.php?t=774

Hope this is useful for someone!

-Ryan J. Geyer-

@rgeyer rgeyer Added Rightscale::S3Interface.store_object_multipart. Not a ton of te…
…sts, since I'm a bit of a Ruby n00b, and since the existing tests don't seem to fully cover the existing code
90df956
@rgeyer
RightScale-Engineering member
rgeyer commented Jan 14, 2011

I just had an opportunity to try this out from a windows box and discovered that it doesn't exactly play nice there. Looks like I have some additional work to do for windows.

It worked great on my OSX box, and linux though. shrug

-Ryan J. Geyer-

@konstantin-dzreev

Hi Ryan,

Your fix may not work correctly because right_aws is not thread-safe. We are looking into this issue now...

Konstantin.

@ryansch
ryansch commented May 19, 2011

I was just wishing this existed and here it is! Thanks again Ryan.

@rgeyer
RightScale-Engineering member
rgeyer commented May 19, 2011

No problem ryansch. As I'm sure you read above, this seems to work fine in *nix environments, but not so much in Windows.

I actually intend to refactor this code a bit such that right_aws is only providing the raw AWS method calls for S3 multi part. Then consuming applications can build their own multi-threaded approach to simultaneous uploading of chunks. I'll likely leave the existing store_object_multipart method in the file, but either comment it out, or move it up the chain to the S3 class which abstracts the S3Interface.

@titas
titas commented Aug 10, 2011

Hi guys, any news on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.