Add support for uploading to Amazon S3 bucket

[mmooney]

I needed this for another project so I just added in support for uploading files to an Amazon S3 bucket. I'm a little new the innards of the dropkick code, but I copied as much as a I could from the copy directory stuff.

I created a ProtoAmazonS3UploadDirectoryTask DSL class, which derives from BaseProtoTask, and implements FromOptions and a new AmazonS3Options (bucket name, access keys, etc).

I also created a AmazonS3UploadDirectoryTask class which derives from BaseIoTask, which then calls a IAmazonService to perform the uploading.

I uses the AWS SDK, which I added as a NuGet reference. I tried to avoid this at first, to avoid adding new dependencies. There is a partially implemented RestAmazonService in there, but getting all of the authentication signature and XML stuff right was a bit of a nightmare, the AWS SDK made it just a few lines of code. Anyhow, I hid it behind a IAmazonService that is injected into AmazonS3UploadDirectoryTask, so it could be changed later.

I'm using my fork of this for a client now, and it works pretty well. They are uploading 2700+ files, over 29 MB, and it takes about 20 minutes. I'm sure there is more that could be improved (clean destination, parallelizing the AWS calls, multipart uploads, etc), but this was good stable version that pretty much solved the problem.

I tried to put in a lot of testing around it. There are a handful of unit tests, and then the integration tests that actually talk to Amazon are marked as explicit, and read their config from a local file that has been .gitignore'd (the first time you run that test, it will tell you the path that it is looking in, and there is a sample file there too). It looked like there were a few different conventions being used in the tests (TinySpec, etc), and I confess I didn't understand them, so I tried to keep my tests pretty small and straightfoward.

Thanks

[ferventcoder]

I've worked with the aws sdk library in the past. It helps tremendously.

[ferventcoder]

You do understand that extensions can be added to dropkick from a different assembly? It's probably time we start looking at splitting some of these off to their own libraries...I have a few other in mind that could be pulled off to their own libraries.

[mmooney]

Yeah sure, do you want me to create a new contrib-type project, or do you already have one you want me to send this Amazon stuff over to?

[ferventcoder]

Acutally I was thinking you would create one called something like dropkick-awsextensions

[ferventcoder]

And add it to a wiki page noting extensions...

[mmooney]

OK cool, I'll get going on that this week. Are there any other extensions out there I could use a reference, or is this relatively uncharted territory? If so, that's cool, but if there are already some conventions in place I'd want to make sure I followed them.

[ferventcoder]

Uncharted... What you make will probably be the example we point to. No
pressure... ;)

On Tuesday, April 23, 2013, Mike Mooney wrote:

OK cool, I'll get going on that this week. Are there any other extensions
out there I could use a reference, or is this relatively uncharted
territory? If so, that's cool, but if there are already some conventions in
place I'd want to make sure I followed them.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/57#issuecomment-16861042
.

---

Rob
"Be passionate in all you do"

http://devlicio.us/blogs/rob_reynolds
http://ferventcoder.com
http://twitter.com/ferventcoder

[mmooney]

Fantastic! :). Do you it added as a PR into the dropkick repo, or as a seperate repo?

[mmooney]

Closing this, building it as an extension instead.

[ferventcoder]

@mmooney thanks for the updates.

[ferventcoder]

Also @mmooney - if you would contact me at gmail, I have some questions for you. My email is my github user name at gmail.