Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Amazon Glacier #929

Closed
vlcinsky opened this issue Aug 21, 2012 · 28 comments
Closed

Support for Amazon Glacier #929

vlcinsky opened this issue Aug 21, 2012 · 28 comments
Labels

Comments

@vlcinsky
Copy link

@vlcinsky vlcinsky commented Aug 21, 2012

@lreimann

This comment has been minimized.

Copy link

@lreimann lreimann commented Aug 21, 2012

vlcinsky, you were some seconds faster than me ;)

I agree on this! Please make an effort in implementing glacier support.

Greetings

@dbanck

This comment has been minimized.

Copy link

@dbanck dbanck commented Aug 21, 2012

+1

2 similar comments
@c35sys

This comment has been minimized.

Copy link

@c35sys c35sys commented Aug 21, 2012

+1

@lury

This comment has been minimized.

Copy link

@lury lury commented Aug 21, 2012

+1

@bcap

This comment has been minimized.

Copy link

@bcap bcap commented Aug 21, 2012

well im here to also vote +1 but probably wont need to :)

@leightonj

This comment has been minimized.

Copy link

@leightonj leightonj commented Aug 21, 2012

+1

@toxinu

This comment has been minimized.

Copy link

@toxinu toxinu commented Aug 21, 2012

Really need it !

@almost

This comment has been minimized.

Copy link
Contributor

@almost almost commented Aug 21, 2012

I've started work on some Python Glacier code. Just the beginnings but maybe it'll be useful to someone: https://gist.github.com/3417915

I'll try and update it as I add more bits and maybe try and beat it into a shape where I can contribute it to Glacier if someone else doesn't beat me to it.

@neilb14

This comment has been minimized.

Copy link

@neilb14 neilb14 commented Aug 21, 2012

+1

@garnaat

This comment has been minimized.

Copy link
Member

@garnaat garnaat commented Aug 22, 2012

I'm starting a glacier module and will commit to a branch soon. I won't have enough time to do the whole thing but it's a start and perhaps some of you will have time to help flesh it out.

@garnaat

This comment has been minimized.

Copy link
Member

@garnaat garnaat commented Aug 22, 2012

Just committed a branch: https://github.com/boto/boto/tree/glacier

This only includes low-level (Layer1) support for Vault and Job operations so far.

@pfig

This comment has been minimized.

Copy link
Contributor

@pfig pfig commented Aug 24, 2012

+1

@almost

This comment has been minimized.

Copy link
Contributor

@almost almost commented Aug 27, 2012

I've started work on Layer2 for Glacier in my branch here: https://github.com/almost/boto/tree/glacier

This is based on @garnaat's layer1 branch (I've sent a pull request) and the code from my gist

@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Aug 28, 2012

+1

3 similar comments
@Knio

This comment has been minimized.

Copy link

@Knio Knio commented Sep 1, 2012

+1

@thanos

This comment has been minimized.

Copy link

@thanos thanos commented Sep 4, 2012

+1

@probabilistic

This comment has been minimized.

Copy link

@probabilistic probabilistic commented Sep 4, 2012

+1

@garnaat

This comment has been minimized.

Copy link
Member

@garnaat garnaat commented Sep 4, 2012

Great to see so much interest. It would be great if some of you could check out the glacier branch and provide any feedback. It's getting close and my plan is to include it in the release that will be happening today or tomorrow.

@yamatt

This comment has been minimized.

Copy link
Contributor

@yamatt yamatt commented Sep 5, 2012

@almost @garnaat I was going to put these in an issue for you on your boto/glacier fork but for some reason there is no issues page. I hope here is ok.

One thing that seems to jump out at my right away is that there doesn't seem to be a way to list archives, or is there something I'm not getting?

The other thing; there doesn't seem to be a huge difference between Vault.upload_archive and Vault.create_archive_from_file. Why have 2?

@almost

This comment has been minimized.

Copy link
Contributor

@almost almost commented Sep 5, 2012

One uploads via a single request and the other uploads using a multi-part upload. I wrote the create_archive_* methods and someone else added the upload_archive* methods and I do think there's a bit too much overlap there, or at least they need to be named sensibly!

The home of the Glacier work is here on boto github, so it's best to create discussions here. The fork on my account I'm just using to send pull requests as I don't have write access to the main repository.

@moliware

This comment has been minimized.

Copy link
Contributor

@moliware moliware commented Sep 5, 2012

In Vault class there are 3 methods for uploading files:

  • upload_archive_single_operation : Upload a file using Layer1.upload_archive (for that reason I used upload_archive*) recommended for archives less than 100MB
  • create_archive_from_file: Upload a file using multipart
  • upload_archive: Depending on the file size, it chooses one of the previous method.

There is nothing overlapped but I think they should be renamed, because now it is not clear. Maybe we could rename create_archive_from_file as upload_archive_multipart or maybe rename the upload_archive* methods. What do you think?

@yamatt There is a small script for upload file from command line in bin/glacier. Maybe it is useful for you.

@yamatt

This comment has been minimized.

Copy link
Contributor

@yamatt yamatt commented Sep 6, 2012

@moliware thanks, yes, I was using that the guide when I got stuck.

I do think the names could be a little clearer. Personally I would rename create_archive_from_file, and upload_archive_single_operation to _create_archive_from_file and _upload_archive_single_operation do denote that they are not expected to be used by the developer but they are there, although I see little use of psudo private names in boto.

To be fair my first stop was upload_archive as it is the most obvious. So if you change nothing and update the doc string of upload_archive to say this is the automated one I doubt there will be a great deal of confusion.

@almost

This comment has been minimized.

Copy link
Contributor

@almost almost commented Sep 6, 2012

How about we change upload_archive to accept a file_obj or a filename, in the case of a filename it will do a one-shot upload for < 100 MB and for fileojb or > 100 MB it will do multipart. Keep create_archive_writer as public as well but maker the others private.

@yamatt

This comment has been minimized.

Copy link
Contributor

@yamatt yamatt commented Sep 10, 2012

Please let me know if there is an issue tracker you'd rather I use, I'll re-add this here. I pulled the latest changes today and had a play.

  • If you try and delete a vault with things in it you get a UnexpectedHTTPResponseError. Wondered if that should be a caught error. Perhaps raising a proper boto error.

Also I think this is more of a limitation of Glacier more than anything but if you lose the archive ID there doesn't seem to be a way of recovering it without raising a ticket. Do let me know if this is incorrect.

@almost

This comment has been minimized.

Copy link
Contributor

@almost almost commented Sep 10, 2012

That error probably should be caught.

It is possible to list the contents of a vault but you have to do it using an "inventory-retrieval" job the same way you'd download an archive. So it can take a few hours (I assume, I haven't tried it yet). We should definitely add support to that to boto, I've just been lazy and haven't needed it in my app yet...

@yamatt

This comment has been minimized.

Copy link
Contributor

@yamatt yamatt commented Sep 10, 2012

Cool. As long as you're on top of it ;)

@dArignac

This comment has been minimized.

Copy link

@dArignac dArignac commented Sep 15, 2012

+1 for this feature! You are awesome!

@garnaat

This comment has been minimized.

Copy link
Member

@garnaat garnaat commented Sep 19, 2012

The glacier branch has been merged to develop. I'm sure there is still more work to do but I'm closing this particular issue at this time. Thanks for all of the interest and support.

@garnaat garnaat closed this Sep 19, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.