s3sync support

[jwhitcraft]

It would be great if there was support for s3sync and s3cmd http://www.s3sync.net/wiki

[mrrooijen]

Hi. Thanks for pointing this out. I already considered this a while ago but for some reason I believe I got ahold of an old version that seemed to be either abandoned or just didn't work, so I assumed it was no longer supported.

I just briefly checked out s3sync.rb and successfully got it up and running (with Ruby 1.8.7 and REE, but not with Ruby 1.9.2 due to the fact that it can't find the md5 encryptor library.

I haven't looked in to this yet myself but if you know why I get the following error and how to fix it, it'd save me some research time.

/Users/Michael/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': no such file to load -- md5 (LoadError)
    from /Users/Michael/.rvm/rubies/ruby-1.9.2-p180/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
    from ./s3sync:29:in `<module:S3sync>'
    from ./s3sync:17:in `<main>'

If not then I'll see why it occurs. Probably used to be in Ruby's core and was removed since 1.9 or something.

However since I got it to work under 1.8 and it seems like a very useful addition to the Backup gem I will most certainly implement this very soon. It also has the ability to mirror using the --delete flag, similar to RSync. However do you by any chance know if it'll transfer the whole file again when it's only partially changed? Like for example a MySQL dump? Or will it always transfer the whole file, even if only a couple of bytes of that particular file changed?

Previously, before I implemented Syncers this was the reason why I didn't use the s3sync approach, because from what I read there was no way to get the real RSync ( diff ) effect on files on the S3 filesystem, so it'd always transfer the full bytes, and since storages in Backup always bundled everything up as a single .tar.gz.enc file, the whole syncing purpose would be pointless since it'd always transfer the whole file. However since the introduction of Syncers in Backup, where you can specify a whole directory (and sub-directories) to directly sync to a remote location, bypassing the whole copy/tar/compress/encrypt procedure then s3sync will regardless be an very awesome addition to the Backup gem.

So let me know if you have any info on the above error which only occurs in Ruby 1.9, and whether you know if a file that only changed a few bytes will be completely re-uploaded.

Thanks!

[mrrooijen]

Something tells me that S3Sync hasn't been updated to work with Ruby 1.9.2 :/

[jwhitcraft]

i dont think it has been updated in a while. I've been using 1.2.6 for the past 3 years. I think it's been abandoned. Let me look to see if I can get it working under 1.9.x in the next few weeks.

[mrrooijen]

Thanks. Yeah actually for Ruby 1.9.2 the path to the md5 library is require 'digest/md5'rather than just require 'md5'. That's the first thing that goes wrong in the s3sync.rb file. The second issue is in the Thread library:

/Users/Michael/Desktop/s3sync/thread_generator.rb:112:in `end?': undefined method `critical=' for Thread:Class (NoMethodError)
    from /Users/Michael/Desktop/s3sync/thread_generator.rb:129:in `next?'
    from ./s3sync.rb:366:in `main'
    from ./s3sync.rb:738:in `<main>'

Hopefully there aren't much more issues that need to be made compatible, then we can make it Ruby 1.9 compatible and add S3Sync to Backup.

[mrrooijen]

I just found this on GitHub, a Ruby 1.9.2 compatible S3Sync:

https://github.com/aproxacs/s3sync

However now I run it with Ruby 1.8.7 and i get errors ha. But this is a good step in the right direction. Going to see what can be done. In the worst case we now have 2 separate libraries that work the same, but one is compaitble for Ruby1.8 and the other for Ruby1.9

[mrrooijen]

Update:

Ruby 1.9 compatible S3Sync
gem install aproxacs-s3sync

Ruby 1.8 compatible S3Sync
gem install s3sync

The user will have to ensure the correct one is installed for their Ruby version.

[mrrooijen]

Well I actually got it working with Backup. I'm just running some more tests on it, and writing more specs, and i'll push a gem soon that includes the S3 Syncer. :) hurray. Works both with Ruby 1.8 and Ruby 1.9, but you gotta use a different gem depending on which you want to use (as stated above in my comment) but that's absolutely no problem, just a different gem install command.

[mrrooijen]

Boom: 69e81d0

This commit integrates S3Sync with the Backup gem. Going to update the Syncer Wiki page.

[mrrooijen]

And it has been released.

gem install backup

and check out the docs here: https://github.com/meskyanichi/backup/wiki/Syncers

[jwhitcraft]

Awesome thanks for the fast response!

[mrrooijen]

You're welcome! I know I'll enjoy this one as well : )

[ponny]

Took me a while to figure this out. Could you add it to the wiki for S3?

[mrrooijen]

@ponny add what to the wiki? the S3syncer? It's here already: https://github.com/meskyanichi/backup/wiki/Syncers

[ponny]

Erk. Right you are. I must be blind.