Permalink
Browse files

Merge branch 'develop'

  • Loading branch information...
2 parents d8b5015 + b70bebe commit 6f90c1a9d24e2c35f144be2f02339f56a25df693 Michael van Rooijen committed Mar 13, 2011
View
@@ -1,18 +1,20 @@
PATH
remote: .
specs:
- backup (3.0.3)
+ backup (3.0.4)
dropbox (~> 1.2.3)
fog (~> 0.5.3)
mail (~> 2.2.15)
net-scp (~> 1.0.4)
net-sftp (~> 2.0.5)
thor (~> 0.14.6)
+ twitter (~> 1.1.2)
GEM
remote: http://rubygems.org/
specs:
activesupport (3.0.5)
+ addressable (2.2.4)
builder (3.0.0)
diff-lcs (1.1.2)
dropbox (1.2.3)
@@ -21,6 +23,12 @@ GEM
multipart-post (>= 1.0)
oauth (>= 0.3.6)
excon (0.5.6)
+ faraday (0.5.7)
+ addressable (~> 2.2.4)
+ multipart-post (~> 1.1.0)
+ rack (< 2, >= 1.1.0)
+ faraday_middleware (0.3.2)
+ faraday (~> 0.5.4)
fog (0.5.3)
builder
excon (>= 0.5.2)
@@ -30,11 +38,12 @@ GEM
net-ssh (>= 2.0.23)
nokogiri (>= 1.4.4)
ruby-hmac
- formatador (0.0.16)
+ formatador (0.1.1)
fuubar (0.0.3)
rspec (~> 2.0)
rspec-instafail (~> 0.1.4)
ruby-progressbar (~> 0.0.9)
+ hashie (1.0.0)
i18n (0.5.0)
infinity_test (1.0.2)
notifiers (>= 1.1.0)
@@ -49,6 +58,8 @@ GEM
nokogiri (>= 1.2.1)
mime-types (1.16)
mocha (0.9.12)
+ multi_json (0.0.5)
+ multi_xml (0.2.1)
multipart-post (1.1.0)
net-scp (1.0.4)
net-ssh (>= 1.99.1)
@@ -59,6 +70,7 @@ GEM
notifiers (1.1.0)
oauth (0.4.4)
polyglot (0.3.1)
+ rack (1.2.1)
rspec (2.5.0)
rspec-core (~> 2.5.0)
rspec-expectations (~> 2.5.0)
@@ -70,10 +82,18 @@ GEM
rspec-mocks (2.5.0)
ruby-hmac (0.4.0)
ruby-progressbar (0.0.9)
+ simple_oauth (0.1.4)
thor (0.14.6)
timecop (0.3.5)
treetop (1.4.9)
polyglot (>= 0.3.1)
+ twitter (1.1.2)
+ faraday (~> 0.5.4)
+ faraday_middleware (~> 0.3.1)
+ hashie (~> 1.0.0)
+ multi_json (~> 0.0.5)
+ multi_xml (~> 0.2.0)
+ simple_oauth (~> 0.1.3)
watchr (0.7)
PLATFORMS
View
@@ -1,12 +1,12 @@
Backup 3
========
-Backup is a RubyGem (for UNIX-like operating systems: Linux, Mac OSX) that allows you to configure and perform backups in a simple manner using an elegant Ruby DSL. It supports various databases (MySQL, PostgreSQL, MongoDB and Redis), it supports various storage locations (Amazon S3, Rackspace Cloud Files, Dropbox, any remote server through FTP, SFTP, SCP and RSync), it can archive files and folders, it can cycle backups, it can do incremental backups, it can compress backups, it can encrypt backups (OpenSSL or GPG), it can notify you about successful and/or failed backups. It is very extensible and easy to add new functionality to. It's easy to use.
+Backup is a RubyGem (for UNIX-like operating systems: Linux, Mac OSX) that allows you to configure and perform backups in a simple manner using an elegant Ruby DSL. It supports various databases (MySQL, PostgreSQL, MongoDB and Redis), it supports various storage locations (Amazon S3, Rackspace Cloud Files, Dropbox, any remote server through FTP, SFTP, SCP and RSync), it can archive files and folders, it can cycle backups, it can do incremental backups, it can compress backups, it can encrypt backups (OpenSSL or GPG), it can notify you about successful and/or failed backups (Email or Twitter). It is very extensible and easy to add new functionality to. It's easy to use.
Author
------
-Michael van Rooijen ( [@meskyanichi](http://twitter.com/#!/meskyanichi) )
+**Michael van Rooijen ( [@meskyanichi](http://twitter.com/#!/meskyanichi) )**
Drop me a message for any questions, suggestions, requests, bugs or submit them to the [issue log](https://github.com/meskyanichi/backup/issues).
@@ -17,14 +17,12 @@ To get the latest stable version
gem install backup
-To get the latest *build* of the latest stable version
-
- gem install backup --pre
-
-Builds **aim** to be stable, but cannot guarantee it. Builds tend to be released a lot more frequent than the stable versions. So if you want to live on the edge and want the latest improvements, install the build gems.
-
You can view the list of released versions over at [RubyGems.org (Backup)](https://rubygems.org/gems/backup/versions)
+Getting Started
+---------------
+
+I recommend you read this README first, and refer to the [Wiki pages](https://github.com/meskyanichi/backup/wiki) afterwards. There's also a [Getting Started wiki page](https://github.com/meskyanichi/backup/wiki/Getting-Started).
What Backup 3 currently supports
================================
@@ -72,6 +70,13 @@ Storage Features
[Storage Wiki Page](https://github.com/meskyanichi/backup/wiki/Storages)
+Syncers
+-------
+
+- RSync
+
+[Syncer Wiki Page](https://github.com/meskyanichi/backup/wiki/Syncers)
+
Compressors
-----------
@@ -91,6 +96,7 @@ Notifiers
---------
- Mail
+- Twitter
[Notifiers Wiki Page](https://github.com/meskyanichi/backup/wiki/Notifiers)
@@ -158,34 +164,50 @@ Below you see a sample configuration file you could create for Backup 3. Just re
s3.keep = 20
end
- store_with RSync do |server|
- server.username = 'my_username'
- server.password = 'my_password'
- server.ip = '123.45.678.90'
- server.path = '~/backups/'
+ sync_with RSync do |rsync|
+ rsync.ip = "123.45.678.90"
+ rsync.username = "my_username"
+ rsync.path = "~/backups/"
+ rsync.mirror = true
+ rsync.compress = true
+
+ rsync.directories do |directory|
+ directory.add "/var/apps/my_app/public/videos"
+ directory.add "/var/apps/my_app/public/music"
+ end
end
notify_by Mail do |mail|
mail.on_success = false
mail.on_failure = true
end
+
+ notify_by Twitter do |tweet|
+ tweet.on_success = true
+ tweet.on_failure = true
+ end
+
end
### Explanation for the above example
-First it dumps all the tables inside the MySQL database "my_sample_mysql_db", except for the "logs" table. It also dumps the MongoDB database "my_sample_mongo_db", but only the collections "users", "events" and "posts". After that it'll create a "user_avatars.tar" archive with all the uploaded avatars of the users. After that it'll create a "logs.tar" archive with the "production.log", "newrelic_agent.log" and "other.log" logs. After that it'll compress the backup file using Gzip (with the mode set to "best", rather than "fast" for best compression). After that it'll encrypt the whole backup file (everything included: databases, archives) using "OpenSSL". Now the Backup can only be extracted when you know the password to decrypt it ("my_secret_password" in this case). Then it'll store the backup file to Amazon S3 in to 'my_bucket/backups'. Next it'll also transfer a copy of the backup file to a remote server using the RSync protocol, and it'll be stored in to the "$HOME/backups/" path on this server. Finally, it'll notify me by email if the backup raises an error/exception during the process indicating that something went wrong. (When setting `mail.on_success = true` it'll also notify you of every successful backup)
+First it dumps all the tables inside the MySQL database "my_sample_mysql_db", except for the "logs" table. It also dumps the MongoDB database "my_sample_mongo_db", but only the collections "users", "events" and "posts". After that it'll create a "user_avatars.tar" archive with all the uploaded avatars of the users. After that it'll create a "logs.tar" archive with the "production.log", "newrelic_agent.log" and "other.log" logs. After that it'll compress the backup file using Gzip (with the mode set to "best", rather than "fast" for best compression). After that it'll encrypt the whole backup file (everything included: databases, archives) using "OpenSSL". Now the Backup can only be extracted when you know the password to decrypt it ("my_secret_password" in this case). Then it'll store the backup file to Amazon S3 in to 'my_bucket/backups'. Next, we're going to use the RSync Syncer to create a mirror of the `/var/apps/my_app/public/videos` and `/var/apps/my_app/public/music` folders on a remote server. (This will not package, compress, encrypt - but will directly sync these folders "as is" to the desired location). Finally, it'll notify me by email if the backup raises an error/exception during the process, indicating that something went wrong. However, it does not notify me by email when successful backups occur because I set `mail.on_success` to `false`. It'll also notify me by Twitter when failed backups occur, but also when successful ones occur because I set the `tweet.on_success` to `true`.
### Things to note
The __keep__ option I passed in to the S3 storage location enables "Backup Cycling". In this case, after the 21st backup file gets pushed, it'll exceed the 20 backup limit, and remove the oldest backup from the S3 bucket.
-The __RSync__ protocol doesn't utilize the __keep__ option. RSync is used to do incremental backups, and only stores a single file on your remote server, which gets incrementally updated with each run. For example, if everything you dump ends up to be about 2000MB, the first time, you'll be transferring the full 2000MB. If by the time the next backup run starts this dump has increased to 2100MB, it'll calculate the difference between the source and destination file and only transfer the remaining 100MB, rather than the full 2100MB. (Note: To reduce bandwidth as much as possible with RSync, ensure you don't use compression or encryption, otherwise RSync isn't able to calculate the difference very well and bandwidth usage greatly increases.)
+The __RSync__ Syncer ( `sync_with` ) is a different kind of __Storage__ method. As mentioned above, it does not follow the same procedure as the __Storage__ ( `store_with` ) method. A Storage method stores the final result of a copied/organized/packaged/compressed/encrypted file to the desired remote location. A Syncer directly syncs the specified directories and **completely bypasses** the copied/organized/packaged/compressed/encrypted process. This is especially good for backing up directories containing gigabytes of data, such as images, music, videos, and similar large formats. Also, rather than transferring the whole directory every time, it'll only transfer files in all these directories that have been modified or new ones that have been added, and it only transfers the bytes of the modified files that changed, and **not** the full file, thus, saving huge amounts of bandwidth, cpu load, time and possibly money.
The __Mail__ notifier. I have not provided the SMTP options to use my Gmail account to notify myself when exceptions are raised during the process. So this won't work, check out the wiki on how to configure this. I left it out in this example.
+The __Twitter__ notifier. You will require your consumer and oauth credentials, which I have also left out of this example.
+
+Check out the Wiki for more information on all the above subjects.
+
### And that's it!
-So as you can see the DSL is straightforward and should be simple to understand and extend to your needs. You can have as many databases, archives, storage locations, compressors, encryptors and notifiers inside the above example as you need and it'll bundle all of it up in a nice packaged archive and transfer it to every specified location (as redundant as you like).
+So as you can see the DSL is straightforward and should be simple to understand and extend to your needs. You can have as many databases, archives, storage locations, syncers, compressors, encryptors and notifiers inside the above example as you need and it'll bundle all of it up in a nice packaged archive and transfer it to every specified location (as redundant as you like).
### Running the example
@@ -216,6 +238,19 @@ Suggestions, Bugs, Requests, Questions
View the [issue log](https://github.com/meskyanichi/backup/issues) and post them there.
+Contributors
+------------
+
+<table>
+ <tr>
+ <th>Contributor</th>
+ <th>Contribution</th>
+ </tr>
+ <tr>
+ <td><a href="https://github.com/asanghi" target="_blank">Aditya Sanghi ( asanghi )</a></td>
+ <td>Twitter Notifier</td>
+ </tr>
+</table>
Want to contribute?
-------------------
View
@@ -7,7 +7,7 @@ Gem::Specification.new do |gem|
##
# General configuration / information
gem.name = 'backup'
- gem.version = Backup::Version.gemspec
+ gem.version = Backup::Version.current
gem.platform = Gem::Platform::RUBY
gem.authors = 'Michael van Rooijen'
gem.email = 'meskyanichi@gmail.com'
@@ -17,7 +17,7 @@ Gem::Specification.new do |gem|
It supports various databases (MySQL, PostgreSQL, MongoDB and Redis), it supports various storage locations
(Amazon S3, Rackspace Cloud Files, Dropbox, any remote server through FTP, SFTP, SCP and RSync), it can archive files and folders,
it can cycle backups, it can do incremental backups, it can compress backups, it can encrypt backups (OpenSSL or GPG),
- it can notify you about successful and/or failed backups. It is very extensible and easy to add new functionality to. It\'s easy to use.'
+ it can notify you about successful and/or failed backups (Email, Twitter). It is very extensible and easy to add new functionality to. It\'s easy to use.'
##
# Files and folder that need to be compiled in to the Ruby Gem
@@ -37,5 +37,6 @@ Gem::Specification.new do |gem|
gem.add_dependency 'mail', ['~> 2.2.15'] # Mail
gem.add_dependency 'net-sftp', ['~> 2.0.5' ] # SFTP Protocol
gem.add_dependency 'net-scp', ['~> 1.0.4' ] # SCP Protocol
+ gem.add_dependency 'twitter', ['~> 1.1.2' ] # Twitter
end
View
@@ -20,8 +20,8 @@ module Backup
STORAGES = ['S3', 'CloudFiles', 'Dropbox', 'FTP', 'SFTP', 'SCP', 'RSync']
COMPRESSORS = ['Gzip']
ENCRYPTORS = ['OpenSSL', 'GPG']
- NOTIFIERS = ['Mail']
SYNCERS = ['RSync']
+ NOTIFIERS = ['Mail', 'Twitter']
##
# Backup's internal paths
@@ -58,8 +58,9 @@ module Configuration
autoload :Helpers, File.join(CONFIGURATION_PATH, 'helpers')
module Notifier
- autoload :Base, File.join(CONFIGURATION_PATH, 'notifier', 'base')
- autoload :Mail, File.join(CONFIGURATION_PATH, 'notifier', 'mail')
+ autoload :Base, File.join(CONFIGURATION_PATH, 'notifier', 'base')
+ autoload :Mail, File.join(CONFIGURATION_PATH, 'notifier', 'mail')
+ autoload :Twitter, File.join(CONFIGURATION_PATH, 'notifier', 'twitter')
end
module Encryptor
@@ -145,9 +146,10 @@ module Encryptor
##
# Autoload notification files
module Notifier
- autoload :Base, File.join(NOTIFIER_PATH, 'base')
- autoload :Binder, File.join(NOTIFIER_PATH, 'binder')
- autoload :Mail, File.join(NOTIFIER_PATH, 'mail')
+ autoload :Base, File.join(NOTIFIER_PATH, 'base')
+ autoload :Binder, File.join(NOTIFIER_PATH, 'binder')
+ autoload :Mail, File.join(NOTIFIER_PATH, 'mail')
+ autoload :Twitter, File.join(NOTIFIER_PATH, 'twitter')
end
##
@@ -0,0 +1,21 @@
+# encoding: utf-8
+
+module Backup
+ module Configuration
+ module Notifier
+ class Twitter < Base
+ class << self
+
+ ##
+ # Twitter consumer key credentials
+ attr_accessor :consumer_key, :consumer_secret
+
+ ##
+ # OAuth credentials
+ attr_accessor :oauth_token, :oauth_token_secret
+
+ end
+ end
+ end
+ end
+end
@@ -19,15 +19,15 @@ class << self
attr_accessor :port
##
- # Files/Folders to Sync
- attr_accessor :folders
+ # Directories to sync
+ attr_accessor :directories
##
- # Path to store the synced files/folders to
+ # Path to store the synced files/directories to
attr_accessor :path
##
- # Flag for mirroring the files/folders
+ # Flag for mirroring the files/directories
attr_accessor :mirror
##
Oops, something went wrong.

0 comments on commit 6f90c1a

Please sign in to comment.