Skip to content

Commit

Permalink
Pipelines and more...
Browse files Browse the repository at this point in the history 
- Use pipelines to:
  - Add compression to Archives
  - Add compression to Databases, where possible
  - Add encryption and splitter to final packaging
- Models only support one Compressor and/or one Encryptor
- Update Dropbox to use 'dropbox-sdk' gem
  - Adds 'access_type' setting (default: 'app_folder')
  - Deprecate 'timeout' setting
- Final packaged archive structure updated
  - Archives are stored in <trigger>/archives
  - Databases are stored in <trigger>/databases
  - see wiki for more details...
- Deprecate Database#utility_path
  - Each Database now uses it's own #<utility_name>_utility method
    to override the full path to the named utility
- Encryptor::OpenSSL now sets @salt to true by default
- Cleanup procedures updated
  - If errors occur, temporary files will no longer be removed
    until the next time the failed model/trigger is run.
  - see wiki for details...
- Storage::Local now performs a 'move' instead of a 'copy' to transfer
  the final archive package files to their destination, if possible.
- Backup's log file will now keep itself truncated to 500KB

This is a rather large single commit. The changes to add the pipeline
capabilities had a widespread effect which resulted in other internal
changes and refactorings. As a result, all the specs have been updated.
  • Loading branch information
Brian D. Burns committed Jan 12, 2012
1 parent ccfe4a1 commit 9be16f7
Show file tree
Hide file tree
Showing 131 changed files with 8,867 additions and 5,463 deletions.
6 changes: 1 addition & 5 deletions Gemfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,8 @@ source 'http://rubygems.org'
# Include gem dependencies from the gemspec for development purposes
gemspec

# Load Backup::Dependency
["cli/helpers", "dependency"].each do |library|
require File.expand_path("../lib/backup/#{library}", __FILE__)
end

# Dynamically define the dependencies specified in Backup::Dependency.all
require File.expand_path("../lib/backup/dependency", __FILE__)
Backup::Dependency.all.each do |name, gemspec|
gem(name, gemspec[:version])
end
Expand Down
92 changes: 44 additions & 48 deletions Gemfile.lock
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,82 +12,80 @@ GEM
Platform (>= 0.4.0)
open4
Platform (0.4.0)
activesupport (3.1.3)
multi_json (~> 1.0)
addressable (2.2.6)
builder (3.0.0)
crack (0.1.8)
diff-lcs (1.1.2)
dropbox (1.3.0)
json (>= 1.2.0)
multipart-post (>= 1.1.0)
oauth (>= 0.3.6)
excon (0.6.6)
faraday (0.7.4)
diff-lcs (1.1.3)
dropbox-sdk (1.1)
json
excon (0.9.4)
faraday (0.7.5)
addressable (~> 2.2.6)
multipart-post (~> 1.1.0)
multipart-post (~> 1.1.3)
rack (>= 1.1.0, < 2)
faraday_middleware (0.7.0)
faraday (~> 0.7.3)
ffi (1.0.9)
fog (0.11.0)
ffi (1.0.11)
fog (1.1.2)
builder
excon (~> 0.6.5)
excon (~> 0.9.0)
formatador (~> 0.2.0)
mime-types
multi_json (~> 1.0.3)
net-scp (~> 1.0.4)
net-ssh (~> 2.1.4)
net-ssh (>= 2.1.3)
nokogiri (~> 1.5.0)
ruby-hmac
formatador (0.2.1)
fuubar (0.0.5)
fuubar (0.0.6)
rspec (~> 2.0)
rspec-instafail (~> 0.1.4)
rspec-instafail (~> 0.1.8)
ruby-progressbar (~> 0.0.10)
growl (1.0.3)
guard (0.3.4)
guard (0.10.0)
ffi (>= 0.5.0)
thor (~> 0.14.6)
guard-rspec (0.3.1)
guard (>= 0.2.2)
hashie (1.1.0)
guard-rspec (0.6.0)
guard (>= 0.10.0)
hipchat (0.4.1)
httparty
httparty (0.7.4)
httparty (0.7.8)
crack (= 0.1.8)
i18n (0.6.0)
json (1.5.4)
libnotify (0.5.5)
libnotify (0.7.1)
mail (2.3.0)
i18n (>= 0.4.0)
mime-types (~> 1.16)
treetop (~> 1.4.8)
mime-types (1.16)
mocha (0.9.12)
multi_json (1.0.3)
multi_xml (0.3.0)
multipart-post (1.1.3)
metaclass (0.0.1)
mime-types (1.17.2)
mocha (0.10.0)
metaclass (~> 0.0.1)
multi_json (1.0.4)
multipart-post (1.1.4)
net-scp (1.0.4)
net-ssh (>= 1.99.1)
net-sftp (2.0.5)
net-ssh (>= 2.0.9)
net-ssh (2.1.4)
nokogiri (1.5.0)
oauth (0.4.5)
open4 (1.3.0)
polyglot (0.3.2)
polyglot (0.3.3)
prowler (1.3.1)
rack (1.3.2)
rb-fsevent (0.4.0)
rb-inotify (0.8.5)
rack (1.4.0)
rb-fsevent (0.4.3.1)
rb-inotify (0.8.8)
ffi (>= 0.5.0)
rspec (2.5.0)
rspec-core (~> 2.5.0)
rspec-expectations (~> 2.5.0)
rspec-mocks (~> 2.5.0)
rspec-core (2.5.1)
rspec-expectations (2.5.0)
rspec (2.8.0)
rspec-core (~> 2.8.0)
rspec-expectations (~> 2.8.0)
rspec-mocks (~> 2.8.0)
rspec-core (2.8.0)
rspec-expectations (2.8.0)
diff-lcs (~> 1.1.2)
rspec-instafail (0.1.7)
rspec-mocks (2.5.0)
rspec-instafail (0.1.9)
rspec-mocks (2.8.0)
ruby-hmac (0.4.0)
ruby-progressbar (0.0.10)
simple_oauth (0.1.5)
Expand All @@ -96,20 +94,18 @@ GEM
treetop (1.4.10)
polyglot
polyglot (>= 0.3.1)
twitter (1.7.1)
faraday (~> 0.7.4)
faraday_middleware (~> 0.7.0)
hashie (~> 1.1.0)
multi_json (~> 1.0.0)
multi_xml (~> 0.3.0)
simple_oauth (~> 0.1.5)
twitter (2.0.2)
activesupport (>= 2.3.9, < 4)
faraday (~> 0.7)
multi_json (~> 1.0)
simple_oauth (~> 0.1)

PLATFORMS
ruby

DEPENDENCIES
backup!
dropbox (~> 1.3.0)
dropbox-sdk (~> 1.1.0)
fog (>= 0.11.0)
fuubar
growl
Expand Down
79 changes: 53 additions & 26 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,6 @@ Author
Drop me a message for any questions, suggestions, requests, bugs or submit them to the [issue log](https://github.com/meskyanichi/backup/issues).


Core Contributor
----------------

**[Brian Burns](https://github.com/burns)**


Installation
------------

Expand Down Expand Up @@ -92,7 +86,7 @@ Storage Features
- **Incremental Backups, applies to:**
- Remote Servers *(Only Protocols: RSync)*

[Storage Wiki Page](https://github.com/meskyanichi/backup/wiki/Storages)
[Cycling Wiki Page](https://github.com/meskyanichi/backup/wiki/Cycling)

[Splitter Wiki Page](https://github.com/meskyanichi/backup/wiki/Splitter)

Expand Down Expand Up @@ -240,21 +234,50 @@ end

### Brief explanation for the above example configuration

It will dump two databases (MySQL and MongoDB), it'll create two (.t)archives (user_avatars and logs). It'll package the two database and two archives together in a single (.t)archive. It'll run the Gzip compressor to compress that archive, and then it'll run the OpenSSL encryptor to encrypt the compressed archive. Then that encrypted archive will be stored to your Amazon S3 account. If all goes well, and no exceptions are raised, you'll be notified via the Twitter notifier that the backup succeeded. If there was an exception raised during the backup process, then you'd receive an email in your inbox containing detailed exception information, as well as receive a simple Twitter message that something went wrong.

Aside of S3, we have also defined two `SFTP` storage methods, and given them two unique identifiers `Server A` and `Server B` to distinguish between the two. With these in place, a copy of the backup will now also be stored on two separate servers: `a.my-backup-server.com` and `b.my-backup-server.com`.

As you can see, you can freely mix and match **archives**, **databases**, **compressors**, **encryptors**, **storages** and **notifiers** for your backups. You could even specify 4 storage locations if you wanted: Amazon S3, Rackspace Cloud Files, Ninefold and Dropbox, it'd then store your packaged backup to 4 separate locations for high redundancy. This also applies to compressors (like Gzip, Bzip2, Lzma) and encryptors, you could double encrypt your backup with OpenSSL followed by GPG if you wanted.

Also, notice the `split_into_chunks_of 4000` at the top of the configuration. This tells Backup to split any backups that exceed in 4000 MEGABYTES of size in to multiple smaller chunks. Assuming your backup file is 12000 MEGABYTES (12GB) in size, then Backup will go ahead and split it in to 3 chunks of 4000 MEGABYTES and transfer them individually. This is useful for when you are using Amazon S3, Rackspace Cloud Files, or other 3rd party storage services which limit you to "5GB per file" uploads. So with this, the backup file size is no longer a constraint.

Additionally we have also defined a **S3 Syncer** ( `sync_with S3` ), which does not follow the above process of archiving/compression/encryption, but instead will directly sync the whole `videos` and `music` folder structures from your machine to your Amazon S3 account. (very efficient and cost-effective since it will only transfer files that were added/changed. Additionally, since we flagged it to 'mirror', it'll also remove files from S3 that no longer exist). If you simply wanted to sync to a separate backup server that you own, you could also use the RSync syncer for even more efficient backups that only transfer the **bytes** of each file that changed.

There are more **archives**, **databases**, **compressors**, **encryptors**, **storages** and **notifiers** than displayed in the example, all available components are listed at the top of this README, as well as in the [Wiki](https://github.com/meskyanichi/backup/wiki) with more detailed information.
First, it will dump the two Databases (MySQL and MongoDB). The MySQL dump will be piped through the Gzip Compressor into
`sample_backup/databases/MySQL/my_sample_mysql_db.sql.gz`. The MongoDB dump will be dumped into
`sample_backup/databases/MongoDB/`, which will then be packaged into `sample_backup/databases/MongoDB-#####.tar.gz`
(`#####` will be a simple unique identifier, in case multiple dumps are performed.)
Next, it will create two _tar_ Archives (user_avatars and logs). Each will be piped through the Gzip Compressor into
`sample_backup/archives/` as `user_archives.tar.gz` and `logs.tar.gz`.
Finally, the `sample_backup` directory will be packaged into an uncompressed _tar_ archive, which will be piped through
the OpenSSL Encryptor to encrypt this final package into `YYYY-MM-DD-hh-mm-ss.sample_backup.tar.enc`. This final
encrypted archive will then be transfered to your Amazon S3 account. If all goes well, and no exceptions are raised,
you'll be notified via the Twitter notifier that the backup succeeded. If any warnings were issued or there was an
exception raised during the backup process, then you'd receive an email in your inbox containing detailed exception
information, as well as receive a simple Twitter message that something went wrong.

Aside of S3, we have also defined two `SFTP` storage methods, and given them two unique identifiers `Server A` and
`Server B` to distinguish between the two. With these in place, a copy of the backup will now also be stored on two
separate servers: `a.my-backup-server.com` and `b.my-backup-server.com`.

As you can see, you can freely mix and match **archives**, **databases**, **compressors**, **encryptors**, **storages**
and **notifiers** for your backups. You could even specify 4 storage locations if you wanted: Amazon S3, Rackspace Cloud
Files, Ninefold and Dropbox, it'd then store your packaged backup to 4 separate locations for high redundancy.

Also, notice the `split_into_chunks_of 4000` at the top of the configuration. This tells Backup to split any backups
that exceed in 4000 MEGABYTES of size in to multiple smaller chunks. Assuming your backup file is 12000 MEGABYTES (12GB)
in size, then Backup will take the output which was piped from _tar_ into the OpenSSL Compressor and additionally pipe
that output through the _split_ utility, which will result in 3 chunks of 4000 MEGABYTES with additional file extensions
of `-aa`, `-ab` and `-ac`. These files will then be individually transfered. This is useful for when you are using
Amazon S3, Rackspace Cloud Files, or other 3rd party storage services which limit you to "5GB per file" uploads. So with
this, the backup file size is no longer a constraint.

Additionally we have also defined a **S3 Syncer** ( `sync_with S3` ), which does not follow the above process of
archiving/compression/encryption, but instead will directly sync the whole `videos` and `music` folder structures from
your machine to your Amazon S3 account. (very efficient and cost-effective since it will only transfer files that were
added/changed. Additionally, since we flagged it to 'mirror', it'll also remove files from S3 that no longer exist). If
you simply wanted to sync to a separate backup server that you own, you could also use the RSync syncer for even more
efficient backups that only transfer the **bytes** of each file that changed.

There are more **archives**, **databases**, **compressors**, **encryptors**, **storages** and **notifiers** than
displayed in the example, all available components are listed at the top of this README, as well as in the
[Wiki](https://github.com/meskyanichi/backup/wiki) with more detailed information.

### Running the example

Notice the `Backup::Model.new(:sample_backup, 'A sample backup configuration') do` at the top of the above example. The `:sample_backup` is called the **trigger**. This is used to identify the backup procedure/file and initialize it.
Notice the `Backup::Model.new(:sample_backup, 'A sample backup configuration') do` at the top of the above example. The
`:sample_backup` is called the **trigger**. This is used to identify the backup procedure/file and initialize it.

``` sh
backup perform -t [--trigger] sample_backup
Expand All @@ -264,20 +287,24 @@ Now it'll run the backup, it's as simple as that.

### Automatic backups

Since Backup is an easy-to-use command line utility, you should write a crontask to invoke it periodically. I recommend using [Whenever](https://github.com/javan/whenever) to manage your crontab. It'll allow you to write to the crontab using pure Ruby, and it provides an elegant DSL to do so. Here's an example:
Since Backup is an easy-to-use command line utility, you should write a crontask to invoke it periodically. I recommend
using [Whenever](https://github.com/javan/whenever) to manage your crontab. It'll allow you to write to the crontab
using pure Ruby, and it provides an elegant DSL to do so. Here's an example:

``` rb
every 6.hours do
command "backup perform --trigger sample_backup"
end
```

With this in place, run `whenever --update-crontab backup` to write the equivalent of the above Ruby syntax to the crontab in cron-syntax. Cron will now invoke `backup perform --trigger sample_backup` every 6 hours. Check out the Whenever project page for more information.
With this in place, run `whenever --update-crontab backup` to write the equivalent of the above Ruby syntax to the
crontab in cron-syntax. Cron will now invoke `backup perform --trigger sample_backup` every 6 hours. Check out the
Whenever project page for more information.

Documentation
-------------

See the [Wiki Pages](https://github.com/meskyanichi/backup/wiki). The subjects labeled **without** the "Backup 2)"-prefix are meant for Backup 3 users.
See the [Wiki Pages](https://github.com/meskyanichi/backup/wiki).


Suggestions, Bugs, Requests, Questions
Expand All @@ -293,6 +320,10 @@ Contributors
<th>Contributor</th>
<th>Contribution</th>
</tr>
<tr>
<td><a href="https://github.com/burns" target="_blank"><b>Brian D. Burns ( burns )</b></a></td>
<td><b>Core Contributor</b></td>
</tr>
<tr>
<td><a href="https://github.com/asanghi" target="_blank">Aditya Sanghi ( asanghi )</a></td>
<td>Twitter Notifier, Dropbox Timeout Configuration</td>
Expand Down Expand Up @@ -409,10 +440,6 @@ Contributors
<td><a href="https://github.com/szymonpk" target="_blank">Szymon ( szymonpk )</a></td>
<td>Pbzip2 compressor</td>
</tr>
<tr>
<td><a href="https://github.com/burns" target="_blank">burns ( burns )</a></td>
<td>Improved Backup cycling implementation by refreshing all user configuration during the cycle procedure</td>
</tr>
</table>


Expand Down
59 changes: 3 additions & 56 deletions lib/backup.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,22 +23,6 @@
# The Backup Ruby Gem
module Backup

##
# List the available database, storage, compressor, encryptor and notifier constants.
# These are used to dynamically define these constants as classes inside Backup::Finder
# to provide a nicer configuration file DSL syntax to the users. Adding existing constants
# to the arrays below will enable the user to use a constant instead of a string.
# Example, instead of:
# database "MySQL" do |mysql|
# You can do:
# database MySQL do |mysql|
DATABASES = ['MySQL', 'PostgreSQL', 'MongoDB', 'Redis', 'Riak']
STORAGES = ['S3', 'CloudFiles', 'Ninefold', 'Dropbox', 'FTP', 'SFTP', 'SCP', 'RSync', 'Local']
COMPRESSORS = ['Gzip', 'Bzip2', 'Pbzip2', 'Lzma']
ENCRYPTORS = ['OpenSSL', 'GPG']
SYNCERS = ['S3', 'Rsync' => ['Push', 'Pull', 'Local']]
NOTIFIERS = ['Mail', 'Twitter', 'Campfire', 'Presently', 'Prowl', 'Hipchat']

##
# Backup's internal paths
LIBRARY_PATH = File.join(File.dirname(__FILE__), 'backup')
Expand All @@ -52,25 +36,15 @@ module Backup
CONFIGURATION_PATH = File.join(LIBRARY_PATH, 'configuration')
TEMPLATE_PATH = File.expand_path('../../templates', __FILE__)

##
# Backup's Environment paths
USER = ENV['USER'] || Etc.getpwuid.name
HOME = File.expand_path(ENV['HOME'] || '')
PATH = File.join(HOME, 'Backup')
CONFIG_FILE = File.join(PATH, 'config.rb')
DATA_PATH = File.join(PATH, 'data')
LOG_PATH = File.join(PATH, 'log')
CACHE_PATH = File.join(PATH, '.cache')
TMP_PATH = File.join(PATH, '.tmp')

##
# Autoload Backup base files
autoload :Model, File.join(LIBRARY_PATH, 'model')
autoload :Archive, File.join(LIBRARY_PATH, 'archive')
autoload :Packager, File.join(LIBRARY_PATH, 'packager')
autoload :Package, File.join(LIBRARY_PATH, 'package')
autoload :Cleaner, File.join(LIBRARY_PATH, 'cleaner')
autoload :Splitter, File.join(LIBRARY_PATH, 'splitter')
autoload :Finder, File.join(LIBRARY_PATH, 'finder')
autoload :Config, File.join(LIBRARY_PATH, 'config')
autoload :Binder, File.join(LIBRARY_PATH, 'binder')
autoload :Template, File.join(LIBRARY_PATH, 'template')
autoload :Dependency, File.join(LIBRARY_PATH, 'dependency')
Expand All @@ -89,7 +63,7 @@ module CLI
# Autoload Backup storage files
module Storage
autoload :Base, File.join(STORAGE_PATH, 'base')
autoload :Object, File.join(STORAGE_PATH, 'object')
autoload :Cycler, File.join(STORAGE_PATH, 'cycler')
autoload :S3, File.join(STORAGE_PATH, 's3')
autoload :CloudFiles, File.join(STORAGE_PATH, 'cloudfiles')
autoload :Ninefold, File.join(STORAGE_PATH, 'ninefold')
Expand Down Expand Up @@ -217,31 +191,4 @@ module Database
end
end

private

def self.create_empty_class(class_name, scope = Backup::Finder)
scope.const_set(class_name, Class.new) unless scope.const_defined?(class_name)
end

def self.get_or_create_empty_module(module_name)
if Backup::Finder.const_defined?(module_name)
return Backup::Finder.const_get(module_name)
else
return Backup::Finder.const_set(module_name, Module.new)
end
end

##
# Dynamically defines all the available database, storage, compressor, encryptor and notifier
# classes inside Backup::Finder to improve the DSL for the configuration file
(DATABASES + STORAGES + COMPRESSORS + ENCRYPTORS + NOTIFIERS + SYNCERS).each do |constant|
if constant.is_a?(Hash)
constant.each do |module_name, class_names|
mod = get_or_create_empty_module(module_name)
class_names.each{ |class_name| create_empty_class(class_name, mod) }
end
else
create_empty_class(constant)
end
end
end
Loading

0 comments on commit 9be16f7

Please sign in to comment.