update Cloud Syncers

- Syncer::Cloud namespace `sync_with S3` -> `sync_with Cloud::S3` `sync_with CloudFiles` -> `sync_with Cloud::CloudFiles` - Warn user if paths contain invalid UTF-8 byte sequences (#288)
backup · Mar 10, 2012 · 603c46a · 603c46a
1 parent e38b5a8
commit 603c46a
Show file tree

Hide file tree

Showing 39 changed files with 1,604 additions and 958 deletions.
diff --git a/README.md b/README.md
@@ -69,20 +69,22 @@ Below you find a list of components that Backup currently supports. If you'd lik
   - Dropbox Web Service
   - Remote Servers *(Only Protocols: FTP, SFTP, SCP)*
   - Local Storage
+
+[Cycling Wiki Page](https://github.com/meskyanichi/backup/wiki/Cycling)
+
 - **Backup Splitting, applies to:**
   - Amazon Simple Storage Service (S3)
   - Rackspace Cloud Files (Mosso)
   - Ninefold Cloud Storage
   - Dropbox Web Service
   - Remote Servers *(Only Protocols: FTP, SFTP, SCP)*
   - Local Storage
-- **Incremental Backups, applies to:**
-  - Remote Servers *(Only Protocols: RSync)*
-
-[Cycling Wiki Page](https://github.com/meskyanichi/backup/wiki/Cycling)
 
 [Splitter Wiki Page](https://github.com/meskyanichi/backup/wiki/Splitter)
 
+- **Incremental Backups, applies to:**
+  - Remote Servers *(Only Protocols: RSync)*
+
 ### Syncers
 
 - RSync (Push, Pull and Local)
@@ -128,7 +130,8 @@ Below you find a list of components that Backup currently supports. If you'd lik
 A sample Backup configuration file
 ----------------------------------
 
-This is a Backup configuration file. Check it out and read the explanation below. Backup has a [great wiki](https://github.com/meskyanichi/backup/wiki) which explains each component of Backup in detail.
+This is a Backup configuration file. Check it out and read the explanation below.  
+Backup has a [great wiki](https://github.com/meskyanichi/backup/wiki) which explains each component of Backup in detail.
 
 ``` rb
 Backup::Model.new(:sample_backup, 'A sample backup configuration') do
@@ -193,7 +196,7 @@ Backup::Model.new(:sample_backup, 'A sample backup configuration') do
     s3.keep               = 20
   end
 
-  sync_with S3 do |s3|
+  sync_with Cloud::S3 do |s3|
     s3.access_key_id     = "my_access_key_id"
     s3.secret_access_key = "my_secret_access_key"
     s3.bucket            = "my-bucket"
@@ -252,7 +255,7 @@ of `-aa`, `-ab` and `-ac`. These files will then be individually transfered. Thi
 Amazon S3, Rackspace Cloud Files, or other 3rd party storage services which limit you to "5GB per file" uploads. So with
 this, the backup file size is no longer a constraint.
 
-Additionally we have also defined a **S3 Syncer** ( `sync_with S3` ), which does not follow the above process of
+Additionally we have also defined a **S3 Syncer** ( `sync_with Cloud::S3` ), which does not follow the above process of
 archiving/compression/encryption, but instead will directly sync the whole `videos` and `music` folder structures from
 your machine to your Amazon S3 account. (very efficient and cost-effective since it will only transfer files that were
 added/changed. Additionally, since we flagged it to 'mirror', it'll also remove files from S3 that no longer exist). If

diff --git a/lib/backup.rb b/lib/backup.rb
@@ -62,10 +62,12 @@ module Storage
   ##
   # Autoload Backup syncer files
   module Syncer
-    autoload :Base,       File.join(SYNCER_PATH, 'base')
-    autoload :Cloud,      File.join(SYNCER_PATH, 'cloud')
-    autoload :CloudFiles, File.join(SYNCER_PATH, 'cloud_files')
-    autoload :S3,         File.join(SYNCER_PATH, 's3')
+    autoload :Base, File.join(SYNCER_PATH, 'base')
+    module Cloud
+      autoload :Base,       File.join(SYNCER_PATH, 'cloud', 'base')
+      autoload :CloudFiles, File.join(SYNCER_PATH, 'cloud', 'cloud_files')
+      autoload :S3,         File.join(SYNCER_PATH, 'cloud', 's3')
+    end
     module RSync
       autoload :Base,  File.join(SYNCER_PATH, 'rsync', 'base')
       autoload :Local, File.join(SYNCER_PATH, 'rsync', 'local')
@@ -160,10 +162,12 @@ module Storage
     end
 
     module Syncer
-      autoload :Base,       File.join(CONFIGURATION_PATH, 'syncer', 'base')
-      autoload :Cloud,      File.join(CONFIGURATION_PATH, 'syncer', 'cloud')
-      autoload :CloudFiles, File.join(CONFIGURATION_PATH, 'syncer', 'cloud_files')
-      autoload :S3,         File.join(CONFIGURATION_PATH, 'syncer', 's3')
+      autoload :Base, File.join(CONFIGURATION_PATH, 'syncer', 'base')
+      module Cloud
+        autoload :Base,       File.join(CONFIGURATION_PATH, 'syncer', 'cloud', 'base')
+        autoload :CloudFiles, File.join(CONFIGURATION_PATH, 'syncer', 'cloud', 'cloud_files')
+        autoload :S3,         File.join(CONFIGURATION_PATH, 'syncer', 'cloud', 's3')
+      end
       module RSync
         autoload :Base,  File.join(CONFIGURATION_PATH, 'syncer', 'rsync', 'base')
         autoload :Local, File.join(CONFIGURATION_PATH, 'syncer', 'rsync', 'local')

diff --git a/lib/backup/config.rb b/lib/backup/config.rb
@@ -111,7 +111,10 @@ def add_dsl_constants!
             # Encryptors
             ['OpenSSL', 'GPG'],
             # Syncers
-            ['Rackspace', 'S3', { 'RSync' => ['Push', 'Pull', 'Local'] }],
+            [
+              { 'Cloud' => ['CloudFiles', 'S3'] },
+              { 'RSync' => ['Push', 'Pull', 'Local'] }
+            ],
             # Notifiers
             ['Mail', 'Twitter', 'Campfire', 'Presently', 'Prowl', 'Hipchat']
           ]

diff --git a/lib/backup/configuration/syncer/base.rb b/lib/backup/configuration/syncer/base.rb
@@ -3,7 +3,19 @@
 module Backup
   module Configuration
     module Syncer
-      class Base < Configuration::Base; end
+      class Base < Configuration::Base
+        class << self
+
+          ##
+          # Path to store the synced files/directories to
+          attr_accessor :path
+
+          ##
+          # Flag for mirroring the files/directories
+          attr_accessor :mirror
+
+        end
+      end
     end
   end
 end

diff --git a/lib/backup/configuration/syncer/cloud.rb b/lib/backup/configuration/syncer/cloud.rb
diff --git a/lib/backup/configuration/syncer/cloud/base.rb b/lib/backup/configuration/syncer/cloud/base.rb
@@ -0,0 +1,26 @@
+# encoding: utf-8
+
+module Backup
+  module Configuration
+    module Syncer
+      module Cloud
+        class Base < Syncer::Base
+          class << self
+
+            ##
+            # Concurrency setting - defaults to false, but can be set to:
+            # - :threads
+            # - :processes
+            attr_accessor :concurrency_type
+
+            ##
+            # Concurrency level - the number of threads or processors to use.
+            # Defaults to 2.
+            attr_accessor :concurrency_level
+
+          end
+        end
+      end
+    end
+  end
+end
diff --git a/lib/backup/configuration/syncer/cloud/cloud_files.rb b/lib/backup/configuration/syncer/cloud/cloud_files.rb
@@ -0,0 +1,36 @@
+# encoding: utf-8
+
+module Backup
+  module Configuration
+    module Syncer
+      module Cloud
+        class CloudFiles < Base
+          class << self
+
+            ##
+            # Rackspace CloudFiles Credentials
+            attr_accessor :api_key, :username
+
+            ##
+            # Rackspace CloudFiles Container
+            attr_accessor :container
+
+            ##
+            # Rackspace AuthURL allows you to connect
+            # to a different Rackspace datacenter
+            # - https://auth.api.rackspacecloud.com     (Default: US)
+            # - https://lon.auth.api.rackspacecloud.com (UK)
+            attr_accessor :auth_url
+
+            ##
+            # Improve performance and avoid data transfer costs
+            # by setting @servicenet to `true`
+            # This only works if Backup runs on a Rackspace server
+            attr_accessor :servicenet
+
+          end
+        end
+      end
+    end
+  end
+end
diff --git a/lib/backup/configuration/syncer/cloud/s3.rb b/lib/backup/configuration/syncer/cloud/s3.rb
@@ -0,0 +1,27 @@
+# encoding: utf-8
+
+module Backup
+  module Configuration
+    module Syncer
+      module Cloud
+        class S3 < Base
+          class << self
+
+            ##
+            # Amazon Simple Storage Service (S3) Credentials
+            attr_accessor :access_key_id, :secret_access_key
+
+            ##
+            # The S3 bucket to store files to
+            attr_accessor :bucket
+
+            ##
+            # The AWS region of the specified S3 bucket
+            attr_accessor :region
+
+          end
+        end
+      end
+    end
+  end
+end
diff --git a/lib/backup/configuration/syncer/cloud_files.rb b/lib/backup/configuration/syncer/cloud_files.rb
diff --git a/lib/backup/configuration/syncer/rsync/base.rb b/lib/backup/configuration/syncer/rsync/base.rb
@@ -7,14 +7,6 @@ module RSync
         class Base < Syncer::Base
           class << self
 
-            ##
-            # Path to store the synced files/directories to
-            attr_accessor :path
-
-            ##
-            # Flag for mirroring the files/directories
-            attr_accessor :mirror
-
             ##
             # Additional options for the rsync cli
             attr_accessor :additional_options

diff --git a/lib/backup/configuration/syncer/s3.rb b/lib/backup/configuration/syncer/s3.rb
diff --git a/lib/backup/model.rb b/lib/backup/model.rb
@@ -125,16 +125,25 @@ def store_with(name, storage_id = nil, &block)
     # methods to use during the backup process
     def sync_with(name, &block)
       ##
-      # Warn user of DSL change from 'RSync' to 'RSync::Local'
-      if name.to_s == 'Backup::Config::RSync'
+      # Warn user of DSL changes
+      case name.to_s
+      when 'Backup::Config::RSync'
         Logger.warn Errors::ConfigError.new(<<-EOS)
           Configuration Update Needed for Syncer::RSync
           The RSync Syncer has been split into three separate modules:
           RSync::Local, RSync::Push and RSync::Pull
-          Please update your configuration for your local RSync Syncer
-          from 'sync_with RSync do ...' to 'sync_with RSync::Local do ...'
+          Please update your configuration.
+          i.e. 'sync_with RSync' is now 'sync_with RSync::Push'
         EOS
-        name = Backup::Config::RSync::Local
+        name = 'RSync::Push'
+      when /(Backup::Config::S3|Backup::Config::CloudFiles)/
+        syncer = $1.split('::')[2]
+        Logger.warn Errors::ConfigError.new(<<-EOS)
+          Configuration Update Needed for '#{ syncer }' Syncer.
+          This Syncer is now referenced as Cloud::#{ syncer }
+          i.e. 'sync_with #{ syncer }' is now 'sync_with Cloud::#{ syncer }'
+        EOS
+        name = "Cloud::#{ syncer }"
       end
       @syncers << get_class_from_scope(Syncer, name).new(&block)
     end

diff --git a/lib/backup/syncer/base.rb b/lib/backup/syncer/base.rb
@@ -6,10 +6,6 @@ class Base
       include Backup::CLI::Helpers
       include Backup::Configuration::Helpers
 
-      ##
-      # Directories to sync
-      attr_accessor :directories
-
       ##
       # Path to store the synced files/directories to
       attr_accessor :path
@@ -18,6 +14,14 @@ class Base
       # Flag for mirroring the files/directories
       attr_accessor :mirror
 
+      def initialize
+        load_defaults!
+
+        @path               ||= 'backups'
+        @mirror             ||= false
+        @directories          = Array.new
+      end
+
       ##
       # Syntactical suger for the DSL for adding directories
       def directories(&block)