Browse files

update Cloud Syncers

- Syncer::Cloud namespace
  `sync_with S3` -> `sync_with Cloud::S3`
  `sync_with CloudFiles` -> `sync_with Cloud::CloudFiles`
- Warn user if paths contain invalid UTF-8 byte sequences (#288)
  • Loading branch information...
1 parent e38b5a8 commit 603c46a7cd42855606f0d9a8e5861e0d11db4ef2 Brian D. Burns committed Mar 4, 2012
Showing with 1,604 additions and 958 deletions.
  1. +10 −7 README.md
  2. +12 −8 lib/backup.rb
  3. +4 −1 lib/backup/config.rb
  4. +13 −1 lib/backup/configuration/syncer/base.rb
  5. +0 −23 lib/backup/configuration/syncer/cloud.rb
  6. +26 −0 lib/backup/configuration/syncer/cloud/base.rb
  7. +36 −0 lib/backup/configuration/syncer/cloud/cloud_files.rb
  8. +27 −0 lib/backup/configuration/syncer/cloud/s3.rb
  9. +0 −30 lib/backup/configuration/syncer/cloud_files.rb
  10. +0 −8 lib/backup/configuration/syncer/rsync/base.rb
  11. +0 −23 lib/backup/configuration/syncer/s3.rb
  12. +14 −5 lib/backup/model.rb
  13. +8 −4 lib/backup/syncer/base.rb
  14. +0 −190 lib/backup/syncer/cloud.rb
  15. +245 −0 lib/backup/syncer/cloud/base.rb
  16. +61 −0 lib/backup/syncer/cloud/cloud_files.rb
  17. +51 −0 lib/backup/syncer/cloud/s3.rb
  18. +0 −56 lib/backup/syncer/cloud_files.rb
  19. +1 −4 lib/backup/syncer/rsync/base.rb
  20. +1 −1 lib/backup/syncer/rsync/pull.rb
  21. +0 −47 lib/backup/syncer/s3.rb
  22. +30 −0 spec/configuration/syncer/base_spec.rb
  23. +34 −0 spec/configuration/syncer/cloud/base_spec.rb
  24. +12 −13 spec/configuration/syncer/{ → cloud}/cloud_files_spec.rb
  25. +12 −13 spec/configuration/syncer/{ → cloud}/s3_spec.rb
  26. +7 −9 spec/configuration/syncer/rsync/base_spec.rb
  27. +17 −2 spec/model_spec.rb
  28. +78 −0 spec/syncer/base_spec.rb
  29. +523 −0 spec/syncer/cloud/base_spec.rb
  30. +153 −0 spec/syncer/cloud/cloud_files_spec.rb
  31. +145 −0 spec/syncer/cloud/s3_spec.rb
  32. +0 −192 spec/syncer/cloud_files_spec.rb
  33. +29 −59 spec/syncer/rsync/base_spec.rb
  34. +5 −23 spec/syncer/rsync/local_spec.rb
  35. +15 −4 spec/syncer/rsync/pull_spec.rb
  36. +0 −4 spec/syncer/rsync/push_spec.rb
  37. +0 −192 spec/syncer/s3_spec.rb
  38. +17 −19 templates/cli/utility/syncer/cloud_files
  39. +18 −20 templates/cli/utility/syncer/s3
View
17 README.md
@@ -69,20 +69,22 @@ Below you find a list of components that Backup currently supports. If you'd lik
- Dropbox Web Service
- Remote Servers *(Only Protocols: FTP, SFTP, SCP)*
- Local Storage
+
+[Cycling Wiki Page](https://github.com/meskyanichi/backup/wiki/Cycling)
+
- **Backup Splitting, applies to:**
- Amazon Simple Storage Service (S3)
- Rackspace Cloud Files (Mosso)
- Ninefold Cloud Storage
- Dropbox Web Service
- Remote Servers *(Only Protocols: FTP, SFTP, SCP)*
- Local Storage
-- **Incremental Backups, applies to:**
- - Remote Servers *(Only Protocols: RSync)*
-
-[Cycling Wiki Page](https://github.com/meskyanichi/backup/wiki/Cycling)
[Splitter Wiki Page](https://github.com/meskyanichi/backup/wiki/Splitter)
+- **Incremental Backups, applies to:**
+ - Remote Servers *(Only Protocols: RSync)*
+
### Syncers
- RSync (Push, Pull and Local)
@@ -128,7 +130,8 @@ Below you find a list of components that Backup currently supports. If you'd lik
A sample Backup configuration file
----------------------------------
-This is a Backup configuration file. Check it out and read the explanation below. Backup has a [great wiki](https://github.com/meskyanichi/backup/wiki) which explains each component of Backup in detail.
+This is a Backup configuration file. Check it out and read the explanation below.
+Backup has a [great wiki](https://github.com/meskyanichi/backup/wiki) which explains each component of Backup in detail.
``` rb
Backup::Model.new(:sample_backup, 'A sample backup configuration') do
@@ -193,7 +196,7 @@ Backup::Model.new(:sample_backup, 'A sample backup configuration') do
s3.keep = 20
end
- sync_with S3 do |s3|
+ sync_with Cloud::S3 do |s3|
s3.access_key_id = "my_access_key_id"
s3.secret_access_key = "my_secret_access_key"
s3.bucket = "my-bucket"
@@ -252,7 +255,7 @@ of `-aa`, `-ab` and `-ac`. These files will then be individually transfered. Thi
Amazon S3, Rackspace Cloud Files, or other 3rd party storage services which limit you to "5GB per file" uploads. So with
this, the backup file size is no longer a constraint.
-Additionally we have also defined a **S3 Syncer** ( `sync_with S3` ), which does not follow the above process of
+Additionally we have also defined a **S3 Syncer** ( `sync_with Cloud::S3` ), which does not follow the above process of
archiving/compression/encryption, but instead will directly sync the whole `videos` and `music` folder structures from
your machine to your Amazon S3 account. (very efficient and cost-effective since it will only transfer files that were
added/changed. Additionally, since we flagged it to 'mirror', it'll also remove files from S3 that no longer exist). If
View
20 lib/backup.rb
@@ -62,10 +62,12 @@ module Storage
##
# Autoload Backup syncer files
module Syncer
- autoload :Base, File.join(SYNCER_PATH, 'base')
- autoload :Cloud, File.join(SYNCER_PATH, 'cloud')
- autoload :CloudFiles, File.join(SYNCER_PATH, 'cloud_files')
- autoload :S3, File.join(SYNCER_PATH, 's3')
+ autoload :Base, File.join(SYNCER_PATH, 'base')
+ module Cloud
+ autoload :Base, File.join(SYNCER_PATH, 'cloud', 'base')
+ autoload :CloudFiles, File.join(SYNCER_PATH, 'cloud', 'cloud_files')
+ autoload :S3, File.join(SYNCER_PATH, 'cloud', 's3')
+ end
module RSync
autoload :Base, File.join(SYNCER_PATH, 'rsync', 'base')
autoload :Local, File.join(SYNCER_PATH, 'rsync', 'local')
@@ -160,10 +162,12 @@ module Storage
end
module Syncer
- autoload :Base, File.join(CONFIGURATION_PATH, 'syncer', 'base')
- autoload :Cloud, File.join(CONFIGURATION_PATH, 'syncer', 'cloud')
- autoload :CloudFiles, File.join(CONFIGURATION_PATH, 'syncer', 'cloud_files')
- autoload :S3, File.join(CONFIGURATION_PATH, 'syncer', 's3')
+ autoload :Base, File.join(CONFIGURATION_PATH, 'syncer', 'base')
+ module Cloud
+ autoload :Base, File.join(CONFIGURATION_PATH, 'syncer', 'cloud', 'base')
+ autoload :CloudFiles, File.join(CONFIGURATION_PATH, 'syncer', 'cloud', 'cloud_files')
+ autoload :S3, File.join(CONFIGURATION_PATH, 'syncer', 'cloud', 's3')
+ end
module RSync
autoload :Base, File.join(CONFIGURATION_PATH, 'syncer', 'rsync', 'base')
autoload :Local, File.join(CONFIGURATION_PATH, 'syncer', 'rsync', 'local')
View
5 lib/backup/config.rb
@@ -111,7 +111,10 @@ def add_dsl_constants!
# Encryptors
['OpenSSL', 'GPG'],
# Syncers
- ['Rackspace', 'S3', { 'RSync' => ['Push', 'Pull', 'Local'] }],
+ [
+ { 'Cloud' => ['CloudFiles', 'S3'] },
+ { 'RSync' => ['Push', 'Pull', 'Local'] }
+ ],
# Notifiers
['Mail', 'Twitter', 'Campfire', 'Presently', 'Prowl', 'Hipchat']
]
View
14 lib/backup/configuration/syncer/base.rb
@@ -3,7 +3,19 @@
module Backup
module Configuration
module Syncer
- class Base < Configuration::Base; end
+ class Base < Configuration::Base
+ class << self
+
+ ##
+ # Path to store the synced files/directories to
+ attr_accessor :path
+
+ ##
+ # Flag for mirroring the files/directories
+ attr_accessor :mirror
+
+ end
+ end
end
end
end
View
23 lib/backup/configuration/syncer/cloud.rb
@@ -1,23 +0,0 @@
-# encoding: utf-8
-
-module Backup
- module Configuration
- module Syncer
- class Cloud < Base
- class << self
- ##
- # Amazon S3 bucket name and path to sync to
- attr_accessor :bucket, :path
-
- ##
- # Directories to sync
- attr_accessor :directories
-
- ##
- # Flag to enable mirroring
- attr_accessor :mirror
- end
- end
- end
- end
-end
View
26 lib/backup/configuration/syncer/cloud/base.rb
@@ -0,0 +1,26 @@
+# encoding: utf-8
+
+module Backup
+ module Configuration
+ module Syncer
+ module Cloud
+ class Base < Syncer::Base
+ class << self
+
+ ##
+ # Concurrency setting - defaults to false, but can be set to:
+ # - :threads
+ # - :processes
+ attr_accessor :concurrency_type
+
+ ##
+ # Concurrency level - the number of threads or processors to use.
+ # Defaults to 2.
+ attr_accessor :concurrency_level
+
+ end
+ end
+ end
+ end
+ end
+end
View
36 lib/backup/configuration/syncer/cloud/cloud_files.rb
@@ -0,0 +1,36 @@
+# encoding: utf-8
+
+module Backup
+ module Configuration
+ module Syncer
+ module Cloud
+ class CloudFiles < Base
+ class << self
+
+ ##
+ # Rackspace CloudFiles Credentials
+ attr_accessor :api_key, :username
+
+ ##
+ # Rackspace CloudFiles Container
+ attr_accessor :container
+
+ ##
+ # Rackspace AuthURL allows you to connect
+ # to a different Rackspace datacenter
+ # - https://auth.api.rackspacecloud.com (Default: US)
+ # - https://lon.auth.api.rackspacecloud.com (UK)
+ attr_accessor :auth_url
+
+ ##
+ # Improve performance and avoid data transfer costs
+ # by setting @servicenet to `true`
+ # This only works if Backup runs on a Rackspace server
+ attr_accessor :servicenet
+
+ end
+ end
+ end
+ end
+ end
+end
View
27 lib/backup/configuration/syncer/cloud/s3.rb
@@ -0,0 +1,27 @@
+# encoding: utf-8
+
+module Backup
+ module Configuration
+ module Syncer
+ module Cloud
+ class S3 < Base
+ class << self
+
+ ##
+ # Amazon Simple Storage Service (S3) Credentials
+ attr_accessor :access_key_id, :secret_access_key
+
+ ##
+ # The S3 bucket to store files to
+ attr_accessor :bucket
+
+ ##
+ # The AWS region of the specified S3 bucket
+ attr_accessor :region
+
+ end
+ end
+ end
+ end
+ end
+end
View
30 lib/backup/configuration/syncer/cloud_files.rb
@@ -1,30 +0,0 @@
-# encoding: utf-8
-
-module Backup
- module Configuration
- module Syncer
- class CloudFiles < Cloud
- class << self
- ##
- # Rackspace CloudFiles Credentials
- attr_accessor :api_key, :username
-
- ##
- # Rackspace CloudFiles Container
- attr_accessor :container
-
- ##
- # Rackspace AuthURL allows you to connect to a different Rackspace datacenter
- # - https://auth.api.rackspacecloud.com (Default: US)
- # - https://lon.auth.api.rackspacecloud.com (UK)
- attr_accessor :auth_url
-
- ##
- # Improve performance and avoid data transfer costs by setting @servicenet to `true`
- # This only works if Backup runs on a Rackspace server
- attr_accessor :servicenet
- end
- end
- end
- end
-end
View
8 lib/backup/configuration/syncer/rsync/base.rb
@@ -8,14 +8,6 @@ class Base < Syncer::Base
class << self
##
- # Path to store the synced files/directories to
- attr_accessor :path
-
- ##
- # Flag for mirroring the files/directories
- attr_accessor :mirror
-
- ##
# Additional options for the rsync cli
attr_accessor :additional_options
View
23 lib/backup/configuration/syncer/s3.rb
@@ -1,23 +0,0 @@
-# encoding: utf-8
-
-module Backup
- module Configuration
- module Syncer
- class S3 < Cloud
- class << self
- ##
- # Amazon Simple Storage Service (S3) Credentials
- attr_accessor :access_key_id, :secret_access_key
-
- ##
- # The S3 bucket to store files to
- attr_accessor :bucket
-
- ##
- # The AWS region of the specified S3 bucket
- attr_accessor :region
- end
- end
- end
- end
-end
View
19 lib/backup/model.rb
@@ -125,16 +125,25 @@ def store_with(name, storage_id = nil, &block)
# methods to use during the backup process
def sync_with(name, &block)
##
- # Warn user of DSL change from 'RSync' to 'RSync::Local'
- if name.to_s == 'Backup::Config::RSync'
+ # Warn user of DSL changes
+ case name.to_s
+ when 'Backup::Config::RSync'
Logger.warn Errors::ConfigError.new(<<-EOS)
Configuration Update Needed for Syncer::RSync
The RSync Syncer has been split into three separate modules:
RSync::Local, RSync::Push and RSync::Pull
- Please update your configuration for your local RSync Syncer
- from 'sync_with RSync do ...' to 'sync_with RSync::Local do ...'
+ Please update your configuration.
+ i.e. 'sync_with RSync' is now 'sync_with RSync::Push'
EOS
- name = Backup::Config::RSync::Local
+ name = 'RSync::Push'
+ when /(Backup::Config::S3|Backup::Config::CloudFiles)/
+ syncer = $1.split('::')[2]
+ Logger.warn Errors::ConfigError.new(<<-EOS)
+ Configuration Update Needed for '#{ syncer }' Syncer.
+ This Syncer is now referenced as Cloud::#{ syncer }
+ i.e. 'sync_with #{ syncer }' is now 'sync_with Cloud::#{ syncer }'
+ EOS
+ name = "Cloud::#{ syncer }"
end
@syncers << get_class_from_scope(Syncer, name).new(&block)
end
View
12 lib/backup/syncer/base.rb
@@ -7,17 +7,21 @@ class Base
include Backup::Configuration::Helpers
##
- # Directories to sync
- attr_accessor :directories
-
- ##
# Path to store the synced files/directories to
attr_accessor :path
##
# Flag for mirroring the files/directories
attr_accessor :mirror
+ def initialize
+ load_defaults!
+
+ @path ||= 'backups'
+ @mirror ||= false
+ @directories = Array.new
+ end
+
##
# Syntactical suger for the DSL for adding directories
def directories(&block)
View
190 lib/backup/syncer/cloud.rb
@@ -1,190 +0,0 @@
-# encoding: utf-8
-
-##
-# Only load the Fog gem, along with the Parallel gem, when the Backup::Syncer::Cloud class is loaded
-Backup::Dependency.load('fog')
-Backup::Dependency.load('parallel')
-
-module Backup
- module Syncer
- class Cloud < Base
-
- ##
- # Create a Mutex to synchronize certain parts of the code
- # in order to prevent race conditions or broken STDOUT.
- MUTEX = Mutex.new
-
- ##
- # Concurrency setting - defaults to false, but can be set to:
- # - :threads
- # - :processes
- attr_accessor :concurrency_type
-
- ##
- # Concurrency level - the number of threads or processors to use. Defaults to 2.
- attr_accessor :concurrency_level
-
- ##
- # Instantiates a new Cloud Syncer object and sets the default
- # configuration specified in the Backup::Configuration::Syncer::S3. Then
- # it sets the object defaults if particular properties weren't set.
- # Finally it'll evaluate the users configuration file and overwrite
- # anything that's been defined.
- def initialize(&block)
- load_defaults!
-
- @path ||= 'backups'
- @directories ||= Array.new
- @mirror ||= false
- @concurrency_type = false
- @concurrency_level = 2
-
- instance_eval(&block) if block_given?
-
- @path = path.sub(/^\//, '')
- end
-
- ##
- # Performs the Sync operation
- def perform!
- Logger.message("#{ syncer_name } started the syncing process:")
-
- directories.each do |directory|
- SyncContext.new(directory, repository_object, path).
- sync! mirror, concurrency_type, concurrency_level
- end
- end
-
- private
-
- class SyncContext
- attr_reader :directory, :bucket, :path
-
- ##
- # Creates a new SyncContext object which handles a single directory
- # from the Syncer::Base @directories array.
- def initialize(directory, bucket, path)
- @directory, @bucket, @path = directory, bucket, path
- end
-
- ##
- # Performs the sync operation using the provided techniques (mirroring/concurrency).
- def sync!(mirror = false, concurrency_type = false, concurrency_level = 2)
- block = Proc.new { |relative_path| sync_file relative_path, mirror }
-
- case concurrency_type
- when FalseClass
- all_file_names.each &block
- when :threads
- Parallel.each all_file_names, :in_threads => concurrency_level, &block
- when :processes
- Parallel.each all_file_names, :in_processes => concurrency_level, &block
- else
- raise Errors::Syncer::Cloud::ConfigurationError,
- "Unknown concurrency_type setting: #{concurrency_type.inspect}"
- end
- end
-
- private
-
- ##
- # Gathers all the remote and local file name and merges them together, removing
- # duplicate keys if any, and sorts the in alphabetical order.
- def all_file_names
- @all_file_names ||= (local_files.keys | remote_files.keys).sort
- end
-
- ##
- # Returns a Hash of local files (the keys are the filesystem paths,
- # the values are the LocalFile objects for that given file)
- def local_files
- @local_files ||= begin
- local_hashes.split("\n").collect { |line|
- LocalFile.new directory, line
- }.inject({}) { |hash, file|
- hash[file.relative_path] = file
- hash
- }
- end
- end
-
- ##
- # Returns a String of file paths and their md5 hashes.
- def local_hashes
- MUTEX.synchronize { Logger.message("\s\sGenerating checksums for #{ directory }") }
- `find #{directory} -print0 | xargs -0 openssl md5 2> /dev/null`
- end
-
- ##
- # Returns a Hash of remote files (the keys are the remote paths,
- # the values are the Fog file objects for that given file)
- def remote_files
- @remote_files ||= bucket.files.to_a.select { |file|
- file.key[%r{^#{remote_base}/}]
- }.inject({}) { |hash, file|
- key = file.key.gsub(/^#{remote_base}\//,
- "#{directory.split('/').last}/")
- hash[key] = file
- hash
- }
- end
-
- ##
- # Creates and returns a String that represents the base remote storage path
- def remote_base
- @remote_base ||= [path, directory.split('/').last].select { |part|
- part && part.strip.length > 0
- }.join('/')
- end
-
- ##
- # Performs a sync operation on a file. When mirroring is enabled
- # and a local file has been removed since the last sync, it will also
- # remove it from the remote location. It will no upload files that
- # have not changed since the last sync. Checks are done using an md5 hash.
- # If a file has changed, or has been newly added, the file will be transferred/overwritten.
- def sync_file(relative_path, mirror)
- local_file = local_files[relative_path]
- remote_file = remote_files[relative_path]
-
- if local_file && File.exist?(local_file.path)
- unless remote_file && remote_file.etag == local_file.md5
- MUTEX.synchronize { Logger.message("\s\s[transferring] #{relative_path}") }
- File.open(local_file.path, 'r') do |file|
- bucket.files.create(
- :key => "#{path}/#{relative_path}".gsub(/^\//, ''),
- :body => file
- )
- end
- else
- MUTEX.synchronize { Logger.message("\s\s[skipping] #{relative_path}") }
- end
- elsif remote_file && mirror
- MUTEX.synchronize { Logger.message("\s\s[removing] #{relative_path}") }
- remote_file.destroy
- end
- end
- end
-
- class LocalFile
- attr_reader :directory, :path, :md5
-
- ##
- # Creates a new LocalFile object using the given directory and line
- # from the md5 hash checkup. This object figures out the path, relative_path and md5 hash
- # for the file.
- def initialize(directory, line)
- @directory = directory
- @path, @md5 = *line.chomp.match(/^MD5\(([^\)]+)\)= (\w+)$/).captures
- end
-
- ##
- # Returns the relative path to the file.
- def relative_path
- @relative_path ||= path.gsub %r{^#{directory}},
- directory.split('/').last
- end
- end
- end
- end
-end
View
245 lib/backup/syncer/cloud/base.rb
@@ -0,0 +1,245 @@
+# encoding: utf-8
+
+##
+# Only load the Fog gem, along with the Parallel gem, when the
+# Backup::Syncer::Cloud class is loaded
+Backup::Dependency.load('fog')
+Backup::Dependency.load('parallel')
+
+module Backup
+ module Syncer
+ module Cloud
+ class Base < Syncer::Base
+
+ ##
+ # Create a Mutex to synchronize certain parts of the code
+ # in order to prevent race conditions or broken STDOUT.
+ MUTEX = Mutex.new
+
+ ##
+ # Concurrency setting - defaults to false, but can be set to:
+ # - :threads
+ # - :processes
+ attr_accessor :concurrency_type
+
+ ##
+ # Concurrency level - the number of threads or processors to use.
+ # Defaults to 2.
+ attr_accessor :concurrency_level
+
+ ##
+ # Instantiates a new Cloud Syncer object and sets the default
+ # configuration specified in the Backup::Configuration::Syncer::S3.
+ # Then it sets the object defaults if particular properties weren't set.
+ # Finally it'll evaluate the users configuration file and overwrite
+ # anything that's been defined.
+ def initialize(&block)
+ super
+
+ @concurrency_type ||= false
+ @concurrency_level ||= 2
+
+ instance_eval(&block) if block_given?
+
+ @path = path.sub(/^\//, '')
+ end
+
+ ##
+ # Performs the Sync operation
+ def perform!
+ Logger.message("#{ syncer_name } started the syncing process:")
+
+ @directories.each do |directory|
+ SyncContext.new(
+ File.expand_path(directory), repository_object, @path
+ ).sync! @mirror, @concurrency_type, @concurrency_level
+ end
+
+ Logger.message("#{ syncer_name } Syncing Complete!")
+ end
+
+ private
+
+ class SyncContext
+ attr_reader :directory, :bucket, :path, :remote_base
+
+ ##
+ # Creates a new SyncContext object which handles a single directory
+ # from the Syncer::Base @directories array.
+ def initialize(directory, bucket, path)
+ @directory, @bucket, @path = directory, bucket, path
+ @remote_base = File.join(path, File.basename(directory))
+ end
+
+ ##
+ # Performs the sync operation using the provided techniques
+ # (mirroring/concurrency).
+ def sync!(mirror = false, concurrency_type = false, concurrency_level = 2)
+ block = Proc.new { |relative_path| sync_file relative_path, mirror }
+
+ case concurrency_type
+ when FalseClass
+ all_file_names.each &block
+ when :threads
+ Parallel.each all_file_names,
+ :in_threads => concurrency_level, &block
+ when :processes
+ Parallel.each all_file_names,
+ :in_processes => concurrency_level, &block
+ else
+ raise Errors::Syncer::Cloud::ConfigurationError,
+ "Unknown concurrency_type setting: #{ concurrency_type.inspect }"
+ end
+ end
+
+ private
+
+ ##
+ # Gathers all the relative paths to the local files
+ # and merges them with the , removing
+ # duplicate keys if any, and sorts the in alphabetical order.
+ def all_file_names
+ @all_file_names ||= (local_files.keys | remote_files.keys).sort
+ end
+
+ ##
+ # Returns a Hash of local files, validated to ensure the path
+ # does not contain invalid UTF-8 byte sequences.
+ # The keys are the filesystem paths, relative to @directory.
+ # The values are the LocalFile objects for that given file.
+ def local_files
+ @local_files ||= begin
+ hash = {}
+ local_hashes.lines.map do |line|
+ LocalFile.new(@directory, line)
+ end.compact.each do |file|
+ hash.merge!(file.relative_path => file)
+ end
+ hash
+ end
+ end
+
+ ##
+ # Returns a String of file paths and their md5 hashes.
+ def local_hashes
+ MUTEX.synchronize {
+ Logger.message("\s\sGenerating checksums for '#{ @directory }'")
+ }
+ `find #{ @directory } -print0 | xargs -0 openssl md5 2> /dev/null`
+ end
+
+ ##
+ # Returns a Hash of remote files
+ # The keys are the remote paths, relative to @remote_base
+ # The values are the Fog file objects for that given file
+ def remote_files
+ @remote_files ||= begin
+ hash = {}
+ @bucket.files.all(:prefix => @remote_base).each do |file|
+ hash.merge!(file.key.sub("#{ @remote_base }/", '') => file)
+ end
+ hash
+ end
+ end
+
+ ##
+ # Performs a sync operation on a file. When mirroring is enabled
+ # and a local file has been removed since the last sync, it will also
+ # remove it from the remote location. It will no upload files that
+ # have not changed since the last sync. Checks are done using an md5
+ # hash. If a file has changed, or has been newly added, the file will
+ # be transferred/overwritten.
+ def sync_file(relative_path, mirror)
+ local_file = local_files[relative_path]
+ remote_file = remote_files[relative_path]
+ remote_path = File.join(@remote_base, relative_path)
+
+ if local_file && File.exist?(local_file.path)
+ unless remote_file && remote_file.etag == local_file.md5
+ MUTEX.synchronize {
+ Logger.message("\s\s[transferring] '#{ remote_path }'")
+ }
+ File.open(local_file.path, 'r') do |file|
+ @bucket.files.create(
+ :key => remote_path,
+ :body => file
+ )
+ end
+ else
+ MUTEX.synchronize {
+ Logger.message("\s\s[skipping] '#{ remote_path }'")
+ }
+ end
+ elsif remote_file
+ if mirror
+ MUTEX.synchronize {
+ Logger.message("\s\s[removing] '#{ remote_path }'")
+ }
+ remote_file.destroy
+ else
+ MUTEX.synchronize {
+ Logger.message("\s\s[leaving] '#{ remote_path }'")
+ }
+ end
+ end
+ end
+ end # class SyncContext
+
+ class LocalFile
+ attr_reader :path, :relative_path, :md5
+
+ ##
+ # Return a new LocalFile object if it's valid.
+ # Otherwise, log a warning and return nil.
+ def self.new(*args)
+ local_file = super(*args)
+ if local_file.invalid?
+ MUTEX.synchronize {
+ Logger.warn(
+ "\s\s[skipping] #{ local_file.path }\n" +
+ "\s\sPath Contains Invalid UTF-8 byte sequences"
+ )
+ }
+ return nil
+ end
+ local_file
+ end
+
+ ##
+ # Creates a new LocalFile object using the given directory and line
+ # from the md5 hash checkup. This object figures out the path,
+ # relative_path and md5 hash for the file.
+ def initialize(directory, line)
+ @invalid = false
+ @directory = sanitize(directory)
+ @path, @md5 = sanitize(line).chomp.
+ match(/^MD5\(([^\)]+)\)= (\w+)$/).captures
+ @relative_path = @path.sub(@directory + '/', '')
+ end
+
+ def invalid?
+ @invalid
+ end
+
+ private
+
+ ##
+ # Sanitize string and replace any invalid UTF-8 characters.
+ # If replacements are made, flag the LocalFile object as invalid.
+ def sanitize(str)
+ str.each_char.map do |char|
+ begin
+ char if !!char.unpack('U')
+ rescue
+ @invalid = true
+ "\xEF\xBF\xBD" # => "\uFFFD"
+ end
+ end.join
+ end
+
+ end # class LocalFile
+
+ end # class Base < Syncer::Base
+ end # module Cloud
+ end
+end
View
61 lib/backup/syncer/cloud/cloud_files.rb
@@ -0,0 +1,61 @@
+# encoding: utf-8
+
+module Backup
+ module Syncer
+ module Cloud
+ class CloudFiles < Base
+
+ ##
+ # Rackspace CloudFiles Credentials
+ attr_accessor :api_key, :username
+
+ ##
+ # Rackspace CloudFiles Container
+ attr_accessor :container
+
+ ##
+ # Rackspace AuthURL allows you to connect
+ # to a different Rackspace datacenter
+ # - https://auth.api.rackspacecloud.com (Default: US)
+ # - https://lon.auth.api.rackspacecloud.com (UK)
+ attr_accessor :auth_url
+
+ ##
+ # Improve performance and avoid data transfer costs
+ # by setting @servicenet to `true`
+ # This only works if Backup runs on a Rackspace server
+ attr_accessor :servicenet
+
+ private
+
+ ##
+ # Established and creates a new Fog storage object for CloudFiles.
+ def connection
+ @connection ||= Fog::Storage.new(
+ :provider => provider,
+ :rackspace_username => username,
+ :rackspace_api_key => api_key,
+ :rackspace_auth_url => auth_url,
+ :rackspace_servicenet => servicenet
+ )
+ end
+
+ ##
+ # Creates a new @repository_object (container).
+ # Fetches it from Cloud Files if it already exists,
+ # otherwise it will create it first and fetch use that instead.
+ def repository_object
+ @repository_object ||= connection.directories.get(container) ||
+ connection.directories.create(:key => container)
+ end
+
+ ##
+ # This is the provider that Fog uses for the Cloud Files
+ def provider
+ "Rackspace"
+ end
+
+ end # class Cloudfiles < Base
+ end # module Cloud
+ end
+end
View
51 lib/backup/syncer/cloud/s3.rb
@@ -0,0 +1,51 @@
+# encoding: utf-8
+
+module Backup
+ module Syncer
+ module Cloud
+ class S3 < Base
+
+ ##
+ # Amazon Simple Storage Service (S3) Credentials
+ attr_accessor :access_key_id, :secret_access_key
+
+ ##
+ # The S3 bucket to store files to
+ attr_accessor :bucket
+
+ ##
+ # The AWS region of the specified S3 bucket
+ attr_accessor :region
+
+ private
+
+ ##
+ # Established and creates a new Fog storage object for S3.
+ def connection
+ @connection ||= Fog::Storage.new(
+ :provider => provider,
+ :aws_access_key_id => access_key_id,
+ :aws_secret_access_key => secret_access_key,
+ :region => region
+ )
+ end
+
+ ##
+ # Creates a new @repository_object (bucket).
+ # Fetches it from S3 if it already exists,
+ # otherwise it will create it first and fetch use that instead.
+ def repository_object
+ @repository_object ||= connection.directories.get(bucket) ||
+ connection.directories.create(:key => bucket, :location => region)
+ end
+
+ ##
+ # This is the provider that Fog uses for the Cloud Files
+ def provider
+ "AWS"
+ end
+
+ end # Class S3 < Base
+ end # module Cloud
+ end
+end
View
56 lib/backup/syncer/cloud_files.rb
@@ -1,56 +0,0 @@
-# encoding: utf-8
-
-module Backup
- module Syncer
- class CloudFiles < Cloud
-
- ##
- # Rackspace CloudFiles Credentials
- attr_accessor :api_key, :username
-
- ##
- # Rackspace CloudFiles Container
- attr_accessor :container
-
- ##
- # Rackspace AuthURL allows you to connect to a different Rackspace datacenter
- # - https://auth.api.rackspacecloud.com (Default: US)
- # - https://lon.auth.api.rackspacecloud.com (UK)
- attr_accessor :auth_url
-
- ##
- # Improve performance and avoid data transfer costs by setting @servicenet to `true`
- # This only works if Backup runs on a Rackspace server
- attr_accessor :servicenet
-
- private
-
- ##
- # Established and creates a new Fog storage object for CloudFiles.
- def connection
- @connection ||= Fog::Storage.new(
- :provider => provider,
- :rackspace_username => username,
- :rackspace_api_key => api_key,
- :rackspace_auth_url => auth_url,
- :rackspace_servicenet => servicenet
- )
- end
-
- ##
- # Creates a new @repository_object (container). Fetches it from Cloud Files
- # if it already exists, otherwise it will create it first and fetch use that instead.
- def repository_object
- @repository_object ||= connection.directories.get(container) ||
- connection.directories.create(:key => container)
- end
-
- ##
- # This is the provider that Fog uses for the Cloud Files
- def provider
- "Rackspace"
- end
-
- end
- end
-end
View
5 lib/backup/syncer/rsync/base.rb
@@ -12,11 +12,8 @@ class Base < Syncer::Base
# Instantiates a new RSync Syncer object
# and sets the default configuration
def initialize
- load_defaults!
+ super
- @path ||= 'backups'
- @directories = Array.new
- @mirror ||= false
@additional_options ||= Array.new
end
View
2 lib/backup/syncer/rsync/pull.rb
@@ -27,7 +27,7 @@ def perform!
private
##
- # Return expanded @path
+ # Return expanded @path, since this path is local
def dest_path
@dest_path ||= File.expand_path(@path)
end
View
47 lib/backup/syncer/s3.rb
@@ -1,47 +0,0 @@
-# encoding: utf-8
-
-module Backup
- module Syncer
- class S3 < Cloud
-
- ##
- # Amazon Simple Storage Service (S3) Credentials
- attr_accessor :access_key_id, :secret_access_key
-
- ##
- # The S3 bucket to store files to
- attr_accessor :bucket
-
- ##
- # The AWS region of the specified S3 bucket
- attr_accessor :region
-
- private
-
- ##
- # Established and creates a new Fog storage object for S3.
- def connection
- @connection ||= Fog::Storage.new(
- :provider => provider,
- :aws_access_key_id => access_key_id,
- :aws_secret_access_key => secret_access_key,
- :region => region
- )
- end
-
- ##
- # Creates a new @repository_object (bucket). Fetches it from S3
- # if it already exists, otherwise it will create it first and fetch use that instead.
- def repository_object
- @repository_object ||= connection.directories.get(bucket) ||
- connection.directories.create(:key => bucket, :location => region)
- end
-
- ##
- # This is the provider that Fog uses for the Cloud Files
- def provider
- "AWS"
- end
- end
- end
-end
View
30 spec/configuration/syncer/base_spec.rb
@@ -0,0 +1,30 @@
+# encoding: utf-8
+
+require File.expand_path('../../../spec_helper.rb', __FILE__)
+
+describe Backup::Configuration::Syncer::Base do
+ before do
+ Backup::Configuration::Syncer::Base.defaults do |rsync|
+ rsync.path = '~/backups/'
+ rsync.mirror = true
+ #rsync.directories = 'cannot_have_a_default_value'
+ end
+ end
+ after { Backup::Configuration::Syncer::Base.clear_defaults! }
+
+ it 'should set the default syncer configuration' do
+ rsync = Backup::Configuration::Syncer::Base
+ rsync.path.should == '~/backups/'
+ rsync.mirror.should == true
+ end
+
+ describe '#clear_defaults!' do
+ it 'should clear all the defaults, resetting them to nil' do
+ Backup::Configuration::Syncer::Base.clear_defaults!
+
+ rsync = Backup::Configuration::Syncer::Base
+ rsync.path.should == nil
+ rsync.mirror.should == nil
+ end
+ end
+end
View
34 spec/configuration/syncer/cloud/base_spec.rb
@@ -0,0 +1,34 @@
+# encoding: utf-8
+
+require File.expand_path('../../../../spec_helper.rb', __FILE__)
+
+describe 'Backup::Configuration::Syncer::Cloud::Base' do
+ it 'should be a subclass of Syncer::Base' do
+ cloud = Backup::Configuration::Syncer::Cloud::Base
+ cloud.superclass.should == Backup::Configuration::Syncer::Base
+ end
+
+ before do
+ Backup::Configuration::Syncer::Cloud::Base.defaults do |cloud|
+ cloud.concurrency_type = 'default_type'
+ cloud.concurrency_level = 'default_level'
+ end
+ end
+ after { Backup::Configuration::Syncer::Cloud::Base.clear_defaults! }
+
+ it 'should set the default cloud files configuration' do
+ cloud = Backup::Configuration::Syncer::Cloud::Base
+ cloud.concurrency_type.should == 'default_type'
+ cloud.concurrency_level.should == 'default_level'
+ end
+
+ describe '#clear_defaults!' do
+ it 'should clear all the defaults, resetting them to nil' do
+ Backup::Configuration::Syncer::Cloud::Base.clear_defaults!
+
+ cloud = Backup::Configuration::Syncer::Cloud::Base
+ cloud.concurrency_type.should == nil
+ cloud.concurrency_level.should == nil
+ end
+ end
+end
View
25 .../configuration/syncer/cloud_files_spec.rb → ...guration/syncer/cloud/cloud_files_spec.rb
@@ -1,44 +1,43 @@
# encoding: utf-8
-require File.expand_path('../../../spec_helper.rb', __FILE__)
+require File.expand_path('../../../../spec_helper.rb', __FILE__)
+
+describe 'Backup::Configuration::Syncer::Cloud::CloudFiles' do
+ it 'should be a subclass of Syncer::Cloud::Base' do
+ cf = Backup::Configuration::Syncer::Cloud::CloudFiles
+ cf.superclass.should == Backup::Configuration::Syncer::Cloud::Base
+ end
-describe Backup::Configuration::Syncer::CloudFiles do
before do
- Backup::Configuration::Syncer::CloudFiles.defaults do |cf|
+ Backup::Configuration::Syncer::Cloud::CloudFiles.defaults do |cf|
cf.username = 'my-username'
cf.api_key = 'my-api-key'
cf.container = 'my-container'
cf.auth_url = 'my-auth-url'
cf.servicenet = true
- cf.path = '/backups/'
- cf.mirror = true
end
end
- after { Backup::Configuration::Syncer::CloudFiles.clear_defaults! }
+ after { Backup::Configuration::Syncer::Cloud::CloudFiles.clear_defaults! }
it 'should set the default cloud files configuration' do
- cf = Backup::Configuration::Syncer::CloudFiles
+ cf = Backup::Configuration::Syncer::Cloud::CloudFiles
cf.username.should == 'my-username'
cf.api_key.should == 'my-api-key'
cf.container.should == 'my-container'
cf.auth_url.should == 'my-auth-url'
cf.servicenet.should == true
- cf.path.should == '/backups/'
- cf.mirror.should == true
end
describe '#clear_defaults!' do
it 'should clear all the defaults, resetting them to nil' do
- Backup::Configuration::Syncer::CloudFiles.clear_defaults!
+ Backup::Configuration::Syncer::Cloud::CloudFiles.clear_defaults!
- cf = Backup::Configuration::Syncer::CloudFiles
+ cf = Backup::Configuration::Syncer::Cloud::CloudFiles
cf.username.should == nil
cf.api_key.should == nil
cf.container.should == nil
cf.auth_url.should == nil
cf.servicenet.should == nil
- cf.path.should == nil
- cf.mirror.should == nil
end
end
end
View
25 spec/configuration/syncer/s3_spec.rb → spec/configuration/syncer/cloud/s3_spec.rb
@@ -1,38 +1,37 @@
# encoding: utf-8
-require File.expand_path('../../../spec_helper.rb', __FILE__)
+require File.expand_path('../../../../spec_helper.rb', __FILE__)
+
+describe 'Backup::Configuration::Syncer::S3' do
+ it 'should be a subclass of Syncer::Cloud::Base' do
+ s3 = Backup::Configuration::Syncer::Cloud::S3
+ s3.superclass.should == Backup::Configuration::Syncer::Cloud::Base
+ end
-describe Backup::Configuration::Syncer::S3 do
before do
- Backup::Configuration::Syncer::S3.defaults do |s3|
+ Backup::Configuration::Syncer::Cloud::S3.defaults do |s3|
s3.access_key_id = 'my_access_key_id'
s3.secret_access_key = 'my_secret_access_key'
s3.bucket = 'my-bucket'
- s3.path = '/backups/'
- s3.mirror = true
end
end
- after { Backup::Configuration::Syncer::S3.clear_defaults! }
+ after { Backup::Configuration::Syncer::Cloud::S3.clear_defaults! }
it 'should set the default s3 configuration' do
- s3 = Backup::Configuration::Syncer::S3
+ s3 = Backup::Configuration::Syncer::Cloud::S3
s3.access_key_id.should == 'my_access_key_id'
s3.secret_access_key.should == 'my_secret_access_key'
s3.bucket.should == 'my-bucket'
- s3.path.should == '/backups/'
- s3.mirror.should == true
end
describe '#clear_defaults!' do
it 'should clear all the defaults, resetting them to nil' do
- Backup::Configuration::Syncer::S3.clear_defaults!
+ Backup::Configuration::Syncer::Cloud::S3.clear_defaults!
- s3 = Backup::Configuration::Syncer::S3
+ s3 = Backup::Configuration::Syncer::Cloud::S3
s3.access_key_id.should == nil
s3.secret_access_key.should == nil
s3.bucket.should == nil
- s3.path.should == nil
- s3.mirror.should == nil
end
end
end
View
16 spec/configuration/syncer/rsync/base_spec.rb
@@ -3,30 +3,28 @@
require File.expand_path('../../../../spec_helper.rb', __FILE__)
describe Backup::Configuration::Syncer::RSync::Base do
+ it 'should be a subclass of Syncer::Base' do
+ rsync = Backup::Configuration::Syncer::RSync::Base
+ rsync.superclass.should == Backup::Configuration::Syncer::Base
+ end
+
before do
Backup::Configuration::Syncer::RSync::Base.defaults do |rsync|
- #rsync.directories = 'cannot_have_a_default_value'
- rsync.path = '~/backups/'
- rsync.mirror = true
- rsync.additional_options = []
+ rsync.additional_options = ['foo']
end
end
after { Backup::Configuration::Syncer::RSync::Base.clear_defaults! }
it 'should set the default rsync configuration' do
rsync = Backup::Configuration::Syncer::RSync::Base
- rsync.path.should == '~/backups/'
- rsync.mirror.should == true
- rsync.additional_options.should == []
+ rsync.additional_options.should == ['foo']
end
describe '#clear_defaults!' do
it 'should clear all the defaults, resetting them to nil' do
Backup::Configuration::Syncer::RSync::Base.clear_defaults!
rsync = Backup::Configuration::Syncer::RSync::Base
- rsync.path.should == nil
- rsync.mirror.should == nil
rsync.additional_options.should == nil
end
end
View
19 spec/model_spec.rb
@@ -209,10 +209,25 @@ def using_fake(const, replacement)
end
end
- it 'should warn user of change from RSync to RSync::Local' do
+ it 'should warn user of change from RSync to RSync::Push' do
Backup::Logger.expects(:warn)
model.sync_with('Backup::Config::RSync')
- model.syncers.first.should be_an_instance_of Backup::Syncer::RSync::Local
+ model.syncers.first.should
+ be_an_instance_of Backup::Syncer::RSync::Push
+ end
+
+ it 'should warn user of change from S3 to Cloud::S3' do
+ Backup::Logger.expects(:warn)
+ model.sync_with('Backup::Config::S3')
+ model.syncers.first.should
+ be_an_instance_of Backup::Syncer::Cloud::S3
+ end
+
+ it 'should warn user of change from CloudFiles to Cloud::CloudFiles' do
+ Backup::Logger.expects(:warn)
+ model.sync_with('Backup::Config::CloudFiles')
+ model.syncers.first.should
+ be_an_instance_of Backup::Syncer::Cloud::CloudFiles
end
end
View
78 spec/syncer/base_spec.rb
@@ -14,6 +14,84 @@
base.included_modules.should include(Backup::Configuration::Helpers)
end
+ describe '#initialize' do
+
+ it 'should use default values' do
+ syncer.path.should == 'backups'
+ syncer.mirror.should == false
+ syncer.directories.should == []
+ end
+
+ context 'when setting configuration defaults' do
+ after { Backup::Configuration::Syncer::Base.clear_defaults! }
+
+ it 'should use the configured defaults' do
+ Backup::Configuration::Syncer::Base.defaults do |base|
+ base.path = 'some_path'
+ base.mirror = 'some_mirror'
+ #base.directories = 'cannot_have_a_default_value'
+ end
+ syncer = Backup::Syncer::Base.new
+ syncer.path.should == 'some_path'
+ syncer.mirror.should == 'some_mirror'
+ syncer.directories.should == []
+ end
+ end
+
+ end # describe '#initialize'
+
+ describe '#directories' do
+ before do
+ syncer.instance_variable_set(
+ :@directories, ['/some/directory', '/another/directory']
+ )
+ end
+
+ context 'when no block is given' do
+ it 'should return @directories' do
+ syncer.directories.should ==
+ ['/some/directory', '/another/directory']
+ end
+ end
+
+ context 'when a block is given' do
+ it 'should evalute the block, allowing #add to add directories' do
+ syncer.directories do
+ add '/new/path'
+ add '/another/new/path'
+ end
+ syncer.directories.should == [
+ '/some/directory',
+ '/another/directory',
+ '/new/path',
+ '/another/new/path'
+ ]
+ end
+ end
+ end # describe '#directories'
+
+ describe '#add' do
+ before do
+ syncer.instance_variable_set(
+ :@directories, ['/some/directory', '/another/directory']
+ )
+ end
+
+ it 'should add the given path to @directories' do
+ syncer.add '/my/path'
+ syncer.directories.should ==
+ ['/some/directory', '/another/directory', '/my/path']
+ end
+
+ # Note: Each Syncer should handle this as needed.
+ # For example, expanding these here would break RSync::Pull
+ it 'should not expand the given paths' do
+ syncer.add 'relative/path'
+ syncer.directories.should ==
+ ['/some/directory', '/another/directory', 'relative/path']
+ end
+ end
+
describe '#syncer_name' do
it 'should return the class name with the Backup:: namespace removed' do
syncer.send(:syncer_name).should == 'Syncer::Base'
View
523 spec/syncer/cloud/base_spec.rb
@@ -0,0 +1,523 @@
+# encoding: utf-8
+require File.expand_path('../../../spec_helper.rb', __FILE__)
+
+describe 'Backup::Syncer::Cloud::Base' do
+ let(:base) { Backup::Syncer::Cloud::Base }
+ let(:syncer) { base.new }
+ let(:s) { sequence '' }
+
+ it 'should be a subclass of Syncer::Base' do
+ base.superclass.should == Backup::Syncer::Base
+ end
+
+ it 'should establish a class constant for a Mutex' do
+ base::MUTEX.should be_an_instance_of Mutex
+ end
+
+ describe '#initialize' do
+
+ it 'should inherit default values from the superclass' do
+ syncer.path.should == 'backups'
+ syncer.mirror.should == false
+ end
+
+ it 'should set default values' do
+ syncer.concurrency_type.should == false
+ syncer.concurrency_level.should == 2
+ end
+
+ it 'should strip any leading slash in path' do
+ syncer = Backup::Syncer::Cloud::Base.new do |cloud|
+ cloud.path = '/cleaned/path'
+ end
+ syncer.path.should == 'cleaned/path'
+ end
+
+ context 'when setting configuration defaults' do
+ after { Backup::Configuration::Syncer::Cloud::Base.clear_defaults! }
+
+ it 'should use the configured defaults' do
+ Backup::Configuration::Syncer::Cloud::Base.defaults do |cloud|
+ cloud.concurrency_type = 'default_concurrency_type'
+ cloud.concurrency_level = 'default_concurrency_level'
+ end
+ syncer = Backup::Syncer::Cloud::Base.new
+ syncer.concurrency_type.should == 'default_concurrency_type'
+ syncer.concurrency_level.should == 'default_concurrency_level'
+ end
+
+ it 'should override the configured defaults' do
+ Backup::Configuration::Syncer::Cloud::Base.defaults do |cloud|
+ cloud.concurrency_type = 'old_concurrency_type'
+ cloud.concurrency_level = 'old_concurrency_level'
+ end
+ syncer = Backup::Syncer::Cloud::Base.new do |cloud|
+ cloud.concurrency_type = 'new_concurrency_type'
+ cloud.concurrency_level = 'new_concurrency_level'
+ end
+
+ syncer.concurrency_type.should == 'new_concurrency_type'
+ syncer.concurrency_level.should == 'new_concurrency_level'
+ end
+ end
+
+ end # describe '#initialize'
+
+ describe '#perform' do
+ let(:sync_context) { mock }
+
+ before do
+ syncer.stubs(:repository_object).returns(:a_repository_object)
+ end
+
+ it 'should sync each directory' do
+ syncer.directories do
+ add '/dir/one'
+ add '/dir/two'
+ end
+
+ Backup::Logger.expects(:message).in_sequence(s).with(
+ 'Syncer::Cloud::Base started the syncing process:'
+ )
+ base::SyncContext.expects(:new).in_sequence(s).with(
+ '/dir/one', :a_repository_object, 'backups'
+ ).returns(sync_context)
+ sync_context.expects(:sync!).in_sequence(s).with(
+ false, false, 2
+ )
+ base::SyncContext.expects(:new).in_sequence(s).with(
+ '/dir/two', :a_repository_object, 'backups'
+ ).returns(sync_context)
+ sync_context.expects(:sync!).in_sequence(s).with(
+ false, false, 2
+ )
+ Backup::Logger.expects(:message).in_sequence(s).with(
+ 'Syncer::Cloud::Base Syncing Complete!'
+ )
+
+ syncer.perform!
+ end
+
+ it 'should ensure each directory path is expanded with no trailing slash' do
+ syncer.directories do
+ add '/dir/one/'
+ add 'dir/two'
+ end
+
+ base::SyncContext.expects(:new).with(
+ '/dir/one', :a_repository_object, 'backups'
+ ).returns(sync_context)
+
+ base::SyncContext.expects(:new).with(
+ File.expand_path('dir/two'), :a_repository_object, 'backups'
+ ).returns(sync_context)
+
+ sync_context.stubs(:sync!)
+
+ syncer.perform!
+ end
+ end # describe '#perform'
+
+ describe 'Cloud::Base::SyncContext' do
+ let(:bucket) { mock }
+ let(:sync_context) do
+ Backup::Syncer::Cloud::Base::SyncContext.new(
+ '/dir/to/sync', bucket, 'backups'
+ )
+ end
+
+ describe '#initialize' do
+ it 'should set variables' do
+ sync_context.directory.should == '/dir/to/sync'
+ sync_context.bucket.should == bucket
+ sync_context.path.should == 'backups'
+ sync_context.remote_base.should == 'backups/sync'
+ end
+ end
+
+ describe '#sync!' do
+ let(:all_files_array) { mock }
+
+ before do
+ sync_context.stubs(:all_file_names).returns(all_files_array)
+ end
+
+ context 'when concurrency_type is set to `false`' do
+ it 'syncs files without concurrency' do
+ all_files_array.expects(:each).in_sequence(s).
+ multiple_yields('foo.file', 'foo_dir/foo.file')
+
+ sync_context.expects(:sync_file).in_sequence(s).
+ with('foo.file', :mirror)
+ sync_context.expects(:sync_file).in_sequence(s).
+ with('foo_dir/foo.file', :mirror)
+
+ sync_context.sync!(:mirror, false, :foo)
+ end
+ end
+
+ context 'when concurrency_type is set to `:threads`' do
+ it 'uses `concurrency_level` number of threads for concurrency' do
+ Parallel.expects(:each).in_sequence(s).with(
+ all_files_array, :in_threads => :num_of_threads
+ ).multiple_yields('foo.file', 'foo_dir/foo.file')
+
+ sync_context.expects(:sync_file).in_sequence(s).
+ with('foo.file', :mirror)
+ sync_context.expects(:sync_file).in_sequence(s).
+ with('foo_dir/foo.file', :mirror)
+
+ sync_context.sync!(:mirror, :threads, :num_of_threads)
+ end
+ end
+
+ context 'when concurrency_type is set to `:processes`' do
+ it 'uses `concurrency_level` number of processes for concurrency' do
+ Parallel.expects(:each).in_sequence(s).with(
+ all_files_array, :in_processes => :num_of_processes
+ ).multiple_yields('foo.file', 'foo_dir/foo.file')
+
+ sync_context.expects(:sync_file).in_sequence(s).
+ with('foo.file', :mirror)
+ sync_context.expects(:sync_file).in_sequence(s).
+ with('foo_dir/foo.file', :mirror)
+
+ sync_context.sync!(:mirror, :processes, :num_of_processes)
+ end
+ end
+
+ context 'when concurrency_type setting is invalid' do
+ it 'should raise an error' do
+ expect do
+ sync_context.sync!(:foo, 'unknown type', :foo)
+ end.to raise_error(
+ Backup::Errors::Syncer::Cloud::ConfigurationError,
+ 'Syncer::Cloud::ConfigurationError: ' +
+ "Unknown concurrency_type setting: \"unknown type\""
+ )
+ end
+ end
+ end # describe '#sync!'
+
+ describe '#all_file_names' do
+ let(:local_files_hash) do
+ { 'file_b' => :foo, 'file_a' => :foo, 'dir_a/file_b' => :foo }
+ end
+ let(:remote_files_hash) do
+ { 'file_c' => :foo, 'file_a' => :foo, 'dir_a/file_a' => :foo }
+ end
+ let(:local_remote_union_array) do
+ ['dir_a/file_a', 'dir_a/file_b', 'file_a', 'file_b', 'file_c']
+ end
+
+ it 'returns and caches a sorted union of local and remote file names' do
+ sync_context.expects(:local_files).once.returns(local_files_hash)
+ sync_context.expects(:remote_files).once.returns(remote_files_hash)
+
+ sync_context.send(:all_file_names).should == local_remote_union_array
+ sync_context.instance_variable_get(:@all_file_names).
+ should == local_remote_union_array
+ sync_context.send(:all_file_names).should == local_remote_union_array
+ end
+ end # describe '#all_file_names'
+
+ describe '#local_files' do
+ let(:local_file_class) { Backup::Syncer::Cloud::Base::LocalFile }
+ let(:local_hashes_data) { "line1\nline2\nbad\xFFline\nline3" }
+
+ let(:local_file_a) { stub(:relative_path => 'file_a') }
+ let(:local_file_b) { stub(:relative_path => 'file_b') }
+ let(:local_file_c) { stub(:relative_path => 'file_c') }
+ let(:local_files_hash) do
+ { 'file_a' => local_file_a,
+ 'file_b' => local_file_b,
+ 'file_c' => local_file_c }
+ end
+
+ it 'should return and caches a hash of LocalFile objects' do
+ sync_context.expects(:local_hashes).once.returns(local_hashes_data)
+
+ local_file_class.expects(:new).once.with('/dir/to/sync', "line1\n").
+ returns(local_file_a)
+ local_file_class.expects(:new).once.with('/dir/to/sync', "line2\n").
+ returns(local_file_b)
+ local_file_class.expects(:new).once.with('/dir/to/sync', "bad\xFFline\n").
+ returns(nil)
+ local_file_class.expects(:new).once.with('/dir/to/sync', "line3").
+ returns(local_file_c)
+
+ sync_context.send(:local_files).should == local_files_hash
+ sync_context.instance_variable_get(:@local_files).
+ should == local_files_hash
+ sync_context.send(:local_files).should == local_files_hash
+ end
+
+ # Note: don't use methods that validate encoding
+ it 'will raise an Exception if String#split is used',
+ :if => RUBY_VERSION >= '1.9' do
+ expect do
+ "line1\nbad\xFFline\nline3".split("\n")
+ end.to raise_error(ArgumentError, 'invalid byte sequence in UTF-8')
+ end
+ end # describe '#local_files'
+
+ describe '#local_hashes' do
+ it 'should collect file paths and MD5 checksums for @directory' do
+ base::MUTEX.expects(:synchronize).yields
+ Backup::Logger.expects(:message).with(
+ "\s\sGenerating checksums for '/dir/to/sync'"
+ )
+ sync_context.expects(:`).with(
+ "find /dir/to/sync -print0 | xargs -0 openssl md5 2> /dev/null"
+ ).returns('MD5(tmp/foo)= 123abcdef')
+
+ sync_context.send(:local_hashes).should == 'MD5(tmp/foo)= 123abcdef'
+ end
+ end
+
+ describe '#remote_files' do
+ let(:repository_object) { mock }
+ let(:repository_files) { mock }
+ let(:file_objects) { mock }
+ let(:file_obj_a) { stub(:key => 'file_a') }
+ let(:file_obj_b) { stub(:key => 'file_b') }
+ let(:file_obj_c) { stub(:key => 'dir/file_c') }
+ let(:remote_files_hash) do
+ { 'file_a' => file_obj_a,
+ 'file_b' => file_obj_b,
+ 'dir/file_c' => file_obj_c }
+ end
+
+ before do
+ sync_context.instance_variable_set(:@bucket, repository_object)
+
+ repository_object.expects(:files).once.returns(repository_files)
+ repository_files.expects(:all).once.with(:prefix => 'backups/sync').
+ returns(file_objects)
+ file_objects.expects(:each).once.multiple_yields(
+ file_obj_a, file_obj_b, file_obj_c
+ )
+
+ # this is to avoid: unexpected invocation: #<Mock>.to_a()
+ # only 1.9.2 seems affected by this
+ if RUBY_VERSION == '1.9.2'
+ file_obj_a.stubs(:to_a)
+ file_obj_b.stubs(:to_a)
+ file_obj_c.stubs(:to_a)
+ end
+ end
+
+ context 'when it returns and caches a hash of repository file objects' do
+ it 'should remove the @remote_base from the path for the hash key' do
+ sync_context.send(:remote_files).should == remote_files_hash
+ sync_context.instance_variable_get(:@remote_files).
+ should == remote_files_hash
+ sync_context.send(:remote_files).should == remote_files_hash
+ end
+ end
+ end # describe '#remote_files'
+
+ describe '#sync_file' do
+ let(:local_file) do
+ stub(:path => '/dir/to/sync/sync.file', :md5 => 'abc123')
+ end
+ let(:remote_file) do
+ stub(:path => 'backups/sync/sync.file')
+ end
+ let(:file) { mock }
+ let(:repository_object) { mock }
+ let(:repository_files) { mock }
+
+ before do
+ sync_context.instance_variable_set(:@bucket, repository_object)
+ repository_object.stubs(:files).returns(repository_files)
+ end
+
+ context 'when the requested file to sync exists locally' do
+ before do
+ sync_context.stubs(:local_files).returns(
+ { 'sync.file' => local_file }
+ )
+ File.expects(:exist?).with('/dir/to/sync/sync.file').returns(true)
+ end
+
+ context 'when the MD5 checksum matches the remote file' do
+ before do
+ remote_file.stubs(:etag).returns('abc123')
+ sync_context.stubs(:remote_files).returns(
+ { 'sync.file' => remote_file }
+ )
+ end
+
+ it 'should skip the file' do
+ File.expects(:open).never
+ base::MUTEX.expects(:synchronize).yields
+ Backup::Logger.expects(:message).with(
+ "\s\s[skipping] 'backups/sync/sync.file'"
+ )
+
+ sync_context.send(:sync_file, 'sync.file', :foo)
+ end
+ end
+
+ context 'when the MD5 checksum does not match the remote file' do
+ before do
+ remote_file.stubs(:etag).returns('dfg456')
+ sync_context.stubs(:remote_files).returns(
+ { 'sync.file' => remote_file }
+ )
+ end
+
+ it 'should upload the file' do
+ base::MUTEX.expects(:synchronize).yields
+ Backup::Logger.expects(:message).with(
+ "\s\s[transferring] 'backups/sync/sync.file'"
+ )
+
+ File.expects(:open).with('/dir/to/sync/sync.file', 'r').yields(file)
+ repository_files.expects(:create).with(
+ :key => 'backups/sync/sync.file',
+ :body => file
+ )
+
+ sync_context.send(:sync_file, 'sync.file', :foo)
+ end
+ end
+
+ context 'when the requested file does not exist on the remote' do
+ before do
+ sync_context.stubs(:remote_files).returns({})
+ end
+
+ it 'should upload the file' do
+ base::MUTEX.expects(:synchronize).yields
+ Backup::Logger.expects(:message).with(
+ "\s\s[transferring] 'backups/sync/sync.file'"
+ )
+
+ File.expects(:open).with('/dir/to/sync/sync.file', 'r').yields(file)
+ repository_files.expects(:create).with(
+ :key => 'backups/sync/sync.file',
+ :body => file
+ )
+
+ sync_context.send(:sync_file, 'sync.file', :foo)
+ end
+ end
+ end
+
+ context 'when the requested file does not exist locally' do
+ before do
+ sync_context.stubs(:remote_files).returns(
+ { 'sync.file' => remote_file }
+ )
+ sync_context.stubs(:local_files).returns({})
+ end
+
+ context 'when the `mirror` option is set to true' do
+ it 'should remove the file from the remote' do
+ base::MUTEX.expects(:synchronize).yields
+ Backup::Logger.expects(:message).with(
+ "\s\s[removing] 'backups/sync/sync.file'"
+ )
+
+ remote_file.expects(:destroy)
+
+ sync_context.send(:sync_file, 'sync.file', true)
+ end
+ end
+
+ context 'when the `mirror` option is set to false' do
+ it 'should leave the file on the remote' do
+ base::MUTEX.expects(:synchronize).yields
+ Backup::Logger.expects(:message).with(
+ "\s\s[leaving] 'backups/sync/sync.file'"
+ )
+
+ remote_file.expects(:destroy).never
+
+ sync_context.send(:sync_file, 'sync.file', false)
+ end
+ end
+ end
+ end # describe '#sync_file'
+ end # describe 'Cloud::Base::SyncContext'
+
+ describe 'Cloud::Base::LocalFile' do
+ let(:local_file_class) { Backup::Syncer::Cloud::Base::LocalFile }
+
+ describe '#new' do
+ describe 'wrapping #initialize and using #sanitize to validate objects' do
+ context 'when the path is valid UTF-8' do
+ let(:local_file) do
+ local_file_class.new('foo', 'MD5(foo)= foo')
+ end
+
+ it 'should return the new object' do
+ base::MUTEX.expects(:synchronize).never
+ Backup::Logger.expects(:warn).never
+
+ local_file.should be_an_instance_of local_file_class
+ end
+ end
+
+ context 'when the path contains invalid UTF-8' do
+ let(:local_file) do
+ local_file_class.new(
+ "/bad/pa\xFFth", "MD5(/bad/pa\xFFth/to/file)= foo"
+ )
+ end
+ it 'should return nil and log a warning' do
+ base::MUTEX.expects(:synchronize).yields
+ Backup::Logger.expects(:warn).with(
+ "\s\s[skipping] /bad/pa\xEF\xBF\xBDth/to/file\n" +
+ "\s\sPath Contains Invalid UTF-8 byte sequences"
+ )
+
+ local_file.should be_nil
+ end
+ end
+ end
+ end # describe '#new'
+
+ describe '#initialize' do
+ let(:local_file) do
+ local_file_class.new(:directory, :line)
+ end
+
+ before do
+ local_file_class.any_instance.expects(:sanitize).with(:directory).
+ returns('/dir/to/sync')
+ local_file_class.any_instance.expects(:sanitize).with(:line).
+ returns("MD5(/dir/to/sync/subdir/sync.file)= 123abcdef\n")
+ end
+
+ it 'should determine @path, @relative_path and @md5' do
+ local_file.path.should == '/dir/to/sync/subdir/sync.file'
+ local_file.relative_path.should == 'subdir/sync.file'
+ local_file.md5.should == '123abcdef'
+ end
+
+ it 'should return nil if the object is invalid' do
+ local_file_class.any_instance.expects(:invalid?).returns(true)
+ local_file.should be_nil
+ end
+ end # describe '#initialize'
+
+ describe '#sanitize' do
+ let(:local_file) do
+ local_file_class.new('foo', 'MD5(foo)= foo')
+ end
+
+ it 'should replace any invalid UTF-8 characters' do
+ local_file.send(:sanitize, "/path/to/d\xFFir/subdir/sync\xFFfile").
+ should == "/path/to/d\xEF\xBF\xBDir/subdir/sync\xEF\xBF\xBDfile"
+ end
+
+ it 'should flag the LocalFile object as invalid' do
+ local_file.send(:sanitize, "/path/to/d\xFFir/subdir/sync\xFFfile")
+ local_file.invalid?.should be_true
+ end
+ end # describe '#sanitize'
+ end # describe 'Cloud::Base::LocalFile'
+end
View
153 spec/syncer/cloud/cloud_files_spec.rb
@@ -0,0 +1,153 @@
+# encoding: utf-8
+require File.expand_path('../../../spec_helper.rb', __FILE__)
+
+describe 'Backup::Syncer::Cloud::CloudFiles' do
+ let(:syncer) do
+ Backup::Syncer::Cloud::CloudFiles.new do |cf|
+ cf.api_key = 'my_api_key'
+ cf.username = 'my_username'
+ cf.container = 'my_container'
+ cf.auth_url = 'my_auth_url'
+ cf.servicenet = true
+ end
+ end
+
+ it 'should be a subclass of Syncer::Cloud::Base' do
+ Backup::Syncer::Cloud::CloudFiles.superclass.
+ should == Backup::Syncer::Cloud::Base
+ end
+
+ describe '#initialize' do
+ it 'should have defined the configuration properly' do
+ syncer.api_key.should == 'my_api_key'
+ syncer.username.should == 'my_username'
+ syncer.container.should == 'my_container'
+ syncer.auth_url.should == 'my_auth_url'
+ syncer.servicenet.should == true
+ end
+
+ it 'should inherit default values from superclasses' do
+ # Syncer::Cloud::Base
+ syncer.concurrency_type.should == false
+ syncer.concurrency_level.should == 2
+
+ # Syncer::Base
+ syncer.path.should == 'backups'
+ syncer.mirror.should == false
+ syncer.directories.should == []
+ end
+
+ context 'when options are not set' do
+ it 'should use default values' do
+ syncer = Backup::Syncer::Cloud::CloudFiles.new
+ syncer.api_key.should == nil
+ syncer.username.should == nil
+ syncer.container.should == nil
+ syncer.auth_url.should == nil
+ syncer.servicenet.should == nil
+ end
+ end
+
+ context 'when setting configuration defaults' do
+ after { Backup::Configuration::Syncer::Cloud::CloudFiles.clear_defaults! }
+
+ it 'should use the configured defaults' do
+ Backup::Configuration::Syncer::Cloud::CloudFiles.defaults do |cf|
+ cf.api_key = 'default_api_key'
+ cf.username = 'default_username'
+ cf.container = 'default_container'
+ cf.auth_url = 'default_auth_url'
+ cf.servicenet = 'default_servicenet'
+ end
+ syncer = Backup::Syncer::Cloud::CloudFiles.new
+ syncer.api_key.should == 'default_api_key'
+ syncer.username.should == 'default_username'
+ syncer.container.should == 'default_container'
+ syncer.auth_url.should == 'default_auth_url'
+ syncer.servicenet.should == 'default_servicenet'
+ end
+
+ it 'should override the configured defaults' do
+ Backup::Configuration::Syncer::Cloud::CloudFiles.defaults do |cf|
+ cf.api_key = 'old_api_key'
+ cf.username = 'old_username'
+ cf.container = 'old_container'
+ cf.auth_url = 'old_auth_url'
+ cf.servicenet = 'old_servicenet'
+ end
+ syncer = Backup::Syncer::Cloud::CloudFiles.new do |cf|
+ cf.api_key = 'new_api_key'
+ cf.username = 'new_username'
+ cf.container = 'new_container'
+ cf.auth_url = 'new_auth_url'
+ cf.servicenet = 'new_servicenet'
+ end
+
+ syncer.api_key.should == 'new_api_key'
+ syncer.username.should == 'new_username'
+ syncer.container.should == 'new_container'
+ syncer.auth_url.should == 'new_auth_url'
+ syncer.servicenet.should == 'new_servicenet'
+ end
+ end # context 'when setting configuration defaults'
+ end # describe '#initialize'
+
+ describe '#connection' do
+ let(:connection) { mock }
+
+ before do
+ Fog::Storage.expects(:new).once.with(
+ :provider => 'Rackspace',
+ :rackspace_username => 'my_username',
+ :rackspace_api_key => 'my_api_key',
+ :rackspace_auth_url => 'my_auth_url',
+ :rackspace_servicenet => true
+ ).returns(connection)
+ end
+
+ it 'should establish and re-use the connection' do
+ syncer.send(:connection).should == connection
+ syncer.instance_variable_get(:@connection).should == connection
+ syncer.send(:connection).should == connection
+ end
+ end
+
+ describe '#repository_object' do
+ let(:connection) { mock }
+ let(:directories) { mock }
+ let(:container) { mock }
+
+ before do
+ syncer.stubs(:connection).returns(connection)
+ connection.stubs(:directories).returns(directories)
+ end
+
+ context 'when the @container does not exist' do
+ before do
+ directories.expects(:get).once.with('my_container').returns(nil)
+ directories.expects(:create).once.with(
+ :key => 'my_container'
+ ).returns(container)
+ end
+
+ it 'should create and re-use the container' do
+ syncer.send(:repository_object).should == container
+ syncer.instance_variable_get(:@repository_object).should == container
+ syncer.send(:repository_object).should == container
+ end
+ end
+
+ context 'when the @container does exist' do
+ before do
+ directories.expects(:get).once.with('my_container').returns(container)
+ directories.expects(:create).never
+ end
+
+ it 'should retrieve and re-use the container' do
+ syncer.send(:repository_object).should == container
+ syncer.instance_variable_get(:@repository_object).should == container
+ syncer.send(:repository_object).should == container
+ end
+ end
+ end
+end
View
145 spec/syncer/cloud/s3_spec.rb
@@ -0,0 +1,145 @@
+# encoding: utf-8
+require File.expand_path('../../../spec_helper.rb', __FILE__)
+
+describe 'Backup::Syncer::Cloud::S3' do
+ let(:syncer) do
+ Backup::Syncer::Cloud::S3.new do |s3|
+ s3.access_key_id = 'my_access_key_id'
+ s3.secret_access_key = 'my_secret_access_key'
+ s3.bucket = 'my_bucket'
+ s3.region = 'my_region'
+ end
+ end
+
+ it 'should be a subclass of Syncer::Cloud::Base' do
+ Backup::Syncer::Cloud::S3.superclass.
+ should == Backup::Syncer::Cloud::Base
+ end
+
+ describe '#initialize' do
+ it 'should have defined the configuration properly' do
+ syncer.access_key_id.should == 'my_access_key_id'
+ syncer.secret_access_key.should == 'my_secret_access_key'
+ syncer.bucket.should == 'my_bucket'
+ syncer.region.should == 'my_region'
+ end
+
+ it 'should inherit default values from superclasses' do
+ # Syncer::Cloud::Base
+ syncer.concurrency_type.should == false
+ syncer.concurrency_level.should == 2
+
+ # Syncer::Base
+ syncer.path.should == 'backups'
+ syncer.mirror.should == false
+ syncer.directories.should == []
+ end
+
+ context 'when options are not set' do
+ it 'should use default values' do
+ syncer = Backup::Syncer::Cloud::S3.new
+ syncer.access_key_id.should == nil
+ syncer.secret_access_key.should == nil
+ syncer.bucket.should == nil
+ syncer.region.should == nil
+ end
+ end
+
+ context 'when setting configuration defaults' do
+ after { Backup::Configuration::Syncer::Cloud::S3.clear_defaults! }
+
+ it 'should use the configured defaults' do
+ Backup::Configuration::Syncer::Cloud::S3.defaults do |s3|
+ s3.access_key_id = 'default_access_key_id'
+ s3.secret_access_key = 'default_secret_access_key'
+ s3.bucket = 'default_bucket'
+ s3.region = 'default_region'
+ end
+ syncer = Backup::Syncer::Cloud::S3.new
+ syncer.access_key_id.should == 'default_access_key_id'
+ syncer.secret_access_key.should == 'default_secret_access_key'
+ syncer.bucket.should == 'default_bucket'
+ syncer.region.should == 'default_region'
+ end
+
+ it 'should override the configured defaults' do
+ Backup::Configuration::Syncer::Cloud::S3.defaults do |s3|
+ s3.access_key_id = 'old_access_key_id'
+ s3.secret_access_key = 'old_secret_access_key'
+ s3.bucket = 'old_bucket'
+ s3.region = 'old_region'
+ end
+ syncer = Backup::Syncer::Cloud::S3.new do |s3|
+ s3.access_key_id = 'new_access_key_id'
+ s3.secret_access_key = 'new_secret_access_key'
+ s3.bucket = 'new_bucket'
+ s3.region = 'new_region'
+ end
+
+ syncer.access_key_id.should == 'new_access_key_id'
+ syncer.secret_access_key.should == 'new_secret_access_key'
+ syncer.bucket.should == 'new_bucket'
+ syncer.region.should == 'new_region'
+ end
+ end # context 'when setting configuration defaults'
+ end # describe '#initialize'
+
+ describe '#connection' do
+ let(:connection) { mock }
+
+ before do
+ Fog::Storage.expects(:new).once.with(
+ :provider => 'AWS',
+ :aws_access_key_id => 'my_access_key_id',
+ :aws_secret_access_key => 'my_secret_access_key',
+ :region => 'my_region'