Permalink
Browse files

Initial checkin: backup_fu

  • Loading branch information...
0 parents commit ad99bdb525230f0ca1ec4cd72529dd1ef65b82c8 shantibraford committed Feb 7, 2008
Showing with 456 additions and 0 deletions.
  1. +20 −0 MIT-LICENSE
  2. +141 −0 README
  3. +31 −0 config/backup_fu.yml.advanced_example
  4. +6 −0 config/backup_fu.yml.example
  5. +1 −0 init.rb
  6. +188 −0 lib/backup_fu.rb
  7. +62 −0 tasks/backup_fu_tasks.rake
  8. +7 −0 test/backup_fu_test.rb
@@ -0,0 +1,20 @@
+Copyright (c) 2008 Shanti A. Braford
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
141 README
@@ -0,0 +1,141 @@
+BackupFu
+========
+
+The backup_fu plugin makes it redonkulous easy to:
+
+ A) dump your database and/or static files to tar/gzipped archives, and
+ B) upload these archives to a private Amazon S3 bucket for safekeeping
+
+Installation
+============
+
+The 'aws-s3' gem is required for backup_fu to function properly. Install with:
+
+ sudo gem install aws-s3
+
+
+Configuration
+=============
+
+Run the following to copy over the example backup_fu.yml config file:
+
+ rake backup_fu:setup
+
+This copies the example config file to: config/backup_fu.yml.
+
+Usage
+=====
+
+For the list of available rake tasks:
+
+ rake -T backup_fu
+
+Backing up your database:
+
+ rake backup
+
+Dumping your database:
+
+ rake backup_fu:dump
+
+Backing up your static files:
+
+ rake backup_fu:static
+
+Backing up both your database + static files:
+
+ rake backup_fu:all
+
+Advanced Configuration
+======================
+
+See vendor/plugins/backup_fu/config/backup_fu.yml.advanced_example for
+the list of advanced configuration options.
+
+Advanced options include:
+* specify static path(s) that should be backed up -- i.e. backup your entire 'public/static' directory
+* change default dump path from RAILS_ROOT/tmp/backup to whatever
+* specify fully-qualified 'mysqldump' path
+* disable tar/gzipping of database dump
+* enable 'nice' with level specification to prevent backup_fu from bogarding your server
+
+Cronjob Installation
+====================
+
+Here are some cron job examples.
+
+ # Backup just the database everyday at 1am
+ 0 1 * * * cd /apps/foo/current; RAILS_ENV=production rake backup > /dev/null
+
+ # Backup db + static @ 2am every 3 days, log the results to ~/backup.log (verbosity should be turned on if logging results)
+ 0 2 1-31/3 * * cd /u/apps/shanti.railsblog/current; RAILS_ENV=production rake backup_fu:all >> ~/backup.log
+
+
+Debugging
+=========
+
+--- Enabling Verbosity
+
+If you are experiencing any difficulties, the first thing you should do is enable verbosity by
+dropping this into config/backup_fu.yml:
+
+ verbose: true
+
+--- Mysqldump Issues
+
+If your 'mysqldump' command is not in your path, you will need to specify it explicitly.
+
+To see if mysqldump is in your path, execute:
+
+ which mysqldump
+
+If you see output like "/usr/bin/which: no mysqldump in (...)" then you will need to specify the path manually.
+
+Use 'locate mysqldump' or a similar tool to find the full path to your mysqldump utility.
+
+Place an entry like the following in your config/backup_fu.yml file:
+
+ mysqldump_path: /usr/local/mysql-standard-5.0.27-linux-i686/bin/mysqldump
+
+--- Database Connection Issues
+
+If you are seeing an error when running 'rake backup' like:
+
+ mysqldump: Got error: 2002: Can't connect to local MySQL server ...
+
+Make sure you are specifying the RAILS_ENV for the target environment. i.e. for production:
+
+ RAILS_ENV=production rake backup
+ or
+ rake backup RAILS_ENV=production
+
+--- Connection reset by peer
+
+When backing up, if you receive an error like:
+
+ rake aborted!
+ Connection reset by peer
+
+Chances are this is because your backup is huuge. There is currently no great solution for
+this problem.
+
+On some systems, I have backed up 4GB+ files without a hitch. On other machines, an 80mb
+backup was choking on the S3 upload. After 3 attempts it went through.
+
+Patching in some kind of email notification system on failure would probably be nice.
+
+Patches welcome =)
+
+--- Tiny Static file .tar.gz Archive (static files not actually getting archived)
+
+This may result if you are using a symlink for your static dir, such as:
+
+ public/static -> /shared/apps/foo/static
+
+The solution to this is to specify the absolute static path in config/backup_fu.yml:
+
+ static_paths: /shared/apps/foo/static
+
+
+
+Copyright (c) 2008 Shanti A. Braford, released under the MIT license
@@ -0,0 +1,31 @@
+ # The app_name is used as the backup filename prefix
+app_name: replace_me
+# You must create the s3_bucket specified below via an external tool like S3 Browser, etc
+s3_bucket: some-s3-bucket
+aws_access_key_id: --replace me with your AWS access key id--
+aws_secret_access_key: --replace me with your AWS secret access key--
+
+# Advanced customizations
+
+# Turn on verbose output for debugging purposes:
+verbose: true
+
+# If the user running the script cannot find 'mysqldump' in its path, specify the full path with:
+mysqldump_path: /usr/local/mysql/bin/mysqldump
+
+# Enable the ability to backup your application's static files with the 'static_paths' config key.
+# Specify either relative (to RAILS_ROOT) paths (space-delimited)
+static_paths: "public/static public/users"
+
+# Or fully-qualified path(s):
+static_paths: "/u/apps/foo/current/public/static"
+
+# The default dump base path is 'tmp/backups'. This can be changed with:
+dump_base_path: /tmp
+
+# To disable the tar/gzipping of the DB backup:
+disable_tar_gzip: true
+
+# To turn on using 'nice' to keep backup CPU processing to a minimum:
+enable_nice: true
+nice_level: 15
@@ -0,0 +1,6 @@
+# The app_name is used as the backup filename prefix
+app_name: replace_me
+# You must create the s3_bucket specified below via an external tool like S3 Browser, etc
+s3_bucket: some-s3-bucket
+aws_access_key_id: --replace me with your AWS access key id--
+aws_secret_access_key: --replace me with your AWS secret access key--
@@ -0,0 +1 @@
+require 'backup_fu'
@@ -0,0 +1,188 @@
+require 'yaml'
+require 'active_support'
+require 'aws/s3'
+
+class BackupFuConfigError < StandardError; end
+class S3ConnectError < StandardError; end
+
+class BackupFu
+
+ def initialize(period = 'daily')
+ @period = period
+ db_conf = YAML.load_file(File.join(RAILS_ROOT, 'config', 'database.yml'))
+ @db_conf = db_conf[RAILS_ENV].symbolize_keys
+ @fu_conf = YAML.load_file(File.join(RAILS_ROOT, 'config', 'backup_fu.yml')).symbolize_keys
+ @verbose = !@fu_conf[:verbose].nil?
+ check_conf
+ create_dirs
+ end
+
+ def dump
+ host, port, password = '', '', ''
+ if @db_conf.has_key?(:host) && @db_conf[:host] != 'localhost'
+ host = "--host=#{@db_conf[:host]}"
+ end
+ if @db_conf.has_key?(:port)
+ port = "--port=#{@db_conf[:port]}"
+ end
+ if @db_conf.has_key?(:password) && !@db_conf[:password].blank?
+ password = "--password=#{@db_conf[:password]}"
+ end
+ full_dump_path = File.join(dump_base_path, db_filename)
+ cmd = niceify "#{mysqldump_path} --complete-insert --skip-extended-insert #{host} #{port} --user=#{@db_conf[:username]} #{password} #{@db_conf[:database]} > #{full_dump_path}"
+ puts cmd if @verbose
+ `#{cmd}`
+
+ if !@fu_conf[:disable_tar_gzip]
+
+ tar_path = File.join(dump_base_path, db_filename_tarred)
+
+ # TAR it up
+ cmd = niceify "tar -cf #{tar_path} -C #{dump_base_path} #{db_filename}"
+ puts "\nTar: #{cmd}\n" if @verbose
+ `#{cmd}`
+
+ # GZip it up
+ cmd = niceify "gzip -f #{tar_path}"
+ puts "\nGzip: #{cmd}" if @verbose
+ `#{cmd}`
+ end
+
+ end
+
+ def backup
+ dump
+ establish_s3_connection
+
+ file = final_db_dump_path()
+ puts "\nBacking up to S3: #{file}\n" if @verbose
+
+ AWS::S3::S3Object.store(File.basename(file), open(file), @fu_conf[:s3_bucket], :access => :private)
+
+ end
+
+ ## Static-file Dump/Backup methods
+
+ def dump_static
+ if !@fu_conf[:static_paths]
+ raise BackupFuConfigError, 'No static paths are defined in config/backup_fu.yml. See README.'
+ end
+ @paths = @fu_conf[:static_paths].split(' ')
+ path_num = 0
+ @paths.each do |p|
+ if p.first != '/'
+ # Make into an Absolute path:
+ p = File.join(RAILS_ROOT, p)
+ end
+
+ puts "Static Path: #{p}" if @verbose
+ if path_num == 0
+ tar_switch = 'c' # for create
+ else
+ tar_switch = 'r' # for append
+ end
+
+ # TAR
+ cmd = niceify "tar -#{tar_switch}f #{static_tar_path} #{p}"
+ puts "\nTar: #{cmd}\n" if @verbose
+ `#{cmd}`
+
+ path_num += 1
+ end
+
+ # GZIP
+ cmd = niceify "gzip -f #{static_tar_path}"
+ puts "\nGzip: #{cmd}" if @verbose
+ `#{cmd}`
+
+ end
+
+ def backup_static
+ dump_static
+ establish_s3_connection
+
+ file = final_static_dump_path()
+ puts "\nBacking up Static files to S3: #{file}\n" if @verbose
+
+ AWS::S3::S3Object.store(File.basename(file), open(file), @fu_conf[:s3_bucket], :access => :private)
+
+ end
+
+ private
+
+ def establish_s3_connection
+ unless AWS::S3::Base.connected?
+ AWS::S3::Base.establish_connection!(
+ :access_key_id => @fu_conf[:aws_access_key_id],
+ :secret_access_key => @fu_conf[:aws_secret_access_key]
+ )
+ end
+ raise S3ConnectError, "\nERROR: Connection to Amazon S3 failed." unless AWS::S3::Base.connected?
+ end
+
+ def check_conf
+ if @fu_conf[:app_name] == 'replace_me'
+ raise BackupFuConfigError, 'Application name (app_name) key not set in config/backup_fu.yml.'
+ elsif @fu_conf[:s3_bucket] == 'some-s3-bucket'
+ raise BackupFuConfigError, 'S3 bucket (s3_bucket) not set in config/backup_fu.yml. This bucket must be created using an external S3 tool like S3 Browser for OS X, or JetS3t (Java-based, cross-platform).'
+ elsif @fu_conf[:aws_access_key_id].include?('--replace me') || @fu_conf[:aws_secret_access_key].include?('--replace me')
+ raise BackupFuConfigError, 'AWS Access Key Id or AWS Secret Key not set in config/backup_fu.yml.'
+ end
+ end
+
+ def mysqldump_path
+ @fu_conf[:mysqldump_path] || 'mysqldump'
+ end
+
+ def dump_base_path
+ @fu_conf[:dump_base_path] || File.join(RAILS_ROOT, 'tmp', 'backup')
+ end
+
+ def db_filename
+ date_formatted = Time.now.strftime("%Y-%m-%d")
+ "#{@fu_conf[:app_name]}_#{date_formatted}_db.sql"
+ end
+
+ def db_filename_tarred
+ db_filename.gsub('.sql', '.tar')
+ end
+
+ def final_db_dump_path
+ if @fu_conf[:disable_tar_gzip]
+ filename = db_filename
+ else
+ filename = db_filename.gsub('.sql', '.tar.gz')
+ end
+ File.join(dump_base_path, filename)
+ end
+
+ def static_tar_path
+ date_formatted = Time.now.strftime("%Y-%m-%d")
+ f = "#{@fu_conf[:app_name]}_#{date_formatted}_static.tar"
+ File.join(dump_base_path, f)
+ end
+
+ def final_static_dump_path
+ date_formatted = Time.now.strftime("%Y-%m-%d")
+ f = "#{@fu_conf[:app_name]}_#{date_formatted}_static.tar.gz"
+ File.join(dump_base_path, f)
+ end
+
+ def create_dirs
+ ensure_directory_exists(dump_base_path)
+ end
+
+ def ensure_directory_exists(dir)
+ FileUtils.mkdir_p(dir) unless File.exist?(dir)
+ end
+
+ def niceify(cmd)
+ if @fu_conf[:enable_nice]
+ "nice -n -#{@fu_conf[:nice_level]} #{cmd}"
+ else
+ cmd
+ end
+ end
+
+
+end
Oops, something went wrong.

0 comments on commit ad99bdb

Please sign in to comment.