Skip to content
Browse files

Initial import

  • Loading branch information...
0 parents commit 27d0461fd33a1d357aaac0a72aec94c4bbb9fc9c @FooBarWidget FooBarWidget committed Apr 18, 2011
Showing with 310 additions and 0 deletions.
  1. +59 −0 README.md
  2. +26 −0 backup-mysql
  3. +27 −0 deny
  4. +14 −0 display-queue
  5. +38 −0 fix-capistrano-permissions
  6. +20 −0 gc-git-repos
  7. +21 −0 notify-if-queue-becomes-large
  8. +40 −0 permit
  9. +11 −0 purge-queue
  10. +34 −0 shared.rb
  11. +12 −0 silence-unless-failed
  12. +4 −0 truncate
  13. +4 −0 watch-queue
59 README.md
@@ -0,0 +1,59 @@
+# Phusion Server Tools
+
+A collection of server administration tools that we use. Everything is
+written in Ruby and are designed to work with Debian. These scripts may
+work with other operating systems or distributions as well, but it's not
+tested.
+
+Install with:
+
+ git clone ... /tools
+
+It's not necessary to install to /tools, you can install to anywhere, but this document assumes that you have installed to /tools.
+
+Each tool has its own prerequities, but here are some common prerequities:
+
+ * Ruby (obviously)
+ * `pv` - `apt-get install pv`. Not required but very useful; allows display of progress bars.
+
+
+## Included tools
+
+### backup-mysql - Rotating MySQL dumps
+
+A script which backs up all MySQL databases to `/var/backups/mysql`. At most 10 backups are kept. All backups are compressed with gzip.
+
+It uses `mysql` to obtain a list of databases and `mysqldump` to dump the database contents. If you want to run this script unattended you should therefore set the right login information in `~/.my.cnf`, sections `[mysql]` and `[mysqldump]`.
+
+Make it run daily at 0:00 AM in cron:
+
+ 0 0 * * * /tools/silence-unless-failed /tools/backup-mysql
+
+### permit and deny - Easily set fine-grained permissions using ACLs
+
+Scripts for giving a user access to a single file, or recursive access to a directory, using ACLs. The standard `setfacl` tool is too hard to use and sometimes does stupid things such as unexpectedly making files executable. These scripts are simple and work as expected.
+
+ # Recursively give web server read-only access to /webapps/foo.
+ /tools/permit www-data /webapps/foo
+
+ # Recursively give user 'deploy' read-write access to /webapps/bar.
+ /tools/permit deploy /webapps/bar --read-write
+
+ # Recursively remove all ACLs for user 'joe' on /secrets/area66.
+ /tools/deny joe /secrets/area66
+
+You need the `getfacl` and `setfacl` commands:
+
+ apt-get install acl
+
+You must also make sure your filesystem is mounted with ACL support, e.g.:
+
+ mount -o remount,acl /
+
+Don't forget to update /etc/fstab too.
+
+### silcence-unless-failed
+
+Runs the given command but only print its output (both STDOUT and STDERR) if its exit code is non-zero. The script's own exit code is the same as the command's exit code.
+
+ /tools/silence-unless-failed my-command arg1 arg2 --arg3
26 backup-mysql
@@ -0,0 +1,26 @@
+#!/usr/bin/env ruby
+require File.expand_path(File.dirname(__FILE__) + '/shared')
+BACKUP_DIR_ROOT = "/var/backups/mysql"
+MAX_BACKUPS = 10
+
+databases = `echo show databases | mysql`.strip.split("\n")
+databases.shift
+databases.delete("information_schema")
+databases.delete("mysql")
+
+now = Time.now.strftime("%Y-%m-%d-%H:%M:%S")
+backup_dir = "#{BACKUP_DIR_ROOT}/#{now}"
+
+sh "mkdir -p #{backup_dir}"
+for database in databases
+ sh "mysqldump #{database} | gzip --best | #{pv_or_cat} > #{backup_dir}/#{database}.sql.gz"
+end
+
+puts "Cleaning up..."
+dirs = Dir["#{BACKUP_DIR_ROOT}/*"].sort.reverse
+keep = dirs[0..MAX_BACKUPS]
+delete = dirs - keep
+delete.each do |dir|
+ sh "rm -rf #{dir}"
+end
+sh "chmod -R o-rwx #{BACKUP_DIR_ROOT}"
27 deny
@@ -0,0 +1,27 @@
+#!/usr/bin/env ruby
+def help
+ puts "Usage: permit <USERNAME> <DIR>"
+ puts "Removes access for USERNAME to directory DIR."
+ exit 1
+end
+
+def sh(command, *args)
+ puts "#{command} #{args.join(' ')}"
+ if !system(command, *args)
+ STDERR.puts "*** ERROR"
+ exit 1
+ end
+end
+
+help if ARGV.size != 2
+username, dir = ARGV
+puts "cd #{dir}"
+Dir.chdir(dir) do
+ executable_files = "/tmp/executable-files.#{$$}"
+ sh "find -type f -executable -print0 > #{executable_files}"
+ sh "find -print0 | xargs -0 -n 1000 -r setfacl -x user:#{username}"
+ sh "find -type d -print0 | xargs -0 -n 1000 -r setfacl -d -x user:#{username}"
+ sh "find -type f -print0 | xargs -0 -n 1000 -r chmod -x"
+ sh "cat #{executable_files} | xargs -0 -n 1000 -r chmod +x"
+ sh "rm -f #{executable_files}"
+end
14 display-queue
@@ -0,0 +1,14 @@
+#!/usr/bin/env ruby
+puts "Name Messages = Ready + Unack Consumers Memory (MB)"
+puts "-------------------------------------------------------------------------------"
+lines = `rabbitmqctl -q list_queues name messages messages_ready messages_unacknowledged consumers memory`
+lines = lines.split("\n")
+lines.each do |line|
+ name, messages, ready, unack, consumers, memory = line.split(/[ \t]+/)
+ messages = messages.to_i
+ ready = ready.to_i
+ unack = unack.to_i
+ consumers = consumers.to_i
+ memory = memory.to_i / 1024.0 / 1024
+ printf("%-23s %-8d = %-5d + %-5d %-9d %1.f\n", name, messages, ready, unack, consumers, memory)
+end
38 fix-capistrano-permissions
@@ -0,0 +1,38 @@
+#!/bin/bash
+CAPISTRANO_DIR=/u/apps
+WWW_USER=www-data
+
+export PATH="/tools:$PATH"
+
+chmod -R g+w,o-rwx $CAPISTRANO_DIR
+setfacl -m user:$WWW_USER:--x $CAPISTRANO_DIR
+setfacl -d -m user:$WWW_USER:r-x $CAPISTRANO_DIR
+for DIR in $CAPISTRANO_DIR/*; do
+ # Give the web server read-only access to everything.
+ # We tighten up permissions in later commands.
+ permit $WWW_USER $DIR
+
+ # Make the application directory itself executable-only
+ # by the web server.
+ if [[ -d $DIR/releases || -d $DIR/shared ]]; then
+ setfacl -m user:$WWW_USER:--x $DIR
+ setfacl -d -m user:$WWW_USER:--x $DIR
+ fi
+
+ # Make the 'releases' directory and immediate
+ # subdirectories executable-only by the web server.
+ if [[ -d $DIR/releases ]]; then
+ setfacl -m user:$WWW_USER:--x $DIR/releases
+ setfacl -m user:$WWW_USER:--x $DIR/releases/*
+ fi
+
+ # Deny web server access to the 'shared' directory.
+ if [[ -d $DIR/shared ]]; then
+ deny $WWW_USER $DIR/shared
+ # If you store attachment files in the 'shared'
+ # directory then you can allow only access to that:
+ #
+ # setfacl -m user:$WWW_USER:--x $DIR/shared
+ # permit $DIR/shared/attachments
+ fi
+done
20 gc-git-repos
@@ -0,0 +1,20 @@
+#!/usr/bin/env ruby
+# This script is automatically called in a cron job by root.
+require 'etc'
+
+DIRS = Dir[
+ "/u/apps/**/.git"
+]
+DIRS.each do |dir|
+ puts "# Garbage collecting #{dir}"
+ Dir.chdir(dir) do
+ stat = File.stat(dir)
+ username = Etc.getpwuid(stat.uid).name
+ command = "su -c 'git gc' #{username}"
+ puts command
+ if !system(command)
+ abort "'git gc' failed"
+ end
+ end
+ puts
+end
21 notify-if-queue-becomes-large
@@ -0,0 +1,21 @@
+#!/usr/bin/env ruby
+THRESHOLD=1000
+
+warnings = []
+`rabbitmqctl list_queues -q name messages`.split("\n").each do |line|
+ name, messages = line.split(/[ \t]+/)
+ messages = messages.to_i
+ if messages > THRESHOLD
+ warnings << "Queue '#{name}' has more than #{THRESHOLD} messages: #{messages}"
+ end
+end
+
+if !warnings.empty?
+ IO.popen("sendmail -t", "w") do |f|
+ f.puts "To: info@phusion.nl"
+ f.puts "From: noreply@unionstationapp.com"
+ f.puts "Subject: [Union Station] Queue going out of control"
+ f.puts
+ f.puts warnings.join("\n")
+ end
+end
40 permit
@@ -0,0 +1,40 @@
+#!/usr/bin/env ruby
+def help
+ puts "Usage: permit <USERNAME> <DIR> [--read-write]"
+ puts "Give USERNAME read-only or read-write permission to directory DIR."
+ puts "Default is read-only unless --read-write is given."
+ exit 1
+end
+
+def sh(command, *args)
+ puts "#{command} #{args.join(' ')}"
+ if !system(command, *args)
+ STDERR.puts "*** ERROR"
+ exit 1
+ end
+end
+
+help if ARGV.size != 2 && ARGV.size != 3
+username, dir, read_write = ARGV
+if read_write && read_write != "--read-write"
+ STDERR.puts "*** Invalid option #{read_write}"
+ help
+end
+
+puts "cd #{dir}"
+Dir.chdir(dir) do
+ executable_files = "/tmp/executable-files.#{$$}"
+ sh "find -type f -executable -print0 > #{executable_files}"
+ if read_write
+ sh "find -type f -print0 | xargs -0 -n 1000 -r setfacl -m user:#{username}:rw-"
+ sh "find -type d -print0 | xargs -0 -n 1000 -r setfacl -m user:#{username}:rwx"
+ sh "find -type d -print0 | xargs -0 -n 1000 -r setfacl -d -m user:#{username}:rwx"
+ else
+ sh "find -type f -print0 | xargs -0 -n 1000 -r setfacl -m user:#{username}:r-"
+ sh "find -type d -print0 | xargs -0 -n 1000 -r setfacl -m user:#{username}:r-x"
+ sh "find -type d -print0 | xargs -0 -n 1000 -r setfacl -d -m user:#{username}:r-x"
+ end
+ sh "find -type f -print0 | xargs -0 -n 1000 -r chmod -x"
+ sh "cat #{executable_files} | xargs -0 -n 1000 -r chmod +x"
+ sh "rm -f #{executable_files}"
+end
11 purge-queue
@@ -0,0 +1,11 @@
+#!/usr/bin/env ruby
+if !ARGV[0]
+ STDERR.puts "You must specify a queue name."
+ exit 1
+end
+require 'rubygems'
+require 'bunny'
+b = Bunny.new
+b.start
+q = b.queue(ARGV[0], :durable => true)
+q.purge
34 shared.rb
@@ -0,0 +1,34 @@
+def sh(command, *args)
+ puts "# #{command} #{args.join(' ')}"
+ if !system(command, *args)
+ STDERR.puts "*** ERROR"
+ exit 1
+ end
+end
+
+# Check whether the specified command is in $PATH, and return its
+# absolute filename. Returns nil if the command is not found.
+#
+# This function exists because system('which') doesn't always behave
+# correctly, for some weird reason.
+def find_command(name)
+ name = name.to_s
+ ENV['PATH'].to_s.split(File::PATH_SEPARATOR).detect do |directory|
+ path = File.join(directory, name)
+ if File.file?(path) && File.executable?(path)
+ return path
+ end
+ end
+ return nil
+end
+
+# Returns "pv" if that command is installed, or "cat" if not.
+# "pv" is the Pipe Viewer tool, very useful for displaying
+# progress bars in pipe operations (apt-get install pv).
+def pv_or_cat
+ if find_command('pv')
+ return 'pv'
+ else
+ return 'cat'
+ end
+end
12 silence-unless-failed
@@ -0,0 +1,12 @@
+#!/bin/bash
+# Executes the given command, silencing all its output.
+# Its output is only printed to STDERR if the command failed.
+
+output="/tmp/output.$$"
+"$@" >$output 2>&1
+exit_code=$?
+if [[ $exit_code != 0 ]]; then
+ cat $output >&2
+fi
+rm -f $output
+exit $exit_code
4 truncate
@@ -0,0 +1,4 @@
+#!/bin/bash
+for F in "$@"; do
+ echo -n > "$F"
+done
4 watch-queue
@@ -0,0 +1,4 @@
+#!/bin/bash
+dir=`dirname "$0"`
+cd "$dir"
+exec watch -d ./display-queue

0 comments on commit 27d0461

Please sign in to comment.
Something went wrong with that request. Please try again.