Skip to content
Browse files

adding background job as a plugin, trying to reduce external dependen…

…cies, migrating away from BackgroundRB
  • Loading branch information...
1 parent 91a6919 commit 5cfdee3b74121856e5c97510a5290800c11ad853 @mikehelmick committed Aug 2, 2009
Showing with 6,820 additions and 0 deletions.
  1. +8 −0 db/migrate/20090803034724_bj_migration0.rb
  2. +679 −0 script/bj
  3. +3 −0 vendor/plugins/bj/HISTORY
  4. +142 −0 vendor/plugins/bj/README
  5. +22 −0 vendor/plugins/bj/Rakefile
  6. +679 −0 vendor/plugins/bj/bin/bj
  7. +33 −0 vendor/plugins/bj/init.rb
  8. +95 −0 vendor/plugins/bj/install.rb
  9. +443 −0 vendor/plugins/bj/lib/arrayfields.rb
  10. +118 −0 vendor/plugins/bj/lib/attributes.rb
  11. +87 −0 vendor/plugins/bj/lib/bj.rb
  12. +161 −0 vendor/plugins/bj/lib/bj/api.rb
  13. +120 −0 vendor/plugins/bj/lib/bj/attributes.rb
  14. +72 −0 vendor/plugins/bj/lib/bj/bj.rb
  15. +4 −0 vendor/plugins/bj/lib/bj/errors.rb
  16. +112 −0 vendor/plugins/bj/lib/bj/joblist.rb
  17. +50 −0 vendor/plugins/bj/lib/bj/logger.rb
  18. +357 −0 vendor/plugins/bj/lib/bj/runner.rb
  19. +86 −0 vendor/plugins/bj/lib/bj/stdext.rb
  20. +384 −0 vendor/plugins/bj/lib/bj/table.rb
  21. +111 −0 vendor/plugins/bj/lib/bj/util.rb
  22. +153 −0 vendor/plugins/bj/lib/fattr.rb
  23. +60 −0 vendor/plugins/bj/lib/main.rb
  24. +515 −0 vendor/plugins/bj/lib/main/base.rb
  25. +100 −0 vendor/plugins/bj/lib/main/cast.rb
  26. +20 −0 vendor/plugins/bj/lib/main/factories.rb
  27. +470 −0 vendor/plugins/bj/lib/main/getoptlong.rb
  28. +51 −0 vendor/plugins/bj/lib/main/logger.rb
  29. +42 −0 vendor/plugins/bj/lib/main/mode.rb
  30. +699 −0 vendor/plugins/bj/lib/main/parameter.rb
  31. +12 −0 vendor/plugins/bj/lib/main/softspoken.rb
  32. +38 −0 vendor/plugins/bj/lib/main/stdext.rb
  33. +211 −0 vendor/plugins/bj/lib/main/usage.rb
  34. +91 −0 vendor/plugins/bj/lib/main/util.rb
  35. +25 −0 vendor/plugins/bj/lib/orderedautohash.rb
  36. +200 −0 vendor/plugins/bj/lib/orderedhash.rb
  37. +299 −0 vendor/plugins/bj/lib/systemu.rb
  38. +55 −0 vendor/plugins/bj/script/bj
  39. +4 −0 vendor/plugins/bj/tasks/bj_tasks.rake
  40. +8 −0 vendor/plugins/bj/test/bj_test.rb
  41. +1 −0 vendor/plugins/bj/uninstall.rb
View
8 db/migrate/20090803034724_bj_migration0.rb
@@ -0,0 +1,8 @@
+ class BjMigration0 < ActiveRecord::Migration
+ def self.up
+ Bj::Table.each{|table| table.up}
+ end
+ def self.down
+ Bj::Table.reverse_each{|table| table.down}
+ end
+ end
View
679 script/bj
@@ -0,0 +1,679 @@
+#! /usr/bin/env ruby
+
+require "bj"
+require "main"
+
+Main {
+ usage["description"] = <<-txt
+ ________________________________
+ Overview
+ --------------------------------
+
+ Backgroundjob (Bj) is a brain dead simple zero admin background priority queue
+ for Rails. Bj is robust, platform independent (including windows), and
+ supports internal or external manangement of the background runner process.
+
+ Jobs can be submitted to the queue directly using the api or from the command
+ line using the ./script/bj:
+
+ api:
+ Bj.submit 'cat /etc/password'
+
+ command line:
+ bj submit cat /etc/password
+
+ Bj's priority queue lives in the database and is therefore durable - your jobs
+ will live across an app crash or machine reboot. The job management is
+ comprehensive capturing stdout, stderr, exit_status, and temporal statistics
+ about each job:
+
+ jobs = Bj.submit array_of_commands, :priority => 42
+
+ ...
+
+ jobs.each do |job|
+ if job.finished?
+ p job.stdout
+ p job.stderr
+ p job.exit_status
+ p job.started_at
+ p job.finished_at
+ end
+ end
+
+ In addition the background runner process logs all commands run and their
+ exit_status to a log named using the following convention:
+
+ rails_root/log/bj.\#{ HOSTNAME }.\#{ RAILS_ENV }.log
+
+ Bj allows you to submit jobs to multiple databases; for instance, if your
+ application is running in development mode you may do:
+
+ Bj.in :production do
+ Bj.submit 'my_job.exe'
+ end
+
+ Bj manages the ever growing list of jobs ran by automatically archiving them
+ into another table (by default jobs > 24 hrs old are archived) to prevent the
+ jobs table from becoming bloated and huge.
+
+ All Bj's tables are namespaced and accessible via the Bj module:
+
+ Bj.table.job.find(:all) # jobs table
+ Bj.table.job_archive.find(:all) # archived jobs
+ Bj.table.config.find(:all) # configuration and runner state
+
+ Bj always arranges for submitted jobs to run with a current working directory
+ of RAILS_ROOT and with the correct RAILS_ENV setting. For example, if you
+ submit a job in production it will have ENV['RAILS_ENV'] == 'production'.
+
+ When Bj manages the background runner it will never outlive the rails
+ application - it is started and stopped on demand as the rails app is started
+ and stopped. This is also true for ./script/console - Bj will automatically
+ fire off the background runner to process jobs submitted using the console.
+
+ Bj ensures that only one background process is running for your application -
+ firing up three mongrels or fcgi processes will result in only one background
+ runner being started. Note that the number of background runners does not
+ determine throughput - that is determined primarily by the nature of the jobs
+ themselves and how much work they perform per process.
+
+
+ ________________________________
+ Architecture
+ --------------------------------
+
+ If one ignores platform specific details the design of Bj is quite simple: the
+ main Rails application submits jobs to table, stored in the database. The act
+ of submitting triggers exactly one of two things to occur:
+
+ 1) a new long running background runner to be started
+
+ 2) an existing background runner to be signaled
+
+ The background runner refuses to run two copies of itself for a given
+ hostname/rails_env combination. For example you may only have one background
+ runner processing jobs on localhost in development mode.
+
+ The background runner, under normal circumstances, is managed by Bj itself -
+ you need do nothing to start, monitor, or stop it - it just works. However,
+ some people will prefer manage their own background process, see 'External
+ Runner' section below for more on this.
+
+ The runner simply processes each job in a highest priority oldest-in fashion,
+ capturing stdout, stderr, exit_status, etc. and storing the information back
+ into the database while logging it's actions. When there are no jobs to run
+ the runner goes to sleep for 42 seconds; however this sleep is interuptable,
+ such as when the runner is signaled that a new job has been submitted so,
+ under normal circumstances there will be zero lag between job submission and
+ job running for an empty queue.
+
+
+ ________________________________
+ External Runner / Clustering
+ --------------------------------
+
+ For the paranoid control freaks out there (myself included) it is quite
+ possible to manage and monitor the runner process manually. This can be
+ desirable in production setups where monitoring software may kill leaking
+ rails apps periodically.
+
+ Recalling that Bj will only allow one copy of itself to process jobs per
+ hostname/rails_env pair we can simply do something like this in cron
+
+ cmd = bj run --forever \\
+ --rails_env=development \\
+ --rails_root=/Users/ahoward/rails_root
+
+ */15 * * * * $cmd
+
+ this will simply attempt the start the background runner every 15 minutes if,
+ and only if, it's not *already* running.
+
+ In addtion to this you'll want to tell Bj not to manage the runner itself
+ using
+
+ Bj.config["production.no_tickle"] = true
+
+ Note that, for clusting setups, it's as simple as adding a crontab and config
+ entry like this for each host. Because Bj throttles background runners per
+ hostname this will allow one runner per hostname - making it quite simple to
+ cluster three nodes behind a besieged rails application.
+
+
+ ________________________________
+ Designing Jobs
+ --------------------------------
+
+ Bj runs it's jobs as command line applications. It ensures that all jobs run
+ in RAILS_ROOT so it's quite natural to apply a pattern such as
+
+ mkdir ./jobs
+ edit ./jobs/background_job_to_run
+
+ ...
+
+ Bj.submit "./jobs/background_job_to_run"
+
+ If you need to run you jobs under an entire rails environment you'll need to
+ do this:
+
+ Bj.submit "./script/runner ./jobs/background_job_to_run"
+
+ Obviously "./script/runner" loads the rails environment for you. It's worth
+ noting that this happens for each job and that this is by design: the reason
+ is that most rails applications leak memory like a sieve so, if one were to
+ spawn a long running process that used the application code base you'd have a
+ lovely doubling of memory usage on you app servers. Although loading the
+ rails environment for each background job requires a little time, a little
+ cpu, and a lot less memory. A future version of Bj will provide a way to load
+ the rails environment once and to process background jobs in this environment,
+ but anyone wanting to use this in production will be required to duct tape
+ their entire chest and have a team of oxen rip off the tape without screaming
+ to prove steelyness of spirit and profound understanding of the other side.
+
+ Don't forget that you can submit jobs with command line arguments:
+
+ Bj.submit "./jobs/a.rb 1 foobar --force"
+
+ and that you can do powerful things by passing stdin to a job that powers
+ through a list of work. For instance, assume a "./jobs/bulkmail" job
+ resembling
+
+ STDIN.each do |line|
+ address = line.strip
+ mail_message_to address
+ end
+
+ then you could
+
+ stdin = [
+ "foo@bar.com",
+ "bar@foo.com",
+ "ara.t.howard@codeforpeople.com",
+ ]
+
+ Bj.submit "./script/runner ./jobs/bulkmail", :stdin => stdin
+
+ and all those emails would be sent in the background.
+
+ Bj's power is putting jobs in the background in a simple and robust fashion.
+ It's your task to build intelligent jobs that leverage batch processing, and
+ other, possibilities. The upshot of building tasks this way is that they are
+ quite easy to test before submitting them from inside your application.
+
+
+ ________________________________
+ Install
+ --------------------------------
+
+ Bj can be installed two ways: as a plugin or via rubygems
+
+ plugin:
+ 1) ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
+ 2) ./script/bj setup
+
+ gem:
+ 1) $sudo gem install bj
+ 2) add "require 'bj'" to config/environment.rb
+ 3) bj setup
+
+ ________________________________
+ Api
+ --------------------------------
+
+ submit jobs for background processing. 'jobs' can be a string or array of
+ strings. options are applied to each job in the 'jobs', and the list of
+ submitted jobs is always returned. options (string or symbol) can be
+
+ :rails_env => production|development|key_in_database_yml
+ when given this keyword causes bj to submit jobs to the
+ specified database. default is RAILS_ENV.
+
+ :priority => any number, including negative ones. default is zero.
+
+ :tag => a tag added to the job. simply makes searching easier.
+
+ :env => a hash specifying any additional environment vars the background
+ process should have.
+
+ :stdin => any stdin the background process should have. must respond_to
+ to_s
+
+ eg:
+
+ jobs = Bj.submit 'echo foobar', :tag => 'simple job'
+
+ jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority => 42
+
+ jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
+
+ jobs = Bj.submit './script/runner /dev/stdin',
+ :stdin => 'p RAILS_ENV',
+ :tag => 'dynamic ruby code'
+
+ jobs Bj.submit array_of_commands, :priority => 451
+
+ when jobs are run, they are run in RAILS_ROOT. various attributes are
+ available *only* once the job has finished. you can check whether or not a
+ job is finished by using the #finished method, which simple does a reload and
+ checks to see if the exit_status is non-nil.
+
+ eg:
+
+ jobs = Bj.submit list_of_jobs, :tag => 'important'
+ ...
+
+ jobs.each do |job|
+ if job.finished?
+ p job.exit_status
+ p job.stdout
+ p job.stderr
+ end
+ end
+
+ See lib/bj/api.rb for more details.
+
+ ________________________________
+ Sponsors
+ --------------------------------
+ http://quintess.com/
+ http://www.engineyard.com/
+ http://igicom.com/
+ http://eparklabs.com/
+
+ http://your_company.com/ <<-- (targeted marketing aimed at *you*)
+
+ ________________________________
+ Version
+ --------------------------------
+ #{ Bj.version }
+txt
+
+ usage["uris"] = <<-txt
+ http://codeforpeople.com/lib/ruby/
+ http://rubyforge.org/projects/codeforpeople/
+ http://codeforpeople.rubyforge.org/svn/rails/plugins/
+ txt
+
+ author "ara.t.howard@gmail.com"
+
+ option("rails_root", "R"){
+ description "the rails_root will be guessed unless you set this"
+ argument_required
+ default RAILS_ROOT
+ }
+
+ option("rails_env", "E"){
+ description "set the rails_env"
+ argument_required
+ default RAILS_ENV
+ }
+
+ option("log", "l"){
+ description "set the logfile"
+ argument_required
+ default STDERR
+ }
+
+
+ mode "migration_code" do
+ description "dump migration code on stdout"
+
+ def run
+ puts Bj.table.migration_code
+ end
+ end
+
+ mode "generate_migration" do
+ description "generate a migration"
+
+ def run
+ Bj.generate_migration
+ end
+ end
+
+ mode "migrate" do
+ description "migrate the db"
+
+ def run
+ Bj.migrate
+ end
+ end
+
+ mode "setup" do
+ description "generate a migration and migrate"
+
+ def run
+ set_rails_env(argv.first) if argv.first
+ Bj.setup
+ end
+ end
+
+ mode "plugin" do
+ description "dump the plugin into rails_root"
+
+ def run
+ Bj.plugin
+ end
+ end
+
+ mode "run" do
+ description "start a job runnner, possibly as a daemon"
+
+ option("--forever"){}
+ option("--ppid"){
+ argument :required
+ cast :integer
+ }
+ option("--wait"){
+ argument :required
+ cast :integer
+ }
+ option("--limit"){
+ argument :required
+ cast :integer
+ }
+ option("--redirect"){
+ argument :required
+ }
+ option("--daemon"){}
+
+ def run
+ options = {}
+
+=begin
+ %w[ forever ].each do |key|
+ options[key.to_sym] = true if param[key].given?
+ end
+=end
+
+ %w[ forever ppid wait limit ].each do |key|
+ options[key.to_sym] = param[key].value if param[key].given?
+ end
+
+#p options
+#exit
+ if param["redirect"].given?
+ open(param["redirect"].value, "a+") do |fd|
+ STDERR.reopen fd
+ STDOUT.reopen fd
+ end
+ STDERR.sync = true
+ STDOUT.sync = true
+ end
+
+ trap("SIGTERM"){
+ info{ "SIGTERM" }
+ exit
+ }
+
+ if param["daemon"].given?
+ daemon{ Bj.run options }
+ else
+ Bj.run options
+ end
+ end
+ end
+
+ mode "submit" do
+ keyword("file"){
+ argument :required
+ attr
+ }
+
+ def run
+ joblist = Bj.joblist.for argv.join(' ')
+
+ case file
+ when "-"
+ joblist.push(Bj.joblist.jobs_from_io(STDIN))
+ when "--", "---"
+ joblist.push(Bj.joblist.jobs_from_yaml(STDIN))
+ else
+ open(file){|io| joblist.push(Bj.joblist.jobs_from_io(io)) }
+ end
+
+ jobs = Bj.submit joblist, :no_tickle => true
+
+ oh = lambda{|job| OrderedHash["id", job.id, "command", job.command]}
+
+ y jobs.map{|job| oh[job]}
+ end
+ end
+
+ mode "list" do
+ def run
+ Bj.transaction do
+ y Bj::Table::Job.find(:all).map(&:to_hash)
+ end
+ end
+ end
+
+ mode "set" do
+ argument("key"){ attr }
+
+ argument("value"){ attr }
+
+ option("hostname", "H"){
+ argument :required
+ default Bj.hostname
+ attr
+ }
+
+ option("cast", "c"){
+ argument :required
+ default "to_s"
+ attr
+ }
+
+ def run
+ Bj.transaction do
+ Bj.config.set(key, value, :hostname => hostname, :cast => cast)
+ y Bj.table.config.for(:hostname => hostname)
+ end
+ end
+ end
+
+ mode "config" do
+ option("hostname", "H"){
+ argument :required
+ default Bj.hostname
+ }
+
+ def run
+ Bj.transaction do
+ y Bj.table.config.for(:hostname => param["hostname"].value)
+ end
+ end
+ end
+
+ mode "pid" do
+ option("hostname", "H"){
+ argument :required
+ default Bj.hostname
+ }
+
+ def run
+ Bj.transaction do
+ config = Bj.table.config.for(:hostname => param["hostname"].value)
+ puts config[ "#{ RAILS_ENV }.pid" ] if config
+ end
+ end
+ end
+
+
+ def run
+ help!
+ end
+
+ def before_run
+ self.logger = param["log"].value
+ Bj.logger = logger
+ set_rails_root(param["rails_root"].value) if param["rails_root"].given?
+ set_rails_env(param["rails_env"].value) if param["rails_env"].given?
+ end
+
+ def set_rails_root rails_root
+ ENV["RAILS_ROOT"] = rails_root
+ ::Object.instance_eval do
+ remove_const :RAILS_ROOT
+ const_set :RAILS_ROOT, rails_root
+ end
+ end
+
+ def set_rails_env rails_env
+ ENV["RAILS_ENV"] = rails_env
+ ::Object.instance_eval do
+ remove_const :RAILS_ENV
+ const_set :RAILS_ENV, rails_env
+ end
+ end
+
+ def daemon
+ ra, wa = IO.pipe
+ rb, wb = IO.pipe
+ if fork
+ at_exit{ exit! }
+ wa.close
+ r = ra
+ rb.close
+ w = wb
+ pid = r.gets
+ w.puts pid
+ Integer pid.strip
+ else
+ ra.close
+ w = wa
+ wb.close
+ r = rb
+ open("/dev/null", "r+") do |fd|
+ STDIN.reopen fd
+ STDOUT.reopen fd
+ STDERR.reopen fd
+ end
+ Process::setsid rescue nil
+ pid =
+ fork do
+ Dir::chdir RAILS_ROOT
+ File::umask 0
+ $DAEMON = true
+ yield
+ exit!
+ end
+ w.puts pid
+ r.gets
+ exit!
+ end
+ end
+}
+
+
+
+
+
+#
+# we setup a few things so the script works regardless of whether it was
+# called out of /usr/local/bin, ./script, or wherever. note that the script
+# does *not* require the entire rails application to be loaded into memory!
+# we could just load boot.rb and environment.rb, but this method let's
+# submitting and running jobs be infinitely more lightweight.
+#
+
+BEGIN {
+#
+# see if we're running out of RAILS_ROOT/script/
+#
+unless defined?(BJ_SCRIPT)
+ BJ_SCRIPT =
+ if %w[ script config app ].map{|d| test ?d, "#{ File.dirname __FILE__ }/../#{ d }"}.all?
+ __FILE__
+ else
+ nil
+ end
+end
+#
+# setup RAILS_ROOT
+#
+unless defined?(RAILS_ROOT)
+ ### grab env var first
+ rails_root = ENV["RAILS_ROOT"]
+
+ ### commandline usage clobbers
+ kv = nil
+ ARGV.delete_if{|arg| arg =~ %r/^RAILS_ROOT=/ and kv = arg}
+ rails_root = kv.split(%r/=/,2).last if kv
+
+ ### we know the rails_root if we are in RAILS_ROOT/script/
+ unless rails_root
+ if BJ_SCRIPT
+ rails_root = File.expand_path "#{ File.dirname __FILE__ }/.."
+ end
+ end
+
+ ### perhaps the current directory is a rails_root?
+ unless rails_root
+ if %w[ script config app ].map{|d| test(?d, d)}.all?
+ rails_root = File.expand_path "."
+ end
+ end
+
+ ### bootstrap
+ RAILS_ROOT = rails_root
+end
+#
+# setup RAILS_ENV
+#
+unless defined?(RAILS_ENV)
+ ### grab env var first
+ rails_env = ENV["RAILS_ENV"]
+
+ ### commandline usage clobbers
+ kv = nil
+ ARGV.delete_if{|arg| arg =~ %r/^RAILS_ENV=/ and kv = arg}
+ rails_env = kv.split(%r/=/,2).last if kv
+
+ ### fallback to development
+ unless rails_env
+ rails_env = "development"
+ end
+
+ ### bootstrap
+ RAILS_ENV = rails_env
+end
+#
+# ensure that rubygems is loaded
+#
+ begin
+ require "rubygems"
+ rescue
+ 42
+ end
+#
+# load gems from plugin dir iff installed as plugin - otherwise load normally
+#
+if RAILS_ROOT and BJ_SCRIPT
+=begin
+ dir = Gem.dir
+ path = Gem.path
+ gem_home = File.join RAILS_ROOT, "vendor", "plugins", "bj", "gem_home"
+ gem_path = [gem_home]
+ Gem.send :use_paths, gem_home, gem_path
+ gem "bj"
+ require "bj"
+ gem "main"
+ require "main"
+=end
+ libdir = File.join(RAILS_ROOT, "vendor", "plugins", "bj", "lib")
+ $LOAD_PATH.unshift libdir
+end
+#
+# hack of #to_s of STDERR/STDOUT for nice help messages
+#
+ class << STDERR
+ def to_s() 'STDERR' end
+ end
+ class << STDOUT
+ def to_s() 'STDOUT' end
+ end
+}
View
3 vendor/plugins/bj/HISTORY
@@ -0,0 +1,3 @@
+
+2007-12-18T14:19:22.86-07:00
+ - corrected install libs
View
142 vendor/plugins/bj/README
@@ -0,0 +1,142 @@
+NAME
+ bj
+
+SYNOPSIS
+ bj (migration_code|generate_migration|migrate|setup|run|submit|list|set|config|pid) [options]+
+
+DESCRIPTION
+ ________________________________
+ Overview
+ --------------------------------
+
+ Backgroundjob (Bj) is a simple to use background priority queue for rails.
+ Although not yet tested on windows, the design of bj is such that operation
+ should be possible on any operating system, including M$.
+
+ Jobs can be submitted to the queue directly using the api or from the
+ commandline using the 'bj' script. For example
+
+ code:
+ Bj.submit 'cat /etc/password'
+
+ cli:
+ bj submit cat /etc/password
+
+ When used from inside a rails application bj arranges that another process
+ will always be running in the background to process the jobs that you submit.
+ By using a separate process to run jobs bj does not impact the resource
+ utilization of your rails application at all and enables several very cool
+ features:
+
+ 1) Bj allows you to sumbit jobs to any of your configured databases and,
+ in each case, spawns a separate background process to run jobs from that
+ queue
+
+ Bj.in :production do
+ Bj.submit 'production_job.exe'
+ end
+
+ Bj.in :development do
+ Bj.submit 'development_job.exe'
+ end
+
+ 2) Although bj ensures that a process is always running to process
+ your jobs, you can start a proces manually. This means that any machine
+ capable of seeing your RAILS_ROOT can run jobs for your application, allowing
+ one to setup a cluster of machines doing the work of a single front end rails
+ applicaiton.
+
+ ________________________________
+ Install
+ --------------------------------
+
+ Bj can be installed two ways: as a gem or as a plugin.
+
+ gem:
+ 1) $sudo gem install bj
+ 2) add "require 'bj'" to config/environment.rb
+ 3) bj setup
+
+ plugin:
+ 1) ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
+ 2) ./script/bj setup
+
+ ________________________________
+ Api
+ --------------------------------
+
+ submit jobs for background processing. 'jobs' can be a string or array of
+ strings. options are applied to each job in the 'jobs', and the list of
+ submitted jobs is always returned. options (string or symbol) can be
+
+ :rails_env => production|development|key_in_database_yml
+ when given this keyword causes bj to submit jobs to the
+ specified database. default is RAILS_ENV.
+
+ :priority => any number, including negative ones. default is zero.
+
+ :tag => a tag added to the job. simply makes searching easier.
+
+ :env => a hash specifying any additional environment vars the background
+ process should have.
+
+ :stdin => any stdin the background process should have.
+
+ eg:
+
+ jobs = Bj.submit 'echo foobar', :tag => 'simple job'
+
+ jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority => 42
+
+ jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
+
+ jobs = Bj.submit './script/runner /dev/stdin',
+ :stdin => 'p RAILS_ENV',
+ :tag => 'dynamic ruby code'
+
+ jobs Bj.submit array_of_commands, :priority => 451
+
+ when jobs are run, they are run in RAILS_ROOT. various attributes are
+ available *only* once the job has finished. you can check whether or not a
+ job is finished by using the #finished method, which simple does a reload and
+ checks to see if the exit_status is non-nil.
+
+ eg:
+
+ jobs = Bj.submit list_of_jobs, :tag => 'important'
+ ...
+
+ jobs.each do |job|
+ if job.finished?
+ p job.exit_status
+ p job.stdout
+ p job.stderr
+ end
+ end
+
+ See lib/bj/api.rb for more details.
+
+ ________________________________
+ Sponsors
+ --------------------------------
+ http://www.engineyard.com/
+ http://quintess.com/
+ http://eparklabs.com/
+
+PARAMETERS
+ --rails_root=rails_root, -R (0 ~> rails_root=)
+ the rails_root will be guessed unless you set this
+ --rails_env=rails_env, -E (0 ~> rails_env=development)
+ set the rails_env
+ --log=log, -l (0 ~> log=STDERR)
+ set the logfile
+ --help, -h
+
+AUTHOR
+ ara.t.howard@gmail.com
+
+URIS
+ http://codeforpeople.com/lib/ruby/
+ http://rubyforge.org/projects/codeforpeople/
+ http://codeforpeople.rubyforge.org/svn/rails/plugins/
+
View
22 vendor/plugins/bj/Rakefile
@@ -0,0 +1,22 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+
+desc 'Default: run unit tests.'
+task :default => :test
+
+desc 'Test the bj plugin.'
+Rake::TestTask.new(:test) do |t|
+ t.libs << 'lib'
+ t.pattern = 'test/**/*_test.rb'
+ t.verbose = true
+end
+
+desc 'Generate documentation for the bj plugin.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+ rdoc.rdoc_dir = 'rdoc'
+ rdoc.title = 'Bj'
+ rdoc.options << '--line-numbers' << '--inline-source'
+ rdoc.rdoc_files.include('README')
+ rdoc.rdoc_files.include('lib/**/*.rb')
+end
View
679 vendor/plugins/bj/bin/bj
@@ -0,0 +1,679 @@
+#! /usr/bin/env ruby
+
+require "bj"
+require "main"
+
+Main {
+ usage["description"] = <<-txt
+ ________________________________
+ Overview
+ --------------------------------
+
+ Backgroundjob (Bj) is a brain dead simple zero admin background priority queue
+ for Rails. Bj is robust, platform independent (including windows), and
+ supports internal or external manangement of the background runner process.
+
+ Jobs can be submitted to the queue directly using the api or from the command
+ line using the ./script/bj:
+
+ api:
+ Bj.submit 'cat /etc/password'
+
+ command line:
+ bj submit cat /etc/password
+
+ Bj's priority queue lives in the database and is therefore durable - your jobs
+ will live across an app crash or machine reboot. The job management is
+ comprehensive capturing stdout, stderr, exit_status, and temporal statistics
+ about each job:
+
+ jobs = Bj.submit array_of_commands, :priority => 42
+
+ ...
+
+ jobs.each do |job|
+ if job.finished?
+ p job.stdout
+ p job.stderr
+ p job.exit_status
+ p job.started_at
+ p job.finished_at
+ end
+ end
+
+ In addition the background runner process logs all commands run and their
+ exit_status to a log named using the following convention:
+
+ rails_root/log/bj.\#{ HOSTNAME }.\#{ RAILS_ENV }.log
+
+ Bj allows you to submit jobs to multiple databases; for instance, if your
+ application is running in development mode you may do:
+
+ Bj.in :production do
+ Bj.submit 'my_job.exe'
+ end
+
+ Bj manages the ever growing list of jobs ran by automatically archiving them
+ into another table (by default jobs > 24 hrs old are archived) to prevent the
+ jobs table from becoming bloated and huge.
+
+ All Bj's tables are namespaced and accessible via the Bj module:
+
+ Bj.table.job.find(:all) # jobs table
+ Bj.table.job_archive.find(:all) # archived jobs
+ Bj.table.config.find(:all) # configuration and runner state
+
+ Bj always arranges for submitted jobs to run with a current working directory
+ of RAILS_ROOT and with the correct RAILS_ENV setting. For example, if you
+ submit a job in production it will have ENV['RAILS_ENV'] == 'production'.
+
+ When Bj manages the background runner it will never outlive the rails
+ application - it is started and stopped on demand as the rails app is started
+ and stopped. This is also true for ./script/console - Bj will automatically
+ fire off the background runner to process jobs submitted using the console.
+
+ Bj ensures that only one background process is running for your application -
+ firing up three mongrels or fcgi processes will result in only one background
+ runner being started. Note that the number of background runners does not
+ determine throughput - that is determined primarily by the nature of the jobs
+ themselves and how much work they perform per process.
+
+
+ ________________________________
+ Architecture
+ --------------------------------
+
+ If one ignores platform specific details the design of Bj is quite simple: the
+ main Rails application submits jobs to table, stored in the database. The act
+ of submitting triggers exactly one of two things to occur:
+
+ 1) a new long running background runner to be started
+
+ 2) an existing background runner to be signaled
+
+ The background runner refuses to run two copies of itself for a given
+ hostname/rails_env combination. For example you may only have one background
+ runner processing jobs on localhost in development mode.
+
+ The background runner, under normal circumstances, is managed by Bj itself -
+ you need do nothing to start, monitor, or stop it - it just works. However,
+ some people will prefer manage their own background process, see 'External
+ Runner' section below for more on this.
+
+ The runner simply processes each job in a highest priority oldest-in fashion,
+ capturing stdout, stderr, exit_status, etc. and storing the information back
+ into the database while logging it's actions. When there are no jobs to run
+ the runner goes to sleep for 42 seconds; however this sleep is interuptable,
+ such as when the runner is signaled that a new job has been submitted so,
+ under normal circumstances there will be zero lag between job submission and
+ job running for an empty queue.
+
+
+ ________________________________
+ External Runner / Clustering
+ --------------------------------
+
+ For the paranoid control freaks out there (myself included) it is quite
+ possible to manage and monitor the runner process manually. This can be
+ desirable in production setups where monitoring software may kill leaking
+ rails apps periodically.
+
+ Recalling that Bj will only allow one copy of itself to process jobs per
+ hostname/rails_env pair we can simply do something like this in cron
+
+ cmd = bj run --forever \\
+ --rails_env=development \\
+ --rails_root=/Users/ahoward/rails_root
+
+ */15 * * * * $cmd
+
+ this will simply attempt the start the background runner every 15 minutes if,
+ and only if, it's not *already* running.
+
+ In addtion to this you'll want to tell Bj not to manage the runner itself
+ using
+
+ Bj.config["production.no_tickle"] = true
+
+ Note that, for clusting setups, it's as simple as adding a crontab and config
+ entry like this for each host. Because Bj throttles background runners per
+ hostname this will allow one runner per hostname - making it quite simple to
+ cluster three nodes behind a besieged rails application.
+
+
+ ________________________________
+ Designing Jobs
+ --------------------------------
+
+ Bj runs it's jobs as command line applications. It ensures that all jobs run
+ in RAILS_ROOT so it's quite natural to apply a pattern such as
+
+ mkdir ./jobs
+ edit ./jobs/background_job_to_run
+
+ ...
+
+ Bj.submit "./jobs/background_job_to_run"
+
+ If you need to run you jobs under an entire rails environment you'll need to
+ do this:
+
+ Bj.submit "./script/runner ./jobs/background_job_to_run"
+
+ Obviously "./script/runner" loads the rails environment for you. It's worth
+ noting that this happens for each job and that this is by design: the reason
+ is that most rails applications leak memory like a sieve so, if one were to
+ spawn a long running process that used the application code base you'd have a
+ lovely doubling of memory usage on you app servers. Although loading the
+ rails environment for each background job requires a little time, a little
+ cpu, and a lot less memory. A future version of Bj will provide a way to load
+ the rails environment once and to process background jobs in this environment,
+ but anyone wanting to use this in production will be required to duct tape
+ their entire chest and have a team of oxen rip off the tape without screaming
+ to prove steelyness of spirit and profound understanding of the other side.
+
+ Don't forget that you can submit jobs with command line arguments:
+
+ Bj.submit "./jobs/a.rb 1 foobar --force"
+
+ and that you can do powerful things by passing stdin to a job that powers
+ through a list of work. For instance, assume a "./jobs/bulkmail" job
+ resembling
+
+ STDIN.each do |line|
+ address = line.strip
+ mail_message_to address
+ end
+
+ then you could
+
+ stdin = [
+ "foo@bar.com",
+ "bar@foo.com",
+ "ara.t.howard@codeforpeople.com",
+ ]
+
+ Bj.submit "./script/runner ./jobs/bulkmail", :stdin => stdin
+
+ and all those emails would be sent in the background.
+
+ Bj's power is putting jobs in the background in a simple and robust fashion.
+ It's your task to build intelligent jobs that leverage batch processing, and
+ other, possibilities. The upshot of building tasks this way is that they are
+ quite easy to test before submitting them from inside your application.
+
+
+ ________________________________
+ Install
+ --------------------------------
+
+ Bj can be installed two ways: as a plugin or via rubygems
+
+ plugin:
+ 1) ./script/plugin install http://codeforpeople.rubyforge.org/svn/rails/plugins/bj
+ 2) ./script/bj setup
+
+ gem:
+ 1) $sudo gem install bj
+ 2) add "require 'bj'" to config/environment.rb
+ 3) bj setup
+
+ ________________________________
+ Api
+ --------------------------------
+
+ submit jobs for background processing. 'jobs' can be a string or array of
+ strings. options are applied to each job in the 'jobs', and the list of
+ submitted jobs is always returned. options (string or symbol) can be
+
+ :rails_env => production|development|key_in_database_yml
+ when given this keyword causes bj to submit jobs to the
+ specified database. default is RAILS_ENV.
+
+ :priority => any number, including negative ones. default is zero.
+
+ :tag => a tag added to the job. simply makes searching easier.
+
+ :env => a hash specifying any additional environment vars the background
+ process should have.
+
+ :stdin => any stdin the background process should have. must respond_to
+ to_s
+
+ eg:
+
+ jobs = Bj.submit 'echo foobar', :tag => 'simple job'
+
+ jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority => 42
+
+ jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
+
+ jobs = Bj.submit './script/runner /dev/stdin',
+ :stdin => 'p RAILS_ENV',
+ :tag => 'dynamic ruby code'
+
+ jobs Bj.submit array_of_commands, :priority => 451
+
+ when jobs are run, they are run in RAILS_ROOT. various attributes are
+ available *only* once the job has finished. you can check whether or not a
+ job is finished by using the #finished method, which simple does a reload and
+ checks to see if the exit_status is non-nil.
+
+ eg:
+
+ jobs = Bj.submit list_of_jobs, :tag => 'important'
+ ...
+
+ jobs.each do |job|
+ if job.finished?
+ p job.exit_status
+ p job.stdout
+ p job.stderr
+ end
+ end
+
+ See lib/bj/api.rb for more details.
+
+ ________________________________
+ Sponsors
+ --------------------------------
+ http://quintess.com/
+ http://www.engineyard.com/
+ http://igicom.com/
+ http://eparklabs.com/
+
+ http://your_company.com/ <<-- (targeted marketing aimed at *you*)
+
+ ________________________________
+ Version
+ --------------------------------
+ #{ Bj.version }
+txt
+
+ usage["uris"] = <<-txt
+ http://codeforpeople.com/lib/ruby/
+ http://rubyforge.org/projects/codeforpeople/
+ http://codeforpeople.rubyforge.org/svn/rails/plugins/
+ txt
+
+ author "ara.t.howard@gmail.com"
+
+ option("rails_root", "R"){
+ description "the rails_root will be guessed unless you set this"
+ argument_required
+ default RAILS_ROOT
+ }
+
+ option("rails_env", "E"){
+ description "set the rails_env"
+ argument_required
+ default RAILS_ENV
+ }
+
+ option("log", "l"){
+ description "set the logfile"
+ argument_required
+ default STDERR
+ }
+
+
+ mode "migration_code" do
+ description "dump migration code on stdout"
+
+ def run
+ puts Bj.table.migration_code
+ end
+ end
+
+ mode "generate_migration" do
+ description "generate a migration"
+
+ def run
+ Bj.generate_migration
+ end
+ end
+
+ mode "migrate" do
+ description "migrate the db"
+
+ def run
+ Bj.migrate
+ end
+ end
+
+ mode "setup" do
+ description "generate a migration and migrate"
+
+ def run
+ set_rails_env(argv.first) if argv.first
+ Bj.setup
+ end
+ end
+
+ mode "plugin" do
+ description "dump the plugin into rails_root"
+
+ def run
+ Bj.plugin
+ end
+ end
+
+ mode "run" do
+ description "start a job runnner, possibly as a daemon"
+
+ option("--forever"){}
+ option("--ppid"){
+ argument :required
+ cast :integer
+ }
+ option("--wait"){
+ argument :required
+ cast :integer
+ }
+ option("--limit"){
+ argument :required
+ cast :integer
+ }
+ option("--redirect"){
+ argument :required
+ }
+ option("--daemon"){}
+
+ def run
+ options = {}
+
+=begin
+ %w[ forever ].each do |key|
+ options[key.to_sym] = true if param[key].given?
+ end
+=end
+
+ %w[ forever ppid wait limit ].each do |key|
+ options[key.to_sym] = param[key].value if param[key].given?
+ end
+
+#p options
+#exit
+ if param["redirect"].given?
+ open(param["redirect"].value, "a+") do |fd|
+ STDERR.reopen fd
+ STDOUT.reopen fd
+ end
+ STDERR.sync = true
+ STDOUT.sync = true
+ end
+
+ trap("SIGTERM"){
+ info{ "SIGTERM" }
+ exit
+ }
+
+ if param["daemon"].given?
+ daemon{ Bj.run options }
+ else
+ Bj.run options
+ end
+ end
+ end
+
+ mode "submit" do
+ keyword("file"){
+ argument :required
+ attr
+ }
+
+ def run
+ joblist = Bj.joblist.for argv.join(' ')
+
+ case file
+ when "-"
+ joblist.push(Bj.joblist.jobs_from_io(STDIN))
+ when "--", "---"
+ joblist.push(Bj.joblist.jobs_from_yaml(STDIN))
+ else
+ open(file){|io| joblist.push(Bj.joblist.jobs_from_io(io)) }
+ end
+
+ jobs = Bj.submit joblist, :no_tickle => true
+
+ oh = lambda{|job| OrderedHash["id", job.id, "command", job.command]}
+
+ y jobs.map{|job| oh[job]}
+ end
+ end
+
+ mode "list" do
+ def run
+ Bj.transaction do
+ y Bj::Table::Job.find(:all).map(&:to_hash)
+ end
+ end
+ end
+
+ mode "set" do
+ argument("key"){ attr }
+
+ argument("value"){ attr }
+
+ option("hostname", "H"){
+ argument :required
+ default Bj.hostname
+ attr
+ }
+
+ option("cast", "c"){
+ argument :required
+ default "to_s"
+ attr
+ }
+
+ def run
+ Bj.transaction do
+ Bj.config.set(key, value, :hostname => hostname, :cast => cast)
+ y Bj.table.config.for(:hostname => hostname)
+ end
+ end
+ end
+
+ mode "config" do
+ option("hostname", "H"){
+ argument :required
+ default Bj.hostname
+ }
+
+ def run
+ Bj.transaction do
+ y Bj.table.config.for(:hostname => param["hostname"].value)
+ end
+ end
+ end
+
+ mode "pid" do
+ option("hostname", "H"){
+ argument :required
+ default Bj.hostname
+ }
+
+ def run
+ Bj.transaction do
+ config = Bj.table.config.for(:hostname => param["hostname"].value)
+ puts config[ "#{ RAILS_ENV }.pid" ] if config
+ end
+ end
+ end
+
+
+ def run
+ help!
+ end
+
+ def before_run
+ self.logger = param["log"].value
+ Bj.logger = logger
+ set_rails_root(param["rails_root"].value) if param["rails_root"].given?
+ set_rails_env(param["rails_env"].value) if param["rails_env"].given?
+ end
+
+ def set_rails_root rails_root
+ ENV["RAILS_ROOT"] = rails_root
+ ::Object.instance_eval do
+ remove_const :RAILS_ROOT
+ const_set :RAILS_ROOT, rails_root
+ end
+ end
+
+ def set_rails_env rails_env
+ ENV["RAILS_ENV"] = rails_env
+ ::Object.instance_eval do
+ remove_const :RAILS_ENV
+ const_set :RAILS_ENV, rails_env
+ end
+ end
+
+ def daemon
+ ra, wa = IO.pipe
+ rb, wb = IO.pipe
+ if fork
+ at_exit{ exit! }
+ wa.close
+ r = ra
+ rb.close
+ w = wb
+ pid = r.gets
+ w.puts pid
+ Integer pid.strip
+ else
+ ra.close
+ w = wa
+ wb.close
+ r = rb
+ open("/dev/null", "r+") do |fd|
+ STDIN.reopen fd
+ STDOUT.reopen fd
+ STDERR.reopen fd
+ end
+ Process::setsid rescue nil
+ pid =
+ fork do
+ Dir::chdir RAILS_ROOT
+ File::umask 0
+ $DAEMON = true
+ yield
+ exit!
+ end
+ w.puts pid
+ r.gets
+ exit!
+ end
+ end
+}
+
+
+
+
+
+#
+# we setup a few things so the script works regardless of whether it was
+# called out of /usr/local/bin, ./script, or wherever. note that the script
+# does *not* require the entire rails application to be loaded into memory!
+# we could just load boot.rb and environment.rb, but this method let's
+# submitting and running jobs be infinitely more lightweight.
+#
+
+BEGIN {
+#
+# see if we're running out of RAILS_ROOT/script/
+#
+unless defined?(BJ_SCRIPT)
+ BJ_SCRIPT =
+ if %w[ script config app ].map{|d| test ?d, "#{ File.dirname __FILE__ }/../#{ d }"}.all?
+ __FILE__
+ else
+ nil
+ end
+end
+#
+# setup RAILS_ROOT
+#
+unless defined?(RAILS_ROOT)
+ ### grab env var first
+ rails_root = ENV["RAILS_ROOT"]
+
+ ### commandline usage clobbers
+ kv = nil
+ ARGV.delete_if{|arg| arg =~ %r/^RAILS_ROOT=/ and kv = arg}
+ rails_root = kv.split(%r/=/,2).last if kv
+
+ ### we know the rails_root if we are in RAILS_ROOT/script/
+ unless rails_root
+ if BJ_SCRIPT
+ rails_root = File.expand_path "#{ File.dirname __FILE__ }/.."
+ end
+ end
+
+ ### perhaps the current directory is a rails_root?
+ unless rails_root
+ if %w[ script config app ].map{|d| test(?d, d)}.all?
+ rails_root = File.expand_path "."
+ end
+ end
+
+ ### bootstrap
+ RAILS_ROOT = rails_root
+end
+#
+# setup RAILS_ENV
+#
+unless defined?(RAILS_ENV)
+ ### grab env var first
+ rails_env = ENV["RAILS_ENV"]
+
+ ### commandline usage clobbers
+ kv = nil
+ ARGV.delete_if{|arg| arg =~ %r/^RAILS_ENV=/ and kv = arg}
+ rails_env = kv.split(%r/=/,2).last if kv
+
+ ### fallback to development
+ unless rails_env
+ rails_env = "development"
+ end
+
+ ### bootstrap
+ RAILS_ENV = rails_env
+end
+#
+# ensure that rubygems is loaded
+#
+ begin
+ require "rubygems"
+ rescue
+ 42
+ end
+#
+# load gems from plugin dir iff installed as plugin - otherwise load normally
+#
+if RAILS_ROOT and BJ_SCRIPT
+=begin
+ dir = Gem.dir
+ path = Gem.path
+ gem_home = File.join RAILS_ROOT, "vendor", "plugins", "bj", "gem_home"
+ gem_path = [gem_home]
+ Gem.send :use_paths, gem_home, gem_path
+ gem "bj"
+ require "bj"
+ gem "main"
+ require "main"
+=end
+ libdir = File.join(RAILS_ROOT, "vendor", "plugins", "bj", "lib")
+ $LOAD_PATH.unshift libdir
+end
+#
+# hack of #to_s of STDERR/STDOUT for nice help messages
+#
+ class << STDERR
+ def to_s() 'STDERR' end
+ end
+ class << STDOUT
+ def to_s() 'STDOUT' end
+ end
+}
View
33 vendor/plugins/bj/init.rb
@@ -0,0 +1,33 @@
+dirname, basename = File.split(File.expand_path(__FILE__))
+libdir = File.join dirname, "lib"
+
+$LOAD_PATH.unshift libdir
+begin
+ require "bj"
+ensure
+ $LOAD_PATH.shift
+end
+
+
+
+=begin
+require "rubygems"
+
+dir = Gem.dir
+path = Gem.path
+
+dirname, basename = File.split(File.expand_path(__FILE__))
+gem_home = File.join dirname, "gem_home"
+gem_path = [gem_home] #, *path]
+
+Gem.send :use_paths, gem_home, gem_path
+
+begin
+ %w[ attributes systemu orderedhash bj ].each do |lib|
+ gem lib
+ require lib
+ end
+ensure
+ Gem.send :use_paths, dir, path
+end
+=end
View
95 vendor/plugins/bj/install.rb
@@ -0,0 +1,95 @@
+#! /usr/bin/env ruby
+
+dirname, basename = File.split File.expand_path(__FILE__)
+
+libidr = 'lib'
+bindir = 'bin'
+gem_home = 'gem_home'
+
+rails_root = File.expand_path File.join(dirname, '../../../')
+bj = File.join rails_root, 'script', 'bj'
+
+gems = %w[ attributes arrayfields main systemu orderedhash bj ]
+
+# in the plugin dir...
+Dir.chdir dirname do
+ puts "in #{ dirname }..."
+
+ # install gems locally
+ puts "installing #{ gems.join ' ' }..."
+ spawn "gem install #{ gems.join ' ' } --install-dir=#{ gem_home } --remote --force --include-dependencies --no-wrappers"
+ puts "."
+
+=begin
+=end
+ # copy libs over to libdir
+ glob = File.join gem_home, "gems/*/lib/*"
+ entries = Dir.glob glob
+ entries.each do |entry|
+ next if entry =~ %r/-\d+\.\d+\.\d+\.rb$/
+ src, dst = entry, libidr
+ puts "#{ src } -->> #{ dst }..."
+ FileUtils.cp_r src, dst
+ puts "."
+ end
+
+ # copy bins over to bindir
+ glob = File.join gem_home, "gems/*/bin/*"
+ entries = Dir.glob glob
+ entries.each do |entry|
+ next if entry =~ %r/-\d+\.\d+\.\d+\.rb$/
+ src, dst = entry, bindir
+ puts "#{ src } -->> #{ dst }..."
+ FileUtils.cp_r src, dst
+ puts "."
+ end
+
+=begin
+ # copy gem_home/bj-x.x.x/bin/bj to rails_root/script/bj
+ glob = File.join gem_home, "gems/bj-*/bin/*"
+ srcs = Dir.glob glob
+ srcs.each do |src|
+ basename = File.basename src
+ dst = File.join rails_root, 'script', basename
+ puts "#{ src } -->> #{ dst }..."
+ FileUtils.cp_r src, dst
+ File.chmod 0755, dst
+ puts "."
+ end
+=end
+
+ # install bin/bj to script/bj
+ src, dst = File.join(bindir, "bj"), File.join(rails_root, "script", "bj")
+ puts "#{ src } -->> #{ dst }..."
+ FileUtils.cp src, dst
+ File.chmod 0755, dst
+ puts "."
+
+ # kill all the local gems
+ FileUtils.rm_rf gem_home
+
+ # dump help
+ puts("=" * 79)
+ ruby = which_ruby
+ system "#{ ruby } #{ bj.inspect } '--help'"
+end
+
+
+
+BEGIN {
+ require 'fileutils'
+ require 'rbconfig'
+
+ def spawn command
+ oe = `#{ command } 2>&1`
+ raise "command <#{ command }> failed with <#{ $?.inspect }>" unless $?.exitstatus == 0
+ oe
+ end
+
+ def which_ruby
+ c = ::Config::CONFIG
+ ruby = File::join(c['bindir'], c['ruby_install_name']) << c['EXEEXT']
+ raise "ruby @ #{ ruby } not executable!?" unless test(?e, ruby)
+ ruby
+ end
+}
View
443 vendor/plugins/bj/lib/arrayfields.rb
@@ -0,0 +1,443 @@
+#
+# The ArrayFields module implements methods which allow an Array to be indexed
+# by String or Symbol. It is not required to manually use this module to
+# extend Arrays - they are auto-extended on a per-object basis when
+# Array#fields= is called
+#
+ module ArrayFields
+ self::VERSION = '4.7.4' unless defined? self::VERSION
+ def self.version() Arrayfields::VERSION end
+ #
+ # multiton cache of fields - wraps fields and fieldpos map to save memory
+ #
+ class FieldSet
+ class << self
+ def new fields
+ @sets[fields] ||= super
+ end
+ def init_sets
+ @sets = {}
+ end
+ end
+
+ init_sets
+
+ attr :fields
+ attr :fieldpos
+ def initialize fields
+ raise ArgumentError, "<#{ fields.inspect }> not inject-able" unless
+ fields.respond_to? :inject
+
+ @fieldpos =
+ fields.inject({}) do |h, f|
+ unless String === f or Symbol === f
+ raise ArgumentError, "<#{ f.inspect }> neither String nor Symbol"
+ end
+ h[f] = h.size
+ h
+ end
+
+ @fields = fields
+ end
+ def pos f
+ return @fieldpos[f] if @fieldpos.has_key? f
+ f = f.to_s
+ return @fieldpos[f] if @fieldpos.has_key? f
+ f = f.intern
+ return @fieldpos[f] if @fieldpos.has_key? f
+ nil
+ end
+ end
+ #
+ # methods redefined to work with fields as well as numeric indexes
+ #
+ def [] idx, *args
+ if @fieldset and (String === idx or Symbol === idx)
+ pos = @fieldset.pos idx
+ return nil unless pos
+ super(pos, *args)
+ else
+ super
+ end
+ end
+ def slice idx, *args
+ if @fieldset and (String === idx or Symbol === idx)
+ pos = @fieldset.pos idx
+ return nil unless pos
+ super(pos, *args)
+ else
+ super
+ end
+ end
+
+ def []=(idx, *args)
+ if @fieldset and (String === idx or Symbol === idx)
+ pos = @fieldset.pos idx
+ unless pos
+ @fieldset.fields << idx
+ @fieldset.fieldpos[idx] = pos = size
+ end
+ super(pos, *args)
+ else
+ super
+ end
+ end
+ def at idx
+ if @fieldset and (String === idx or Symbol === idx)
+ pos = @fieldset.pos idx
+ return nil unless pos
+ super pos
+ else
+ super
+ end
+ end
+ def delete_at idx
+ if @fieldset and (String === idx or Symbol === idx)
+ pos = @fieldset.pos idx
+ return nil unless pos
+ new_fields = fields.dup
+ new_fields.delete_at(pos)
+ self.fields = new_fields
+ super pos
+ else
+ super
+ end
+ end
+ def fill(obj, *args)
+ idx = args.first
+ if idx and @fieldset and (String === idx or Symbol === idx)
+ idx = args.shift
+ pos = @fieldset.pos idx
+ super(obj, pos, *args)
+ else
+ super
+ end
+ end
+
+ def values_at(*idxs)
+ idxs.flatten!
+ if @fieldset
+ idxs.map!{|i| (String === i or Symbol === i) ? @fieldset.pos(i) : i}
+ end
+ super(*idxs)
+ end
+ def indices(*idxs)
+ idxs.flatten!
+ if @fieldset
+ idxs.map!{|i| (String === i or Symbol === i) ? @fieldset.pos(i) : i}
+ end
+ super(*idxs)
+ end
+ def indexes(*idxs)
+ idxs.flatten!
+ if @fieldset
+ idxs.map!{|i| (String === i or Symbol === i) ? @fieldset.pos(i) : i}
+ end
+ super(*idxs)
+ end
+
+ def slice!(*args)
+ ret = self[*args]
+ self[*args] = nil
+ ret
+ end
+ def each_with_field
+ each_with_index do |elem, i|
+ yield elem, @fieldset.fields[i]
+ end
+ end
+ #
+ # methods which give a hash-like interface
+ #
+ def each_pair
+ each_with_index do |elem, i|
+ yield @fieldset.fields[i], elem
+ end
+ end
+ def each_key
+ @fieldset.each{|field| yield field}
+ end
+ def each_value *args, &block
+ each(*args, &block)
+ end
+ def fetch key
+ self[key] or raise IndexError, 'key not found'
+ end
+
+ def has_key? key
+ @fieldset.fields.include? key
+ end
+ def member? key
+ @fieldset.fields.include? key
+ end
+ def key? key
+ @fieldset.fields.include? key
+ end
+
+ def has_value? value
+ if respond_to? 'include?'
+ self.include? value
+ else
+ a = []
+ each{|val| a << val}
+ a.include? value
+ end
+ end
+ def value? value
+ if respond_to? 'include?'
+ self.include? value
+ else
+ a = []
+ each{|val| a << val}
+ a.include? value
+ end
+ end
+
+ def keys
+ fields
+ end
+ def store key, value
+ self[key] = value
+ end
+ def values
+ if respond_to? 'to_ary'
+ self.to_ary
+ else
+ a = []
+ each{|val| a << val}
+ a
+ end
+ end
+
+ def to_hash
+ if respond_to? 'to_ary'
+ h = {}
+ @fieldset.fields.zip(to_ary){|f,e| h[f] = e}
+ h
+ else
+ a = []
+ each{|val| a << val}
+ h = {}
+ @fieldset.fields.zip(a){|f,e| h[f] = e}
+ h
+ end
+ end
+ def to_h
+ if respond_to? 'to_ary'
+ h = {}
+ @fieldset.fields.zip(to_ary){|f,e| h[f] = e}
+ h
+ else
+ a = []
+ each{|val| a << val}
+ h = {}
+ @fieldset.fields.zip(a){|f,e| h[f] = e}
+ h
+ end
+ end
+
+ def update other
+ other.each{|k,v| self[k] = v}
+ to_hash
+ end
+ def replace other
+ Hash === other ? update(other) : super
+ end
+ def invert
+ to_hash.invert
+ end
+
+ def to_pairs
+ fields.zip values
+ end
+ alias_method 'pairs', 'to_pairs'
+
+ def clone
+ clone = super
+ ensure
+ clone.fields = fields.clone
+ end
+
+ def dup
+ dup = super
+ ensure
+ dup.fields = fields.dup
+ end
+
+ def deepcopy
+ cp = Marshal.load(Marshal.dump(self))
+ cp.fields = Marshal.load(Marshal.dump(self.fields))
+ cp
+ end
+ end
+ Arrayfields = ArrayFields
+
+ module Arrayfields
+ def self.new *pairs
+ pairs = pairs.map{|pair| Enumerable === pair ? pair.to_a : pair}.flatten
+ raise ArgumentError, "pairs must be evenly sized" unless(pairs.size % 2 == 0)
+ (( array = [] )).fields = []
+ 0.step(pairs.size - 2, 2) do |a|
+ b = a + 1
+ array[ pairs[a] ] = pairs[b]
+ end
+ array
+ end
+ def self.[] *pairs
+ new(*pairs)
+ end
+ end
+ def Arrayfields(*a, &b) Arrayfields.new(*a, &b) end
+#
+# Fieldable encapsulates methods in common for classes which may have their
+# fields set and subsequently be auto-extended by ArrayFields
+#
+ module Fieldable
+ #
+ # sets fields an dynamically extends this Array instance with methods for
+ # keyword access
+ #
+ def fields= fields
+ extend ArrayFields unless ArrayFields === self
+
+ @fieldset =
+ if ArrayFields::FieldSet === fields
+ fields
+ else
+ ArrayFields::FieldSet.new fields
+ end
+ end
+ #
+ # access to fieldset
+ #
+ def fieldset
+ @fieldset ||= nil
+ end
+ #
+ # access to field list
+ #
+ def fields *values
+ return(send('fields=', *values)) unless values.empty?
+ fieldset and fieldset.fields
+ end
+ end
+#
+# Array instances are extened with two methods only: Fieldable#fields= and
+# Fieldable#fields. only when Fieldable#fields= is called will the full set
+# of ArrayFields methods auto-extend the Array instance. the Array class also
+# has added a class generator when the fields are known apriori.
+#
+ class Array
+ include Fieldable
+
+ class << self
+ def struct *fields
+ fields = fields.flatten
+ Class.new(self) do
+ include ArrayFields
+ const_set :FIELDS, ArrayFields::FieldSet.new(fields)
+ fields.each do |field|
+ field = field.to_s
+ if field =~ %r/^[a-zA-Z_][a-zA-Z0-9_]*$/
+ begin
+ module_eval <<-code
+ def #{ field } *a
+ a.size == 0 ? self['#{ field }'] : (self.#{ field } = a.shift)
+ end
+ def #{ field }= value
+ self['#{ field }'] = value
+ end
+ code
+ rescue SyntaxError
+ :by_ignoring_it
+ end
+ end
+ end
+ def initialize *a, &b
+ super
+ ensure
+ @fieldset = self.class.const_get :FIELDS
+ end
+ def self.[] *elements
+ array = new
+ array.replace elements
+ array
+ end
+ end
+ end
+ def fields *fields, &block
+ (( array = new(&block) )).fields = fields.map{|x| Enumerable === x ? x.to_a : x}.flatten
+ array
+ end
+ end
+ end
+#
+# proxy class that allows an array to be wrapped in a way that still allows #
+# keyword access. also facilitate usage of ArrayFields with arraylike objects.
+# thnx to Sean O'Dell for the suggestion.
+#
+# sample usage
+#
+# fa = FieldedArray.new %w(zero one two), [0,1,2]
+# p fa['zero'] #=> 0
+#
+#
+ class FieldedArray
+ include Fieldable
+ class << self
+ def [](*pairs)
+ pairs.flatten!
+ raise ArgumentError, "argument must be key/val pairs" unless
+ (pairs.size % 2 == 0)
+ fields, elements = [], []
+ while((f = pairs.shift) and (e = pairs.shift))
+ fields << f and elements << e
+ end
+ new fields, elements
+ end
+ end
+ def initialize fields = [], array = []
+ @a = array
+ self.fields = fields
+ end
+ def method_missing(meth, *args, &block)
+ @a.send(meth, *args, &block)
+ end
+ delegates =
+ %w(
+ to_s
+ to_str
+ inspect
+ )
+ delegates.each do |meth|
+ class_eval "def #{ meth }(*a,&b); @a.#{ meth }(*a,&b);end"
+ end
+ end
+ Fieldedarray = FieldedArray
+
+ class PseudoHash < ::Array
+ class << self
+ def [](*pairs)
+ pairs.flatten!
+ raise ArgumentError, "argument must be key/val pairs" unless
+ (pairs.size % 2 == 0 and pairs.size >= 2)
+ keys, values = [], []
+ while((k = pairs.shift) and (v = pairs.shift))
+ keys << k and values << v
+ end
+ new keys, values
+ end
+ end
+ def initialize keys = [], values = []
+ self.fields = keys
+ self.replace values
+ end
+ def to_yaml opts = {}
+ YAML::quick_emit object_id, opts do |out|
+ out.map taguri, to_yaml_style do |map|
+ each_pair{|f,v| map.add f,v}
+ end
+ end
+ end
+ end
+ Pseudohash = PseudoHash
View
118 vendor/plugins/bj/lib/attributes.rb
@@ -0,0 +1,118 @@
+module Attributes
+ Attributes::VERSION = '5.0.0' unless defined? Attributes::VERSION
+ def self.version() Attributes::VERSION end
+
+ class List < ::Array
+ def << element
+ super
+ self
+ ensure
+ uniq!
+ index!
+ end
+ def index!
+ @index ||= Hash.new
+ each{|element| @index[element] = true}
+ end
+ def include? element
+ @index ||= Hash.new
+ @index[element] ? true : false
+ end
+ def initializers
+ @initializers ||= Hash.new
+ end
+ end
+
+ def attributes *a, &b
+ unless a.empty?
+ returned = Hash.new
+
+ hashes, names = a.partition{|x| Hash === x}
+ names_and_defaults = {}
+ hashes.each{|h| names_and_defaults.update h}
+ names.flatten.compact.each{|name| names_and_defaults.update name => nil}
+
+ initializers = __attributes__.initializers
+
+ names_and_defaults.each do |name, default|
+ raise NameError, "bad instance variable name '@#{ name }'" if "@#{ name }" =~ %r/[!?=]$/o
+ name = name.to_s
+
+ initialize = b || lambda { default }
+ initializer = lambda do |this|
+ Object.instance_method('instance_eval').bind(this).call &initialize
+ end
+ initializer_id = initializer.object_id
+ __attributes__.initializers[name] = initializer
+
+ module_eval <<-code
+ def #{ name }=(*value, &block)
+ value.unshift block if block
+ @#{ name } = value.first
+ end
+ code
+
+ module_eval <<-code
+ def #{ name }(*value, &block)
+ value.unshift block if block
+ return self.send('#{ name }=', value.first) unless value.empty?
+ #{ name }! unless defined? @#{ name }
+ @#{ name }
+ end
+ code
+
+ module_eval <<-code
+ def #{ name }!
+ initializer = ObjectSpace._id2ref #{ initializer_id }
+ self.#{ name } = initializer.call(self)
+ @#{ name }
+ end
+ code
+
+ module_eval <<-code
+ def #{ name }?
+ #{ name }
+ end
+ code
+
+ attributes << name
+ returned[name] = initializer
+ end
+
+ returned
+ else
+ begin
+ __attribute_list__
+ rescue NameError
+ singleton_class =
+ class << self
+ self
+ end
+ klass = self
+ singleton_class.module_eval do
+ attribute_list = List.new
+ define_method('attribute_list'){ klass == self ? attribute_list : raise(NameError) }
+ alias_method '__attribute_list__', 'attribute_list'
+ end
+ __attribute_list__
+ end
+ end
+ end
+
+ %w( __attributes__ __attribute__ attribute ).each{|dst| alias_method dst, 'attributes'}
+end
+
+class Object
+ def attributes *a, &b
+ sc =
+ class << self
+ self
+ end
+ sc.attributes *a, &b
+ end
+ %w( __attributes__ __attribute__ attribute ).each{|dst| alias_method dst, 'attributes'}
+end
+
+class Module
+ include Attributes
+end
View
87 vendor/plugins/bj/lib/bj.rb
@@ -0,0 +1,87 @@
+unless defined? Bj
+ class Bj
+ #
+ # constants and associated attrs
+ #
+ Bj::VERSION = "1.0.1" #unless defined? Bj::VERSION
+ def self.version() Bj::VERSION end
+
+ Bj::LIBDIR = File.expand_path(File::join(File.dirname(__FILE__), "bj")) + File::SEPARATOR unless
+ defined? Bj::LIBDIR
+ def self.libdir(*value)
+ unless value.empty?
+ File.join libdir, *value
+ else
+ Bj::LIBDIR
+ end
+ end
+
+ module EXIT
+ SUCCESS = 0
+ FAILURE = 1
+ WARNING = 42
+ end
+ #
+ # built-in
+ #
+ require "socket"
+ require "yaml"
+ require "thread"
+ require "rbconfig"
+ require "set"
+ require "erb"
+ require "tempfile"
+ #
+ # bootstrap rubygems
+ #
+ begin
+ require "rubygems"
+ rescue LoadError
+ 42
+ end
+ #
+ # rubyforge/remote
+ #
+ require "active_record"
+ #
+ # rubyforge/remote or local/lib
+ #
+ #%w[ attributes systemu orderedhash ].each do |lib|
+ %w[ systemu orderedhash ].each do |lib|
+ begin
+ require lib
+ rescue
+ require libdir(lib)
+ end
+ end
+ #
+ # local
+ #
+ load libdir("attributes.rb")
+ load libdir("stdext.rb")
+ load libdir("util.rb")
+ load libdir("errors.rb")
+ load libdir("logger.rb")
+ load libdir("bj.rb")
+ load libdir("joblist.rb")
+ load libdir("table.rb")
+ load libdir("runner.rb")
+ load libdir("api.rb")
+ #
+ # an imperfect reloading hook - because neither rails' plugins nor gems provide one, sigh...
+ #
+ def self.reload!
+ background = nil
+ ::Object.module_eval do
+ background = Bj.runner.background
+ remove_const :Bj rescue nil
+ remove_const :BackgroundJob rescue nil
+ end
+ returned = load __FILE__ rescue nil
+ Bj.runner.background = background if background
+ returned
+ end
+ end
+
+ BackgroundJob = Bj
+end
View
161 vendor/plugins/bj/lib/bj/api.rb
@@ -0,0 +1,161 @@
+class Bj
+#
+# the api exposes nearly all the bj code you'll likely need, with the
+# exception of accessing the job table for searching, which is done using
+#
+# eg.
+#
+# Bj.table.job.find :all
+#
+ module API
+ #
+ # submit jobs for background processing. 'jobs' can be a string or array of
+ # strings. options are applied to each job in the 'jobs', and the list of
+ # submitted jobs is always returned. options (string or symbol) can be
+ #
+ # :rails_env => production|development|key_in_database_yml
+ # when given this keyword causes bj to submit jobs to the
+ # specified database. default is RAILS_ENV.
+ #
+ # :priority => any number, including negative ones. default is zero.
+ #
+ # :tag => a tag added to the job. simply makes searching easier.
+ #
+ # :env => a hash specifying any additional environment vars the background
+ # process should have.
+ #
+ # :stdin => any stdin the background process should have.
+ #
+ # eg:
+ #
+ # jobs = Bj.submit 'echo foobar', :tag => 'simple job'
+ #
+ # jobs = Bj.submit '/bin/cat', :stdin => 'in the hat', :priority => 42
+ #
+ # jobs = Bj.submit './script/runner ./scripts/a.rb', :rails_env => 'production'
+ #
+ # jobs = Bj.submit './script/runner /dev/stdin', :stdin => 'p RAILS_ENV', :tag => 'dynamic ruby code'
+ #
+ # jobs = Bj.submit array_of_commands, :priority => 451
+ #
+ # when jobs are run, they are run in RAILS_ROOT. various attributes are
+ # available *only* once the job has finished. you can check whether or not
+ # a job is finished by using the #finished method, which simple does a
+ # reload and checks to see if the exit_status is non-nil.
+ #
+ # eg:
+ #
+ # jobs = Bj.submit list_of_jobs, :tag => 'important'
+ # ...
+ #
+ # jobs.each do |job|
+ # if job.finished?
+ # p job.exit_status
+ # p job.stdout
+ # p job.stderr
+ # end
+ # end
+ #
+ #
+ def submit jobs, options = {}, &block
+ options.to_options!
+ Bj.transaction(options) do
+ table.job.submit jobs, options, &block
+ end
+ ensure
+ Bj.runner.tickle unless options[:no_tickle]
+ end
+ #
+ # this method changes the context under which bj is operating. a context is
+ # a RAILS_ENV. the method accepts a block and it used to alter the
+ # behaviour of the bj lib on a global scale such that all operations,
+ # spawning of background runnner processes, etc, occur in that context.
+ #
+ # eg:
+ #
+ # Bj.in :production do
+ # Bj.submit './script/runner ./scripts/facebook_notification.rb'
+ # end
+ #
+ # Bj.in :development do
+ # Bj.submit 'does_this_eat_memory.exe'
+ # end
+ #
+ def in rails_env = Bj.rails_env, &block
+ transaction(:rails_env => rails_env.to_s, &block)
+ end
+ #
+ # list simply returns a list of all jobs in the job table
+ #
+ def list options = {}, &block
+ options.to_options!
+ Bj.transaction(options) do
+ options.delete :rails_env
+ table.job.find(:all, options)
+ end
+ end
+ #
+ #
+ #
+ def run options = {}
+ runner.run options
+ end
+ #
+ # generate a migration and migrate a database (production/development/etc)
+ #
+ def setup options = {}
+ options.to_options!
+ chroot do
+ generate_migration options
+ migrate options
+ end
+ end
+ #
+ # generate_migration, suprisingly, generates the single migration needed for
+ # bj. you'll notice the the migration is very short as the migration
+ # classes themselves are inner classes of the respective bj table class.
+ # see lib/bj/table.rb for details.
+ #
+ def generate_migration options = {}