72 changes: 43 additions & 29 deletions README.textile
View file
Expand Up @@ -17,8 +17,7 @@ h2. Installation
To install as a gem, add the following to @config/environment.rb@:

<pre>
config.gem 'collectiveidea-delayed_job', :lib => 'delayed_job',
:source => 'http://gems.github.com'
config.gem 'delayed_job'
</pre>

Rake tasks are not automatically loaded from gems, so you'll need to add the following to your Rakefile:
Expand All @@ -37,19 +36,34 @@ To install as a plugin:
script/plugin install git://github.com/collectiveidea/delayed_job.git
</pre>

After delayed_job is installed, run:
After delayed_job is installed, you will need to setup the backend.

h2. Backends

delayed_job supports multiple backends for storing the job queue. There are currently implementations for Active Record, MongoMapper, and DataMapper.

h3. Active Record

The default is Active Record, which requires a jobs table.

<pre>
script/generate delayed_job
rake db:migrate
$ script/generate delayed_job
$ rake db:migrate
</pre>

h2. Upgrading to 1.8
h3. MongoMapper

<pre>
# config/initializers/delayed_job.rb
Delayed::Worker.backend = :mongo_mapper
</pre>

If you are upgrading from a previous release, you will need to generate the new @script/delayed_job@:
h3. DataMapper

<pre>
script/generate delayed_job --skip-migration
# config/initializers/delayed_job.rb
Delayed::Worker.backend = :data_mapper
Delayed::Worker.backend.auto_upgrade!
</pre>

h2. Queuing Jobs
Expand Down Expand Up @@ -80,7 +94,7 @@ device.deliver

h2. Running Jobs

@script/delayed_job@ can be used to manage a background process which will start working off jobs.
@script/delayed_job@ can be used to manage a background process which will start working off jobs. Make sure you've run `script/generate delayed_job`.

<pre>
$ RAILS_ENV=production script/delayed_job start
Expand Down Expand Up @@ -113,39 +127,39 @@ h2. Gory Details

The library evolves around a delayed_jobs table which looks as follows:

create_table :delayed_jobs, :force => true do |table|
table.integer :priority, :default => 0 # Allows some jobs to jump to the front of the queue
table.integer :attempts, :default => 0 # Provides for retries, but still fail eventually.
table.text :handler # YAML-encoded string of the object that will do work
table.text :last_error # reason for last failure (See Note below)
table.datetime :run_at # When to run. Could be Time.zone.now for immediately, or sometime in the future.
table.datetime :locked_at # Set when a client is working on this object
table.datetime :failed_at # Set when all retries have failed (actually, by default, the record is deleted instead)
table.string :locked_by # Who is working on this object (if locked)
table.timestamps
end
<pre>
create_table :delayed_jobs, :force => true do |table|
table.integer :priority, :default => 0 # Allows some jobs to jump to the front of the queue
table.integer :attempts, :default => 0 # Provides for retries, but still fail eventually.
table.text :handler # YAML-encoded string of the object that will do work
table.text :last_error # reason for last failure (See Note below)
table.datetime :run_at # When to run. Could be Time.zone.now for immediately, or sometime in the future.
table.datetime :locked_at # Set when a client is working on this object
table.datetime :failed_at # Set when all retries have failed (actually, by default, the record is deleted instead)
table.string :locked_by # Who is working on this object (if locked)
table.timestamps
end
</pre>

On failure, the job is scheduled again in 5 seconds + N ** 4, where N is the number of retries.

The default Job::max_attempts is 25. After this, the job either deleted (default), or left in the database with "failed_at" set.
The default Worker.max_attempts is 25. After this, the job either deleted (default), or left in the database with "failed_at" set.
With the default of 25 attempts, the last retry will be 20 days later, with the last interval being almost 100 hours.

The default Job::max_run_time is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to
The default Worker.max_run_time is 4.hours. If your job takes longer than that, another computer could pick it up. It's up to you to
make sure your job doesn't exceed this time. You should set this to the longest time you think the job could take.

By default, it will delete failed jobs (and it always deletes successful jobs). If you want to keep failed jobs, set
Delayed::Job.destroy_failed_jobs = false. The failed jobs will be marked with non-null failed_at.
Delayed::Worker.destroy_failed_jobs = false. The failed jobs will be marked with non-null failed_at.

Here is an example of changing job parameters in Rails:

<pre>
# config/initializers/delayed_job_config.rb
Delayed::Job.destroy_failed_jobs = false
silence_warnings do
Delayed::Worker::sleep_delay = 60
Delayed::Job::max_attempts = 3
Delayed::Job::max_run_time = 5.minutes
end
Delayed::Worker.destroy_failed_jobs = false
Delayed::Worker.sleep_delay = 60
Delayed::Worker.max_attempts = 3
Delayed::Worker.max_run_time = 5.minutes
</pre>

h3. Cleaning up
Expand Down
19 changes: 15 additions & 4 deletions Rakefile
View file
Expand Up @@ -2,7 +2,7 @@
begin
require 'jeweler'
rescue LoadError
puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
puts "Jeweler not available. Install it with: sudo gem install jeweler"
exit 1
end

Expand All @@ -11,14 +11,24 @@ Jeweler::Tasks.new do |s|
s.summary = "Database-backed asynchronous priority queue system -- Extracted from Shopify"
s.email = "tobi@leetsoft.com"
s.homepage = "http://github.com/collectiveidea/delayed_job"
s.description = "Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks."
s.description = "Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.\n\nThis gem is collectiveidea's fork (http://github.com/collectiveidea/delayed_job)."
s.authors = ["Brandon Keepers", "Tobias Lütke"]

s.has_rdoc = true
s.rdoc_options = ["--main", "README.textile", "--inline-source", "--line-numbers"]
s.extra_rdoc_files = ["README.textile"]

s.test_files = Dir['spec/**/*']
s.test_files = Dir['spec/*_spec.rb']

s.add_dependency "daemons"
s.add_development_dependency "rspec"
s.add_development_dependency "sqlite3-ruby"
s.add_development_dependency "mongo_mapper"
s.add_development_dependency "dm-core"
s.add_development_dependency "dm-observer"
s.add_development_dependency "dm-aggregates"
s.add_development_dependency "do_sqlite3"
s.add_development_dependency "database_cleaner"
end

require 'spec/rake/spectask'
Expand All @@ -28,7 +38,8 @@ task :default => :spec
desc 'Run the specs'
Spec::Rake::SpecTask.new(:spec) do |t|
t.libs << 'lib'
t.pattern = 'spec/**/*_spec.rb'
t.pattern = 'spec/*_spec.rb'
t.verbose = true
end
task :spec => :check_dependencies

2 changes: 1 addition & 1 deletion VERSION
View file
@@ -1 +1 @@
1.8.4
2.0.0
19 changes: 19 additions & 0 deletions benchmarks.rb
View file
@@ -0,0 +1,19 @@
$:.unshift(File.dirname(__FILE__) + '/lib')
require 'rubygems'
require 'logger'
require 'delayed_job'
require 'benchmark'

Delayed::Worker.logger = Logger.new('/dev/null')

Benchmark.bm(10) do |x|
[:active_record, :mongo_mapper, :data_mapper].each do |backend|
require "spec/setup/#{backend}"
Delayed::Worker.backend = backend

n = 10000
n.times { "foo".send_later :length }

x.report(backend.to_s) { Delayed::Worker.new(:quiet => true).work_off(n) }
end
end
4 changes: 2 additions & 2 deletions contrib/delayed_job.monitrc
View file
Expand Up @@ -10,5 +10,5 @@

check process delayed_job
with pidfile /var/www/apps/{app_name}/shared/pids/delayed_job.pid
start program = "RAILS_ENV=production /var/www/apps/{app_name}/current/script/delayed_job start"
stop program = "RAILS_ENV=production /var/www/apps/{app_name}/current/script/delayed_job stop"
start program = "/usr/bin/env RAILS_ENV=production /var/www/apps/{app_name}/current/script/delayed_job start"
stop program = "/usr/bin/env RAILS_ENV=production /var/www/apps/{app_name}/current/script/delayed_job stop"
71 changes: 57 additions & 14 deletions delayed_job.gemspec
View file
@@ -1,16 +1,18 @@
# Generated by jeweler
# DO NOT EDIT THIS FILE
# Instead, edit Jeweler::Tasks in Rakefile, and run `rake gemspec`
# DO NOT EDIT THIS FILE DIRECTLY
# Instead, edit Jeweler::Tasks in Rakefile, and run the gemspec command
# -*- encoding: utf-8 -*-

Gem::Specification.new do |s|
s.name = %q{delayed_job}
s.version = "1.8.4"
s.version = "2.0.0"

s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
s.authors = ["Brandon Keepers", "Tobias L\303\274tke"]
s.date = %q{2009-10-06}
s.description = %q{Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.}
s.date = %q{2010-04-03}
s.description = %q{Delayed_job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background. It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks.
This gem is collectiveidea's fork (http://github.com/collectiveidea/delayed_job).}
s.email = %q{tobi@leetsoft.com}
s.extra_rdoc_files = [
"README.textile"
Expand All @@ -21,46 +23,87 @@ Gem::Specification.new do |s|
"README.textile",
"Rakefile",
"VERSION",
"benchmarks.rb",
"contrib/delayed_job.monitrc",
"delayed_job.gemspec",
"generators/delayed_job/delayed_job_generator.rb",
"generators/delayed_job/templates/migration.rb",
"generators/delayed_job/templates/script",
"init.rb",
"lib/delayed/backend/active_record.rb",
"lib/delayed/backend/base.rb",
"lib/delayed/backend/data_mapper.rb",
"lib/delayed/backend/mongo_mapper.rb",
"lib/delayed/command.rb",
"lib/delayed/job.rb",
"lib/delayed/message_sending.rb",
"lib/delayed/performable_method.rb",
"lib/delayed/recipes.rb",
"lib/delayed/tasks.rb",
"lib/delayed/worker.rb",
"lib/delayed_job.rb",
"rails/init.rb",
"recipes/delayed_job.rb",
"spec/database.rb",
"spec/backend/active_record_job_spec.rb",
"spec/backend/data_mapper_job_spec.rb",
"spec/backend/mongo_mapper_job_spec.rb",
"spec/backend/shared_backend_spec.rb",
"spec/delayed_method_spec.rb",
"spec/job_spec.rb",
"spec/performable_method_spec.rb",
"spec/sample_jobs.rb",
"spec/setup/active_record.rb",
"spec/setup/data_mapper.rb",
"spec/setup/mongo_mapper.rb",
"spec/spec_helper.rb",
"spec/story_spec.rb",
"spec/worker_spec.rb",
"tasks/jobs.rake"
]
s.homepage = %q{http://github.com/collectiveidea/delayed_job}
s.rdoc_options = ["--main", "README.textile", "--inline-source", "--line-numbers"]
s.require_paths = ["lib"]
s.rubygems_version = %q{1.3.5}
s.rubygems_version = %q{1.3.6}
s.summary = %q{Database-backed asynchronous priority queue system -- Extracted from Shopify}
s.test_files = [
"spec/database.rb",
"spec/delayed_method_spec.rb",
"spec/job_spec.rb",
"spec/story_spec.rb"
"spec/delayed_method_spec.rb",
"spec/performable_method_spec.rb",
"spec/story_spec.rb",
"spec/worker_spec.rb"
]

if s.respond_to? :specification_version then
current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
s.specification_version = 3

if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
s.add_runtime_dependency(%q<daemons>, [">= 0"])
s.add_development_dependency(%q<rspec>, [">= 0"])
s.add_development_dependency(%q<sqlite3-ruby>, [">= 0"])
s.add_development_dependency(%q<mongo_mapper>, [">= 0"])
s.add_development_dependency(%q<dm-core>, [">= 0"])
s.add_development_dependency(%q<dm-observer>, [">= 0"])
s.add_development_dependency(%q<dm-aggregates>, [">= 0"])
s.add_development_dependency(%q<do_sqlite3>, [">= 0"])
s.add_development_dependency(%q<database_cleaner>, [">= 0"])
else
s.add_dependency(%q<daemons>, [">= 0"])
s.add_dependency(%q<rspec>, [">= 0"])
s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
s.add_dependency(%q<mongo_mapper>, [">= 0"])
s.add_dependency(%q<dm-core>, [">= 0"])
s.add_dependency(%q<dm-observer>, [">= 0"])
s.add_dependency(%q<dm-aggregates>, [">= 0"])
s.add_dependency(%q<do_sqlite3>, [">= 0"])
s.add_dependency(%q<database_cleaner>, [">= 0"])
end
else
s.add_dependency(%q<daemons>, [">= 0"])
s.add_dependency(%q<rspec>, [">= 0"])
s.add_dependency(%q<sqlite3-ruby>, [">= 0"])
s.add_dependency(%q<mongo_mapper>, [">= 0"])
s.add_dependency(%q<dm-core>, [">= 0"])
s.add_dependency(%q<dm-observer>, [">= 0"])
s.add_dependency(%q<dm-aggregates>, [">= 0"])
s.add_dependency(%q<do_sqlite3>, [">= 0"])
s.add_dependency(%q<database_cleaner>, [">= 0"])
end
end

2 changes: 1 addition & 1 deletion generators/delayed_job/delayed_job_generator.rb
View file
Expand Up @@ -4,7 +4,7 @@ class DelayedJobGenerator < Rails::Generator::Base
def manifest
record do |m|
m.template 'script', 'script/delayed_job', :chmod => 0755
unless options[:skip_migration]
if !options[:skip_migration] && defined?(ActiveRecord)
m.migration_template "migration.rb", 'db/migrate',
:migration_file_name => "create_delayed_jobs"
end
Expand Down
3 changes: 2 additions & 1 deletion generators/delayed_job/templates/migration.rb
View file
Expand Up @@ -11,7 +11,8 @@ def self.up
table.string :locked_by # Who is working on this object (if locked)
table.timestamps
end


add_index :delayed_jobs, [:priority, :run_at], :name => 'delayed_jobs_priority'
end

def self.down
Expand Down
1 change: 0 additions & 1 deletion init.rb
View file

This file was deleted.