Skip to content
Browse files

Merge pull request #1519 from mperham/3-0

Sidekiq 3.0 changes
  • Loading branch information...
2 parents 404069a + 47ec731 commit fae29136870e01cdddb2f49f9f75e3ccb6f79790 @mperham committed
View
24 Changes.md
@@ -1,3 +1,27 @@
+3.0.0
+-----------
+
+Please see [Upgrading.md](Upgrading.md) for more comprehensive upgrade notes.
+
+- **Global Error Handlers** - blocks of code which handle errors that
+ occur anywhere within Sidekiq, not just within middleware.
+- **Dead Job Queue** - jobs which run out of retries are now moved to a dead
+ job queue. These jobs must be retried manually or they will expire
+ after 6 months or 10,000 jobs. The Web UI contains a "Dead" tab
+ exposing these jobs.
+- **Remove official support for Ruby 1.9** Things still might work but
+ I no longer actively test on it.
+- **Remove built-in support for Redis-to-Go**.
+ Heroku users: `heroku config:set REDIS_PROVIDER=REDISTOGO_URL`
+- Removed 'sidekiq/yaml\_patch', this was never documented or recommended.
+- Removed the 'started' worker data, it originally provided compatibility with resque-web
+ but overlaps the 'run\_at' worker data.
+- **Remove built-in error integration for Airbrake, Honeybadger, ExceptionNotifier and Exceptional**.
+ Each error gem should provide its own Sidekiq integration. Update your error gem to the latest
+ version to pick up Sidekiq support.
+- Remove deprecated Sidekiq::Client.registered\_\* APIs
+- Remove deprecated support for the old Sidekiq::Worker#retries\_exhausted method.
+
2.17.7
-----------
View
50 Upgrading.md
@@ -0,0 +1,50 @@
+# Upgrading to Sidekiq 3.0
+
+Sidekiq 3.0 brings several new features but also removes old APIs and
+changes a few data elements in Redis. To upgrade cleanly:
+
+* Upgrade to the latest Sidekiq 2.x and run it for a few weeks.
+ `gem 'sidekiq', '< 3'`
+ This is only needed if you have retries pending.
+* API changes:
+ - `Sidekiq::Client.registered_workers` replaced by `Sidekiq::Workers.new`
+ - `Sidekiq::Client.registered_queues` replaced by `Sidekiq::Queue.all`
+ - `Sidekiq::Worker#retries_exhausted` replaced by `Sidekiq::Worker.sidekiq_retries_exhausted`
+ - `Sidekiq::Workers#each` has removed the third block argument `worker, msg, started_at`
+ since it was redundant with `msg['run_at']`
+* Redis-to-Go is no longer transparently activated on Heroku so as to not play
+ favorites with any particular Redis service. You need to set a config option
+ for your app:
+ `heroku config:set REDIS_PROVIDER=REDISTOGO_URL`
+* Anyone using Airbrake, Honeybadger, Exceptional or ExceptionNotifier
+ will need to update their error gem version to the latest to pull in
+ Sidekiq support. Sidekiq will not provide explicit support for these
+ services so as to not play favorites with any particular error service.
+* Ruby 1.9 is no longer officially supported. Sidekiq's official
+ support policy is to support the current and previous major releases
+ of Ruby and Rails. As of February 2014, that's Ruby 2.1, Ruby 2.0, Rails 4.0
+ and Rails 3.2. I will consider PRs to fix issues found by users.
+
+## Error Service Providers
+
+If you previously provided a middleware to capture job errors, you
+should instead provide a global error handler with Sidekiq 3.0. This
+ensures **any** error within Sidekiq will be logged appropriately, not
+just during job execution.
+
+```ruby
+if Sidekiq::VERSION < '3'
+ # old behavior
+ Sidekiq.configure_server do |config|
+ config.server_middleware do |chain|
+ chain.add MyErrorService::Middleware
+ end
+ end
+else
+ Sidekiq.configure_server do |config|
+ config.error_handlers << Proc.new {|ex,context| MyErrorService.notify(ex, context) }
+ end
+end
+```
+
+Your error handler must respond to `call(exception, context_hash)`.
View
6 lib/sidekiq.rb
@@ -4,7 +4,6 @@
require 'sidekiq/client'
require 'sidekiq/worker'
require 'sidekiq/redis_connection'
-require 'sidekiq/util'
require 'sidekiq/api'
require 'json'
@@ -20,6 +19,7 @@ module Sidekiq
:environment => nil,
:timeout => 8,
:profile => false,
+ :error_handlers => [],
}
def self.❨╯°□°❩╯︵┻━┻
@@ -117,6 +117,10 @@ def self.poll_interval=(interval)
self.options[:poll_interval] = interval
end
+ def self.error_handlers
+ self.options[:error_handlers]
+ end
+
end
require 'sidekiq/extensions/class_methods'
View
34 lib/sidekiq/api.rb
@@ -44,6 +44,10 @@ def retry_size
Sidekiq.redis {|c| c.zcard('retry') }
end
+ def dead_size
+ Sidekiq.redis {|c| c.zcard('dead') }
+ end
+
class History
def initialize(days_previous, start_date = nil)
@days_previous = days_previous
@@ -209,6 +213,7 @@ def [](name)
class SortedEntry < Job
attr_reader :score
+ attr_reader :parent
def initialize(parent, score, item)
super(item)
@@ -243,11 +248,11 @@ def add_to_queue
end
def retry
- raise "Retry not available on jobs not in the Retry queue." unless item["failed_at"]
+ raise "Retry not available on jobs which have not failed" unless item["failed_at"]
Sidekiq.redis do |conn|
results = conn.multi do
- conn.zrangebyscore('retry', score, score)
- conn.zremrangebyscore('retry', score, score)
+ conn.zrangebyscore(parent.name, score, score)
+ conn.zremrangebyscore(parent.name, score, score)
end.first
results.map do |message|
msg = Sidekiq.load_json(message)
@@ -398,6 +403,18 @@ def retry_all
end
end
+ class DeadSet < SortedSet
+ def initialize
+ super 'dead'
+ end
+
+ def retry_all
+ while size > 0
+ each(&:retry)
+ end
+ end
+ end
+
##
# Programmatic access to the current active worker set.
@@ -410,11 +427,11 @@ def retry_all
#
# workers = Sidekiq::Workers.new
# workers.size => 2
- # workers.each do |name, work, started_at|
+ # workers.each do |name, work|
# # name is a unique identifier per worker
# # work is a Hash which looks like:
# # { 'queue' => name, 'run_at' => timestamp, 'payload' => msg }
- # # started_at is a String rep of the time when the worker started working on the job
+ # # run_at is an epoch Integer.
# end
class Workers
@@ -424,10 +441,9 @@ def each(&block)
Sidekiq.redis do |conn|
workers = conn.smembers("workers")
workers.each do |w|
- json = conn.get("worker:#{w}")
- next unless json
- msg = Sidekiq.load_json(json)
- block.call(w, msg, Time.at(msg['run_at']).to_s)
+ msg = conn.get("worker:#{w}")
+ next unless msg
+ block.call(w, Sidekiq.load_json(msg))
end
end
end
View
2 lib/sidekiq/cli.rb
@@ -310,7 +310,7 @@ def parse_options(argv)
def initialize_logger
Sidekiq::Logging.initialize_logger(options[:logfile]) if options[:logfile]
- Sidekiq.logger.level = Logger::DEBUG if options[:verbose]
+ Sidekiq.logger.level = ::Logger::DEBUG if options[:verbose]
end
def write_pid
View
12 lib/sidekiq/client.rb
@@ -82,18 +82,6 @@ def default
@default ||= new
end
- # deprecated
- def registered_workers
- puts "registered_workers is deprecated, please use Sidekiq::Workers.new"
- Sidekiq.redis { |x| x.smembers('workers') }
- end
-
- # deprecated
- def registered_queues
- puts "registered_queues is deprecated, please use Sidekiq::Queue.all"
- Sidekiq::Queue.all.map(&:name)
- end
-
def push(item)
default.push(item)
end
View
47 lib/sidekiq/exception_handler.rb
@@ -1,39 +1,30 @@
+require 'sidekiq'
+
module Sidekiq
module ExceptionHandler
- def handle_exception(ex, ctxHash={})
- Sidekiq.logger.warn(ctxHash) if !ctxHash.empty?
- Sidekiq.logger.warn ex
- Sidekiq.logger.warn ex.backtrace.join("\n") unless ex.backtrace.nil?
- # This list of services is getting a bit ridiculous.
- # For future error services, please add your own
- # middleware like BugSnag does:
- # https://github.com/bugsnag/bugsnag-ruby/blob/master/lib/bugsnag/sidekiq.rb
- send_to_airbrake(ctxHash, ex) if defined?(::Airbrake)
- send_to_honeybadger(ctxHash, ex) if defined?(::Honeybadger)
- send_to_exceptional(ctxHash, ex) if defined?(::Exceptional)
- send_to_exception_notifier(ctxHash, ex) if defined?(::ExceptionNotifier)
- end
-
- private
-
- def send_to_airbrake(hash, ex)
- ::Airbrake.notify_or_ignore(ex, :parameters => hash)
- end
+ class Logger
+ def call(ex, ctxHash)
+ Sidekiq.logger.warn(ctxHash) if !ctxHash.empty?
+ Sidekiq.logger.warn ex
+ Sidekiq.logger.warn ex.backtrace.join("\n") unless ex.backtrace.nil?
+ end
- def send_to_honeybadger(hash, ex)
- ::Honeybadger.notify_or_ignore(ex, :parameters => hash)
+ # Set up default handler which just logs the error
+ Sidekiq.error_handlers << Sidekiq::ExceptionHandler::Logger.new
end
- def send_to_exceptional(hash, ex)
- if ::Exceptional::Config.should_send_to_api?
- ::Exceptional.context(hash)
- ::Exceptional::Remote.error(::Exceptional::ExceptionData.new(ex))
+ def handle_exception(ex, ctxHash={})
+ Sidekiq.error_handlers.each do |handler|
+ begin
+ handler.call(ex, ctxHash)
+ rescue => ex
+ Sidekiq.logger.error "!!! ERROR HANDLER THREW AN ERROR !!!"
+ Sidekiq.logger.error ex
+ Sidekiq.logger.error ex.backtrace.join("\n") unless ex.backtrace.nil?
+ end
end
end
- def send_to_exception_notifier(hash, ex)
- ::ExceptionNotifier.notify_exception(ex, :data => {:message => hash})
- end
end
end
View
19 lib/sidekiq/middleware/server/retry_jobs.rb
@@ -108,15 +108,24 @@ def call(worker, msg, queue)
private
+ DEAD_JOB_TIMEOUT = 180 * 24 * 60 * 60 # 6 months
+ MAX_JOBS = 10_000
+
def retries_exhausted(worker, msg)
logger.debug { "Dropping message after hitting the retry maximum: #{msg}" }
- if worker.respond_to?(:retries_exhausted)
- logger.warn { "Defining #{worker.class.name}#retries_exhausted as a method is deprecated, use `sidekiq_retries_exhausted` callback instead http://git.io/Ijju8g" }
- worker.retries_exhausted(*msg['args'])
- elsif worker.sidekiq_retries_exhausted_block?
+ if worker.sidekiq_retries_exhausted_block?
worker.sidekiq_retries_exhausted_block.call(msg)
+ else
+ Sidekiq.logger.info { "Adding a dead #{msg['class']} job" }
+ payload = Sidekiq.dump_json(msg)
+ Sidekiq.redis do |conn|
+ conn.multi do
+ conn.zadd('dead', Time.now.to_f, payload)
+ conn.zremrangebyscore('dead', '-inf', (Time.now.to_i - DEAD_JOB_TIMEOUT).to_f)
+ conn.zremrangebyrank('dead', 0, -MAX_JOBS)
+ end
+ end
end
-
rescue Exception => e
handle_exception(e, { :context => "Error calling retries_exhausted" })
end
View
73 lib/sidekiq/processor.rb
@@ -34,32 +34,30 @@ def process(work)
msgstr = work.message
queue = work.queue_name
- do_defer do
- @boss.async.real_thread(proxy_id, Thread.current)
-
- ack = true
- begin
- msg = Sidekiq.load_json(msgstr)
- klass = msg['class'].constantize
- worker = klass.new
- worker.jid = msg['jid']
-
- stats(worker, msg, queue) do
- Sidekiq.server_middleware.invoke(worker, msg, queue) do
- worker.perform(*cloned(msg['args']))
- end
+ @boss.async.real_thread(proxy_id, Thread.current)
+
+ ack = true
+ begin
+ msg = Sidekiq.load_json(msgstr)
+ klass = msg['class'].constantize
+ worker = klass.new
+ worker.jid = msg['jid']
+
+ stats(worker, msg, queue) do
+ Sidekiq.server_middleware.invoke(worker, msg, queue) do
+ worker.perform(*cloned(msg['args']))
end
- rescue Sidekiq::Shutdown
- # Had to force kill this job because it didn't finish
- # within the timeout. Don't acknowledge the work since
- # we didn't properly finish it.
- ack = false
- rescue Exception => ex
- handle_exception(ex, msg || { :message => msgstr })
- raise
- ensure
- work.acknowledge if ack
end
+ rescue Sidekiq::Shutdown
+ # Had to force kill this job because it didn't finish
+ # within the timeout. Don't acknowledge the work since
+ # we didn't properly finish it.
+ ack = false
+ rescue Exception => ex
+ handle_exception(ex, msg || { :message => msgstr })
+ raise
+ ensure
+ work.acknowledge if ack
end
@boss.async.processor_done(current_actor)
@@ -71,35 +69,19 @@ def inspect
private
- # We use Celluloid's defer to workaround tiny little
- # Fiber stacks (4kb!) in MRI 1.9.
- #
- # For some reason, Celluloid's thread dispatch, TaskThread,
- # is unstable under heavy concurrency but TaskFiber has proven
- # itself stable.
- NEED_DEFER = (RUBY_ENGINE == 'ruby' && RUBY_VERSION < '2.0.0')
-
- def do_defer(&block)
- if NEED_DEFER
- defer(&block)
- else
- yield
- end
- end
-
def identity
@str ||= "#{hostname}:#{process_id}-#{Thread.current.object_id}:default"
end
def stats(worker, msg, queue)
- # Do not conflate errors from the job with errors caused by updating stats so calling code can react appropriately
+ # Do not conflate errors from the job with errors caused by updating
+ # stats so calling code can react appropriately
retry_and_suppress_exceptions do
+ hash = Sidekiq.dump_json({:queue => queue, :payload => msg, :run_at => Time.now.to_i })
redis do |conn|
conn.multi do
conn.sadd('workers', identity)
- conn.setex("worker:#{identity}:started", EXPIRY, Time.now.to_s)
- hash = {:queue => queue, :payload => msg, :run_at => Time.now.to_i }
- conn.setex("worker:#{identity}", EXPIRY, Sidekiq.dump_json(hash))
+ conn.setex("worker:#{identity}", EXPIRY, hash)
end
end
end
@@ -125,7 +107,6 @@ def stats(worker, msg, queue)
result = conn.multi do
conn.srem("workers", identity)
conn.del("worker:#{identity}")
- conn.del("worker:#{identity}:started")
conn.incrby("stat:processed", 1)
conn.incrby(processed, 1)
end
@@ -158,7 +139,7 @@ def retry_and_suppress_exceptions(max_retries = 2)
sleep(1)
retry
else
- Sidekiq.logger.info {"Exhausted #{max_retries} retries due to Redis timeouts: #{e.inspect}"}
+ handle_exception(e, { :message => "Exhausted #{max_retries} retries"})
end
end
end
View
4 lib/sidekiq/redis_connection.rb
@@ -68,9 +68,7 @@ def log_info(options)
end
def determine_redis_provider
- # REDISTOGO_URL is only support for legacy reasons
- provider = ENV['REDIS_PROVIDER'] || 'REDIS_URL'
- ENV[provider] || ENV['REDISTOGO_URL']
+ ENV[ENV['REDIS_PROVIDER'] || 'REDIS_URL']
end
end
View
2 lib/sidekiq/version.rb
@@ -1,3 +1,3 @@
module Sidekiq
- VERSION = "2.17.7"
+ VERSION = "3.0.0"
end
View
54 lib/sidekiq/web.rb
@@ -24,6 +24,7 @@ class Web < Sinatra::Base
"Queues" => 'queues',
"Retries" => 'retries',
"Scheduled" => 'scheduled',
+ "Dead" => 'morgue',
}
class << self
@@ -70,6 +71,59 @@ def custom_tabs
redirect_with_query("#{root_path}queues/#{params[:name]}")
end
+ get '/morgue' do
+ @count = (params[:count] || 25).to_i
+ (@current_page, @total_size, @dead) = page("dead", params[:page], @count)
+ @dead = @dead.map {|msg, score| Sidekiq::SortedEntry.new(nil, score, msg) }
+ erb :morgue
+ end
+
+ get "/morgue/:key" do
+ halt 404 unless params['key']
+ @dead = Sidekiq::DeadSet.new.fetch(*parse_params(params['key'])).first
+ redirect "#{root_path}morgue" if @dead.nil?
+ erb :dead
+ end
+
+ post '/morgue' do
+ halt 404 unless params['key']
+
+ params['key'].each do |key|
+ job = Sidekiq::DeadSet.new.fetch(*parse_params(key)).first
+ next unless job
+ if params['retry']
+ job.retry
+ elsif params['delete']
+ job.delete
+ end
+ end
+ redirect_with_query("#{root_path}morgue")
+ end
+
+ post "/morgue/all/delete" do
+ Sidekiq::DeadSet.new.clear
+ redirect "#{root_path}morgue"
+ end
+
+ post "/morgue/all/retry" do
+ Sidekiq::DeadSet.new.retry_all
+ redirect "#{root_path}morgue"
+ end
+
+ post "/morgue/:key" do
+ halt 404 unless params['key']
+ job = Sidekiq::DeadSet.new.fetch(*parse_params(params['key'])).first
+ if job
+ if params['retry']
+ job.retry
+ elsif params['delete']
+ job.delete
+ end
+ end
+ redirect_with_query("#{root_path}morgue")
+ end
+
+
get '/retries' do
@count = (params[:count] || 25).to_i
(@current_page, @total_size, @retries) = page("retry", params[:page], @count)
View
21 lib/sidekiq/yaml_patch.rb
@@ -1,21 +0,0 @@
-# YAML marshalling of instances can fail in some circumstances,
-# e.g. when the instance has a handle to a Proc. This monkeypatch limits
-# the YAML serialization to just AR's internal @attributes hash.
-# The paperclip gem litters AR instances with Procs, for instance.
-#
-# Courtesy of @ryanlecompte https://gist.github.com/007b88ae90372d1a3321
-#
-
-if defined?(::ActiveRecord)
- class ActiveRecord::Base
- yaml_as "tag:ruby.yaml.org,2002:ActiveRecord"
-
- def self.yaml_new(klass, tag, val)
- klass.unscoped.find(val['attributes'][klass.primary_key])
- end
-
- def to_yaml_properties
- ['@attributes']
- end
- end
-end
View
5 test/test_api.rb
@@ -352,15 +352,14 @@ class ApiWorker
c.multi do
c.sadd('workers', s)
c.set("worker:#{s}", data)
- c.set("worker:#{s}:started", Time.now.to_s)
end
end
assert_equal 1, w.size
- w.each do |x, y, z|
+ w.each do |x, y|
assert_equal s, x
assert_equal 'default', y['queue']
- assert_equal Time.now.year, DateTime.parse(z).year
+ assert_equal Time.now.year, Time.at(y['run_at']).year
end
s = '12346'
View
83 test/test_exception_handler.rb
@@ -1,5 +1,4 @@
require 'helper'
-require 'sidekiq'
require 'sidekiq/exception_handler'
require 'stringio'
require 'logger'
@@ -53,86 +52,4 @@ class TestExceptionHandler < Sidekiq::Test
end
end
- describe "with fake Airbrake" do
- before do
- ::Airbrake = Minitest::Mock.new
- end
-
- after do
- Object.__send__(:remove_const, "Airbrake") # HACK should probably inject Airbrake etc into this class in the future
- end
-
- it "notifies Airbrake" do
- ::Airbrake.expect(:notify_or_ignore,nil,[TEST_EXCEPTION,:parameters => { :a => 1 }])
- Component.new.invoke_exception(:a => 1)
- ::Airbrake.verify
- end
- end
-
- describe "with fake Honeybadger" do
- before do
- ::Honeybadger = Minitest::Mock.new
- end
-
- after do
- Object.__send__(:remove_const, "Honeybadger") # HACK should probably inject Honeybadger etc into this class in the future
- end
-
- it "notifies Honeybadger" do
- ::Honeybadger.expect(:notify_or_ignore,nil,[TEST_EXCEPTION,:parameters => { :a => 1 }])
- Component.new.invoke_exception(:a => 1)
- ::Honeybadger.verify
- end
- end
-
- describe "with fake ExceptionNotifier" do
- before do
- ::ExceptionNotifier = MiniTest::Mock.new
- end
-
- after do
- Object.__send__(:remove_const, "ExceptionNotifier")
- end
-
- it "notifies ExceptionNotifier" do
- ::ExceptionNotifier.expect(:notify_exception,true,[TEST_EXCEPTION, :data => { :message => { :b => 2 } }])
- Component.new.invoke_exception(:b => 2)
- ::ExceptionNotifier.verify
- end
- end
-
- describe "with fake Exceptional" do
- before do
- ::Exceptional = Class.new do
-
- def self.context(msg)
- @msg = msg
- end
-
- def self.check_context
- @msg
- end
- end
-
- ::Exceptional::Config = Minitest::Mock.new
- ::Exceptional::Remote = Minitest::Mock.new
- ::Exceptional::ExceptionData = Minitest::Mock.new
- end
-
- after do
- Object.__send__(:remove_const, "Exceptional")
- end
-
- it "notifies Exceptional" do
- ::Exceptional::Config.expect(:should_send_to_api?,true)
- exception_data = Object.new
- ::Exceptional::Remote.expect(:error,nil,[exception_data])
- ::Exceptional::ExceptionData.expect(:new,exception_data,[TEST_EXCEPTION])
- Component.new.invoke_exception(:c => 3)
- assert_equal({:c => 3},::Exceptional.check_context,"did not record arguments properly")
- ::Exceptional::Config.verify
- ::Exceptional::Remote.verify
- ::Exceptional::ExceptionData.verify
- end
- end
end
View
6 test/test_redis_connection.rb
@@ -81,12 +81,6 @@ def with_env_var(var, uri, skip_provider=false)
ENV[var] = nil
end
- describe "with REDISTOGO_URL set" do
- it "sets connection URI to RedisToGo" do
- with_env_var 'REDISTOGO_URL', 'redis://redis-to-go:6379/0'
- end
- end
-
describe "with REDISTOGO_URL and a parallel REDIS_PROVIDER set" do
it "sets connection URI to the provider" do
uri = 'redis://sidekiq-redis-provider:6379/0'
View
18 test/test_retry.rb
@@ -196,24 +196,6 @@ def @redis.with; yield self; end
let(:worker) { Minitest::Mock.new }
let(:msg){ {"class"=>"Bob", "args"=>[1, 2, "foo"], "queue"=>"default", "error_message"=>"kerblammo!", "error_class"=>"RuntimeError", "failed_at"=>Time.now.to_f, "retry"=>3, "retry_count"=>3} }
- describe "worker method" do
- let(:worker) do
- klass = Class.new do
- include Sidekiq::Worker
-
- def self.name; "Worker"; end
-
- def retries_exhausted(*args)
- args << "retried_method"
- end
- end
- end
-
- it 'calls worker.retries_exhausted after too many retries' do
- assert_equal [1,2, "foo", "retried_method"], handler.__send__(:retries_exhausted, worker.new, msg)
- end
- end
-
describe "worker block" do
let(:worker) do
Class.new do
View
57 test/test_web.rb
@@ -32,7 +32,6 @@ def perform(a, b)
Sidekiq.redis do |conn|
identity = 'foo:1234-123abc:default'
conn.sadd('workers', identity)
- conn.setex("worker:#{identity}:started", 10, Time.now.to_s)
hash = {:queue => 'critical', :payload => { 'class' => WebWorker.name, 'args' => [1,'abc'] }, :run_at => Time.now.to_i }
conn.setex("worker:#{identity}", 10, Sidekiq.dump_json(hash))
end
@@ -283,7 +282,6 @@ def perform(a, b)
Sidekiq.redis do |conn|
identity = 'foo:1234-123abc:default'
conn.sadd('workers', identity)
- conn.setex("worker:#{identity}:started", 10, Time.now.to_s)
hash = {:queue => 'critical', :payload => { 'class' => "FailWorker", 'args' => ["<a>hello</a>"] }, :run_at => Time.now.to_i }
conn.setex("worker:#{identity}", 10, Sidekiq.dump_json(hash))
end
@@ -394,6 +392,41 @@ def perform(a, b)
end
end
+ describe 'dead jobs' do
+ it 'shows empty index' do
+ get 'morgue'
+ assert_equal 200, last_response.status
+ end
+
+ it 'shows index with jobs' do
+ (_, score) = add_dead
+ get 'morgue'
+ assert_equal 200, last_response.status
+ assert_match /#{score}/, last_response.body
+ end
+
+ it 'can delete all dead' do
+ 3.times { add_dead }
+
+ assert_equal 3, Sidekiq::DeadSet.new.size
+ post "/morgue/all/delete", 'delete' => 'Delete'
+ assert_equal 0, Sidekiq::DeadSet.new.size
+ assert_equal 302, last_response.status
+ assert_equal 'http://example.org/morgue', last_response.header['Location']
+ end
+
+ it 'can retry a dead job' do
+ params = add_dead
+ post "/morgue/#{job_params(*params)}", 'retry' => 'Retry'
+ assert_equal 302, last_response.status
+ assert_equal 'http://example.org/morgue', last_response.header['Location']
+
+ get '/queues/foo'
+ assert_equal 200, last_response.status
+ assert_match /#{params.first['args'][2]}/, last_response.body
+ end
+ end
+
def add_scheduled
score = Time.now.to_f
msg = { 'class' => 'HardWorker',
@@ -421,6 +454,22 @@ def add_retry
[msg, score]
end
+ def add_dead
+ msg = { 'class' => 'HardWorker',
+ 'args' => ['bob', 1, Time.now.to_f],
+ 'queue' => 'foo',
+ 'error_message' => 'Some fake message',
+ 'error_class' => 'RuntimeError',
+ 'retry_count' => 0,
+ 'failed_at' => Time.now.utc,
+ 'jid' => SecureRandom.hex(12) }
+ score = Time.now.to_f
+ Sidekiq.redis do |conn|
+ conn.zadd('dead', score, Sidekiq.dump_json(msg))
+ end
+ [msg, score]
+ end
+
def add_xss_retry
msg = { 'class' => 'FailWorker',
'args' => ['<a>hello</a>'],
@@ -441,8 +490,8 @@ def add_worker
process_id = rand(1000)
msg = "{\"queue\":\"default\",\"payload\":{\"retry\":true,\"queue\":\"default\",\"timeout\":20,\"backtrace\":5,\"class\":\"HardWorker\",\"args\":[\"bob\",10,5],\"jid\":\"2b5ad2b016f5e063a1c62872\"},\"run_at\":1361208995}"
Sidekiq.redis do |conn|
- conn.sadd("workers", "mercury.home:#{process_id}-70215157189060:started")
- conn.set("worker:mercury.home:#{process_id}-70215157189060:started", msg)
+ conn.sadd("workers", "mercury.home:#{process_id}-70215157189060:default")
+ conn.set("worker:mercury.home:#{process_id}-70215157189060:default", msg)
end
end
end
View
3 web/locales/en.yml
@@ -61,3 +61,6 @@ en: # <---- change this to your locale code
ThreeMonths: 3 months
SixMonths: 6 months
Failures: Failures
+ DeadJobs: Dead Jobs
+ NoDeadJobsFound: No dead jobs were found
+ Dead: Dead
View
10 web/views/_job_info.erb
@@ -53,12 +53,12 @@
</tr>
<tr>
<th><%= t('LastRetry') %></th>
- <td><%= relative_time(job['retried_at'].is_a?(Numeric) ? Time.at(job['retried_at']) : Time.parse(job['retried_at'])) %></td>
+ <td><%= relative_time(Time.at(job['retried_at'])) %></td>
</tr>
<% else %>
<tr>
<th><%= t('OriginallyFailed') %></th>
- <td><%= relative_time(job['failed_at'].is_a?(Numeric) ? Time.at(job['failed_at']) : Time.parse(job['failed_at'])) %></td>
+ <td><%= relative_time(Time.at(job['failed_at'])) %></td>
</tr>
<% end %>
<tr>
@@ -72,5 +72,11 @@
<td><%= relative_time(job.at) %></td>
</tr>
<% end %>
+ <% if type == :dead %>
+ <tr>
+ <th><%= t('LastRetry') %></th>
+ <td><%= relative_time(job.at) %></td>
+ </tr>
+ <% end %>
</tbody>
</table>
View
18 web/views/_summary.erb
@@ -1,34 +1,40 @@
<ul class="list-unstyled summary row">
- <li class="processed col-sm-2">
+ <li class="processed col-sm-1">
<span class="count"><%= number_with_delimiter(stats.processed) %></span>
<span class="desc"><%= t('Processed') %></span>
</li>
- <li class="failed col-sm-2">
+ <li class="failed col-sm-1">
<span class="count"><%= number_with_delimiter(stats.failed) %></span>
<span class="desc"><%= t('Failed') %></span>
</li>
- <li class="busy col-sm-2">
+ <li class="busy col-sm-1">
<a href="<%= root_path %>workers">
<span class="count"><%= number_with_delimiter(workers_size) %></span>
<span class="desc"><%= t('Busy') %></span>
</a>
</li>
- <li class="enqueued col-sm-2">
+ <li class="enqueued col-sm-1">
<a href="<%= root_path %>queues">
<span class="count"><%= number_with_delimiter(stats.enqueued) %></span>
<span class="desc"><%= t('Enqueued') %></span>
</a>
</li>
- <li class="retries col-sm-2">
+ <li class="retries col-sm-1">
<a href="<%= root_path %>retries">
<span class="count"><%= number_with_delimiter(stats.retry_size) %></span>
<span class="desc"><%= t('Retries') %></span>
</a>
</li>
- <li class="scheduled col-sm-2">
+ <li class="scheduled col-sm-1">
<a href="<%= root_path %>scheduled">
<span class="count"><%= number_with_delimiter(stats.scheduled_size) %></span>
<span class="desc"><%= t('Scheduled') %></span>
</a>
</li>
+ <li class="dead col-sm-1">
+ <a href="<%= root_path %>morgue">
+ <span class="count"><%= number_with_delimiter(stats.dead_size) %></span>
+ <span class="desc"><%= t('Dead') %></span>
+ </a>
+ </li>
</ul>
View
4 web/views/_workers.erb
@@ -6,7 +6,7 @@
<th><%= t('Arguments') %></th>
<th><%= t('Started') %></th>
</thead>
- <% workers.each_with_index do |(worker, msg, run_at), index| %>
+ <% workers.each_with_index do |(worker, msg), index| %>
<tr>
<td><%= worker %></td>
<td>
@@ -16,7 +16,7 @@
<td>
<div class="args"><%= display_args(msg['payload']['args']) %></div>
</td>
- <td><%= relative_time(run_at.is_a?(String) ? DateTime.parse(run_at) : run_at) %></td>
+ <td><%= relative_time(Time.at(msg['run_at'])) %></td>
</tr>
<% end %>
</table>
View
30 web/views/dead.erb
@@ -0,0 +1,30 @@
+<%= erb :_job_info, :locals => {:job => @dead, :type => :dead} %>
+
+<h3><%= t('Error') %></h3>
+<table class="error table table-bordered table-striped">
+ <tbody>
+ <tr>
+ <th><%= t('ErrorClass') %></th>
+ <td>
+ <code><%= @dead['error_class'] %></code>
+ </td>
+ </tr>
+ <tr>
+ <th><%= t('ErrorMessage') %></th>
+ <td><%= h(@dead['error_message']) %></td>
+ </tr>
+ <% if !@dead['error_backtrace'].nil? %>
+ <tr>
+ <th><%= t('ErrorBacktrace') %></th>
+ <td>
+ <code><%= @dead['error_backtrace'].join("<br/>") %></code>
+ </td>
+ </tr>
+ <% end %>
+ </tbody>
+</table>
+<form class="form-horizontal" action="<%= root_path %>morgue/<%= job_params(@dead, @dead.score) %>" method="post">
+ <a class="btn btn-default" href="<%= root_path %>morgue"><%= t('GoBack') %></a>
+ <input class="btn btn-primary" type="submit" name="retry" value="<%= t('RetryNow') %>" />
+ <input class="btn btn-danger" type="submit" name="delete" value="<%= t('Delete') %>" />
+</form>
View
66 web/views/morgue.erb
@@ -0,0 +1,66 @@
+<header class="row">
+ <div class="col-sm-5">
+ <h3><%= t('DeadJobs') %></h3>
+ </div>
+ <% if @dead.size > 0 && @total_size > @count %>
+ <div class="col-sm-4">
+ <%= erb :_paging, :locals => { :url => "#{root_path}morgue" } %>
+ </div>
+ <% end %>
+ <%= filtering('dead') %>
+</header>
+
+<% if @dead.size > 0 %>
+ <form action="<%= root_path %>morgue" method="post">
+ <table class="table table-striped table-bordered table-white">
+ <thead>
+ <tr>
+ <th width="20px" class="table-checkbox">
+ <label>
+ <input type="checkbox" class="check_all" />
+ </label>
+ </th>
+ <th width="25%"><%= t('LastRetry') %></th>
+ <th><%= t('Queue') %></th>
+ <th><%= t('Worker') %></th>
+ <th><%= t('Arguments') %></th>
+ <th><%= t('Error') %></th>
+ </tr>
+ </thead>
+ <% @dead.each do |entry| %>
+ <tr>
+ <td class="table-checkbox">
+ <label>
+ <input type='checkbox' name='key[]' value='<%= job_params(entry.item, entry.score) %>' />
+ </label>
+ </td>
+ <td>
+ <a href="<%= root_path %>morgue/<%= job_params(entry.item, entry.score) %>"><%= relative_time(entry.at) %></a>
+ </td>
+ <td>
+ <a href="<%= root_path %>queues/<%= entry.queue %>"><%= entry.queue %></a>
+ </td>
+ <td><%= entry.klass %></td>
+ <td>
+ <div class="args"><%= display_args(entry.args) %></div>
+ </td>
+ <td>
+ <div><%= h truncate("#{entry['error_class']}: #{entry['error_message']}", 200) %></div>
+ </td>
+ </tr>
+ <% end %>
+ </table>
+ <input class="btn btn-primary btn-xs pull-left" type="submit" name="retry" value="<%= t('RetryNow') %>" />
+ <input class="btn btn-danger btn-xs pull-left" type="submit" name="delete" value="<%= t('Delete') %>" />
+ </form>
+
+ <form action="<%= root_path %>morgue/all/delete" method="post">
+ <input class="btn btn-danger btn-xs pull-right" type="submit" name="delete" value="<%= t('DeleteAll') %>" data-confirm="<%= t('AreYouSure') %>" />
+ </form>
+ <form action="<%= root_path %>morgue/all/retry" method="post">
+ <input class="btn btn-danger btn-xs pull-right" type="submit" name="retry" value="<%= t('RetryAll') %>" data-confirm="<%= t('AreYouSure') %>" />
+ </form>
+
+<% else %>
+ <div class="alert alert-success"><%= t('NoDeadJobsFound') %></div>
+<% end %>

0 comments on commit fae2913

Please sign in to comment.
Something went wrong with that request. Please try again.