undefined method `get_sidekiq_options' for "MyScheduledWorker":String #27

Closed
girak opened this Issue Dec 26, 2013 · 17 comments

Comments

6 participants
@girak

girak commented Dec 26, 2013

Our Sidekiq redis instance is shared among multiple services.
So workers are available on one of the repo while not on the others.

When a scheduled or retried job is being consumed by the sidekiq poller running on another service, we need to to safely re-enqueue the task without exception.

Currently Sidekiq does re-enqueue jobs without the worker class without issue, but apparently the unique jobs middleware is not accounting for this case.

I have workaround the issue with a monkey patch, but an official fix is much appreciated. Thanks for the hard work!

@mhenrixon

This comment has been minimized.

Show comment Hide comment
@mhenrixon

mhenrixon Dec 31, 2013

Owner

@girak could you point me to where that job is re-enqueued in sidekiq? I'm pretty sure we could mimic that behaviour.

Owner

mhenrixon commented Dec 31, 2013

@girak could you point me to where that job is re-enqueued in sidekiq? I'm pretty sure we could mimic that behaviour.

@girak

This comment has been minimized.

Show comment Hide comment
@mhenrixon

This comment has been minimized.

Show comment Hide comment
@mhenrixon

mhenrixon Jan 27, 2014

Owner

Ok so this happened to me as well now that I split up our backend processing in multiple different projects. I will see if I can get this solved somehow. First a failing test case.

Owner

mhenrixon commented Jan 27, 2014

Ok so this happened to me as well now that I split up our backend processing in multiple different projects. I will see if I can get this solved somehow. First a failing test case.

@mhenrixon

This comment has been minimized.

Show comment Hide comment
@mhenrixon

mhenrixon Jan 27, 2014

Owner

For us it looks like the error is with Sidekiq. Could you provide a stacktrace?

Owner

mhenrixon commented Jan 27, 2014

For us it looks like the error is with Sidekiq. Could you provide a stacktrace?

@felixyz

This comment has been minimized.

Show comment Hide comment
@felixyz

felixyz Feb 3, 2014

I've got a quirky little RSpec spec here: https://gist.github.com/Felixyz/8781057
It test drives (synchronously) the Sidekiq retry mechanism with sidekiq-unique-jobs inserted in the middleware chain and attempts to mimic the setup we have, with two queues. The script does not reproduce the bug, from which I draw the conclusion that the bug (if it is a bug) is probably in Sidekiq's higher-level coordination classes, like Manager, Fetcher or Launcher.

Whatever it is, it's perhaps not so much a bug as an uncompatibility in sidekiq-unique-job's handling of the parameters? By which I mean: this is probably not an issue that we would report with Sidekiq itself, right?

felixyz commented Feb 3, 2014

I've got a quirky little RSpec spec here: https://gist.github.com/Felixyz/8781057
It test drives (synchronously) the Sidekiq retry mechanism with sidekiq-unique-jobs inserted in the middleware chain and attempts to mimic the setup we have, with two queues. The script does not reproduce the bug, from which I draw the conclusion that the bug (if it is a bug) is probably in Sidekiq's higher-level coordination classes, like Manager, Fetcher or Launcher.

Whatever it is, it's perhaps not so much a bug as an uncompatibility in sidekiq-unique-job's handling of the parameters? By which I mean: this is probably not an issue that we would report with Sidekiq itself, right?

@felixyz

This comment has been minimized.

Show comment Hide comment
@felixyz

felixyz Feb 3, 2014

@girak Would you care to share your patch?

felixyz commented Feb 3, 2014

@girak Would you care to share your patch?

@felixyz

This comment has been minimized.

Show comment Hide comment
@felixyz

felixyz Feb 3, 2014

One more observation that might be useful: we're getting fewer of these errors every day. My impression is that they might occur after a restart of the Sidekiq processes. We are running Sidekiq Pro. Are others who have been seeing this problem running Sidekiq or Sidekiq Pro?

felixyz commented Feb 3, 2014

One more observation that might be useful: we're getting fewer of these errors every day. My impression is that they might occur after a restart of the Sidekiq processes. We are running Sidekiq Pro. Are others who have been seeing this problem running Sidekiq or Sidekiq Pro?

@mhenrixon

This comment has been minimized.

Show comment Hide comment
@mhenrixon

mhenrixon Feb 3, 2014

Owner

Yeah so @felixyz is right and that's why I was asking for a stacktrace. @girak I can't replicate that sidekiq unique jobs should be the problem in this situation. The crash actually happens in Sidekiq and the reason it happens in Sidekiq for us is because the wrong server was processing jobs it could not handle (we assigned it to the wrong queue).

Owner

mhenrixon commented Feb 3, 2014

Yeah so @felixyz is right and that's why I was asking for a stacktrace. @girak I can't replicate that sidekiq unique jobs should be the problem in this situation. The crash actually happens in Sidekiq and the reason it happens in Sidekiq for us is because the wrong server was processing jobs it could not handle (we assigned it to the wrong queue).

@felixyz

This comment has been minimized.

Show comment Hide comment
@felixyz

felixyz Feb 3, 2014

@mhenrixon I'm not sure you can get a great stack trace because of the async nature of Sidekiq, but (as you probably know) the line that causes the exception is:
/app/vendor/bundle/ruby/2.0.0/gems/sidekiq-unique-jobs-2.6.7/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:in `call'

In any case: yes, we also concluded that a worker was assigned a job from the wrong queue. However, I haven't been able to track down the reason why this happens. Any tips for that?
(The only thing I can think of is that our two kinds of workers inherit from a common base class, which is the one that mixes in Sidekiq::Worker.)

felixyz commented Feb 3, 2014

@mhenrixon I'm not sure you can get a great stack trace because of the async nature of Sidekiq, but (as you probably know) the line that causes the exception is:
/app/vendor/bundle/ruby/2.0.0/gems/sidekiq-unique-jobs-2.6.7/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:in `call'

In any case: yes, we also concluded that a worker was assigned a job from the wrong queue. However, I haven't been able to track down the reason why this happens. Any tips for that?
(The only thing I can think of is that our two kinds of workers inherit from a common base class, which is the one that mixes in Sidekiq::Worker.)

@mhenrixon

This comment has been minimized.

Show comment Hide comment
@mhenrixon

mhenrixon Feb 3, 2014

Owner

@felixyz for us it was just me assigning the job to the wrong queue simple as that. Unfortunately we had some trouble updating Heroku so deploys would be listed as successful even though it failed in CI. Took me a while to figure out why the code changes didn't "take". We have seen a couple of incidents after but I honestly have no idea. The only thing I've been able to think of is that Sidekiq simply picks the wrong job.

Owner

mhenrixon commented Feb 3, 2014

@felixyz for us it was just me assigning the job to the wrong queue simple as that. Unfortunately we had some trouble updating Heroku so deploys would be listed as successful even though it failed in CI. Took me a while to figure out why the code changes didn't "take". We have seen a couple of incidents after but I honestly have no idea. The only thing I've been able to think of is that Sidekiq simply picks the wrong job.

@sahin

This comment has been minimized.

Show comment Hide comment
@sahin

sahin Mar 5, 2014

we have the same problem , I hope this helps...

NoMethodError - undefined method get_sidekiq_options' for "DataTAWorker":String: /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sidekiq-unique-jobs-2.3.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:incall'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:124:in block in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:incall'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:159:inprocess_single'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:47:in push' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:86:inpush'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:260:in block (2 levels) in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:inmap'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:in block in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/connection_pool-1.2.0/lib/connection_pool.rb:55:inwith'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq.rb:67:in redis' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:252:inretry'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:inblock (2 levels) in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:in each' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:inblock in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:in loop' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:ineach'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:401:in retry_all' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/web.rb:162:inblock in class:Web'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:inblock in compile!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:in []' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:inblock (3 levels) in route!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:976:in `route_eval'

sahin commented Mar 5, 2014

we have the same problem , I hope this helps...

NoMethodError - undefined method get_sidekiq_options' for "DataTAWorker":String: /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sidekiq-unique-jobs-2.3.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:11:incall'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:124:in block in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:incall'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/middleware/chain.rb:127:in invoke' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:159:inprocess_single'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:47:in push' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/client.rb:86:inpush'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:260:in block (2 levels) in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:inmap'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:257:in block in retry' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/connection_pool-1.2.0/lib/connection_pool.rb:55:inwith'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq.rb:67:in redis' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:252:inretry'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:301:inblock (2 levels) in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:in each' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:300:inblock in each'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:in loop' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:292:ineach'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/api.rb:401:in retry_all' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/bundler/gems/sidekiq-8dae3840a3f8/lib/sidekiq/web.rb:162:inblock in class:Web'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:in call' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:1593:inblock in compile!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:in []' /var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:957:inblock (3 levels) in route!'
/var/www/vhosts/movielala.com/workers_app/shared/bundle/ruby/2.1.0/gems/sinatra-1.4.4/lib/sinatra/base.rb:976:in `route_eval'

@girak

This comment has been minimized.

Show comment Hide comment
@girak

girak Mar 5, 2014

@felixyz Hey guys, here's the monkey patch i did on our repos.

module SidekiqUniqueJobs
  module Middleware
    module Client
      class UniqueJobs
        def unique_enabled?
          if worker_class.respond_to? :get_sidekiq_options
            worker_class.get_sidekiq_options['unique'] || item['unique']
          else
            false
          end
        end

        def unique_job_expiration
          if worker_class.respond_to? :get_sidekiq_options
            worker_class.get_sidekiq_options['unique_job_expiration']
          else
            SidekiqUniqueJobs::Config.default_expiration
          end
        end
      end
    end
  end
end

girak commented Mar 5, 2014

@felixyz Hey guys, here's the monkey patch i did on our repos.

module SidekiqUniqueJobs
  module Middleware
    module Client
      class UniqueJobs
        def unique_enabled?
          if worker_class.respond_to? :get_sidekiq_options
            worker_class.get_sidekiq_options['unique'] || item['unique']
          else
            false
          end
        end

        def unique_job_expiration
          if worker_class.respond_to? :get_sidekiq_options
            worker_class.get_sidekiq_options['unique_job_expiration']
          else
            SidekiqUniqueJobs::Config.default_expiration
          end
        end
      end
    end
  end
end
@sahin

This comment has been minimized.

Show comment Hide comment
@sahin

sahin Mar 6, 2014

thx a lot.... When do u think u can merge it to master?

sahin commented Mar 6, 2014

thx a lot.... When do u think u can merge it to master?

@sahin

This comment has been minimized.

Show comment Hide comment
@sahin

sahin Mar 6, 2014

we had problems using monkey patch in the new sidekiq 3.0, for now, we are doing this in sidetiq

def perform

uniq = Hash.new
Sidekiq::DeadSet.new.each {|job|
  md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
  digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))

  if uniq[digest]
    job.delete
  else
    uniq[digest] = job['jid']
  end

}

uniq = Hash.new
Sidekiq::RetrySet.new.each {|job|
  md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
  digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))

  if uniq[digest]
    job.delete
  else
    uniq[digest] = job['jid']
  end

}

end

sahin commented Mar 6, 2014

we had problems using monkey patch in the new sidekiq 3.0, for now, we are doing this in sidetiq

def perform

uniq = Hash.new
Sidekiq::DeadSet.new.each {|job|
  md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
  digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))

  if uniq[digest]
    job.delete
  else
    uniq[digest] = job['jid']
  end

}

uniq = Hash.new
Sidekiq::RetrySet.new.each {|job|
  md5_arguments = {class: job['class'], queue: job['queue'], args: job['args']}
  digest = Digest::MD5.hexdigest(Sidekiq.dump_json(md5_arguments))

  if uniq[digest]
    job.delete
  else
    uniq[digest] = job['jid']
  end

}

end

@Altonymous

This comment has been minimized.

Show comment Hide comment
@Altonymous

Altonymous Aug 5, 2014

Any updates on this issue?

2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: undefined method `get_sidekiq_options' for "Reporter":String
2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: /Users/nunya/.rvm/gems/ruby-2.1.0@reporting-bot/gems/sidekiq-unique-jobs-3.0.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:91:in `unique_enabled?'

Any updates on this issue?

2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: undefined method `get_sidekiq_options' for "Reporter":String
2014-08-05T18:54:21.401Z 14837 TID-ox8y02exc ERROR: /Users/nunya/.rvm/gems/ruby-2.1.0@reporting-bot/gems/sidekiq-unique-jobs-3.0.2/lib/sidekiq-unique-jobs/middleware/client/unique_jobs.rb:91:in `unique_enabled?'
@andypalmer

This comment has been minimized.

Show comment Hide comment
@andypalmer

andypalmer Nov 11, 2014

I've run into this issue too (I'm using sidekiq-unique-jobs 3.0.9). It comes when trying to use Sidekiq::RetrySet.new.retry_all
The problem appears to be that retry_all is inserting the failed jobs back into the execution queue without inflating the worker class (which is reasonable, this is Sidekiq queue maintenance, not Worker work). The problem comes about because sidekiq-unique-jobs intercepts the insertion and is expecting it to be something that has been called with Worker.perform_async, but it's actually just the class name (as a string)

This is why @girak 's monkey patch works. String does not respond to get_sidekiq_options, so we ignore it.

Thinking out loud
I'm not sure about the best way to handle this atm. Ideally we want the sidekiq queue insertion to be consistent, but we need sidekiq-unique-jobs to be able to handle the case where sidekiq-unique-jobs has been inserted into the middleware chain, but the worker class has not been "required". Maybe we need to add more metadata into the hash so sidekiq-unique-jobs can tell whether this job has already been "uniquified"

I've run into this issue too (I'm using sidekiq-unique-jobs 3.0.9). It comes when trying to use Sidekiq::RetrySet.new.retry_all
The problem appears to be that retry_all is inserting the failed jobs back into the execution queue without inflating the worker class (which is reasonable, this is Sidekiq queue maintenance, not Worker work). The problem comes about because sidekiq-unique-jobs intercepts the insertion and is expecting it to be something that has been called with Worker.perform_async, but it's actually just the class name (as a string)

This is why @girak 's monkey patch works. String does not respond to get_sidekiq_options, so we ignore it.

Thinking out loud
I'm not sure about the best way to handle this atm. Ideally we want the sidekiq queue insertion to be consistent, but we need sidekiq-unique-jobs to be able to handle the case where sidekiq-unique-jobs has been inserted into the middleware chain, but the worker class has not been "required". Maybe we need to add more metadata into the hash so sidekiq-unique-jobs can tell whether this job has already been "uniquified"

@mhenrixon

This comment has been minimized.

Show comment Hide comment
@mhenrixon

mhenrixon Dec 20, 2014

Owner

This should be fixed since some time ago.

Owner

mhenrixon commented Dec 20, 2014

This should be fixed since some time ago.

@mhenrixon mhenrixon closed this Dec 20, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment