Sidekiq worker using a a “custom” queue name gets assigned to “default” queue

[JigneshGohel-BoTreeConsulting]

Rails 3.2.9
Ruby 1.9.3dev (2011-09-23 revision 33323) [i686-linux]
sidekiq  2.10.1

If I define a worker like below with a custom queue name

  class BulkEmailWorker
    include Sidekiq::Worker
   # https://github.com/mperham/sidekiq/wiki/Advanced-Options
    sidekiq_options(queue: :bulk_mails, backtrace: true)

   # https://github.com/mperham/sidekiq/wiki/Best-Practices
  # Section: Make your jobs small and simple
  def perform
   # https://github.com/mperham/sidekiq/wiki/Delayed-Extensions
   scheduled_mails = MailTemplate.with_schedule
    scheduled_mails.each do |mail_template|
     BulkMailer.delay.general_mail(mail_template.id)
   end
 end

end

and I invoke the worker like this:

  BulkEmailWorker.perform_async

it doesn't work (in other words doesn't send email).

Inspecting the data in redis I found following:

redis 127.0.0.1:6379> keys *

1. "myapp:stat:processed:2013-04-24"
2. "myapp:queue:default"
3. "myapp:stat:processed"
4. "myapp:queues"

  redis 127.0.0.1:6379> lrange myapp:queue:default -100 100

    1) "{\"retry\":true,\"queue\":\"default\",\"timeout\":30,\"class\":\"Sidekiq::Extensions::DelayedMailer\",\"args\":[\"---\\n- !ruby/class 'BulkMailer'\\n- :general_mail\\n- - 1\\n\"],\"jid\":\"e46693944febf7ae26ec67a0\"}"

As can be seen above the worker is assigned to queue "default" and I guess due to this the worker is unable to process.

I started sidekiq using following command for above scenario:

bundle exec sidekiq -e dev_mysql -C config/sidekiq.yml -q bulk_mails

However when I remove from my worker following

sidekiq_options(queue: :bulk_mails, backtrace: true)

and start sidekiq using following command:

bundle exec sidekiq -e dev_mysql -C config/sidekiq.yml

it works (in other words it sends email successfully).

/config/sidekiq.yml

 # https://github.com/mperham/sidekiq/wiki/Logging
 # http://stackoverflow.com/questions/15260634/sidekiq-configuration-for-multiple-   environments
 ---
  :verbose: true
 :pidfile: ./tmp/pids/sidekiq.pid
 :logfile: ./log/sidekiq.log
 :concurrency:  25

I have just started using Sidekiq and am novice to it.So please bear with me if I there is some concept which I misunderstood or unaware about.

I need to get rid of the mentioned problem so that my worker gets assigned to the desired queue and also processed successfully.

Thanks,
Jignesh

[mperham]

There's two different Workers in this case. The delay extension uses a separate worker which uses the default queue.

On 24 Apr 2013, at 22:45, JigneshGohel-BoTreeConsulting notifications@github.com wrote:

Rails 3.2.9
Ruby 1.9.3dev (2011-09-23 revision 33323) [i686-linux]
sidekiq 2.10.1
If I define a worker like below with a custom queue name

class BulkEmailWorker
include Sidekiq::Worker

https://github.com/mperham/sidekiq/wiki/Advanced-Options

sidekiq_options(queue: :bulk_mails, backtrace: true)

https://github.com/mperham/sidekiq/wiki/Best-Practices

Section: Make your jobs small and simple

def perform

https://github.com/mperham/sidekiq/wiki/Delayed-Extensions

scheduled_mails = MailTemplate.with_schedule
scheduled_mails.each do |mail_template|
BulkMailer.delay.general_mail(mail_template.id)
end
end
end

and I invoke the worker like this:

BulkEmailWorker.perform_async
it doesn't work (in other words doesn't send email).

Inspecting the data in redis I found following:

redis 127.0.0.1:6379> keys *

1. "myapp:stat:processed:2013-04-24"
2. "myapp:queue:default"
3. "myapp:stat:processed"
4. "myapp:queues"

redis 127.0.0.1:6379> lrange myapp:queue:default -100 100

1) "{\"retry\":true,\"queue\":\"default\",\"timeout\":30,\"class\":\"Sidekiq::Extensions::DelayedMailer\",\"args\":[\"---\\n- !ruby/class 'BulkMailer'\\n- :general_mail\\n- - 1\\n\"],\"jid\":\"e46693944febf7ae26ec67a0\"}"

As can be seen above the worker is assigned to queue "default" and I guess due to this the worker is unable to process.

I started sidekiq using following command for above scenario:

bundle exec sidekiq -e dev_mysql -C config/sidekiq.yml -q bulk_mails
However when I remove from my worker following

sidekiq_options(queue: :bulk_mails, backtrace: true)

and start sidekiq using following command:

bundle exec sidekiq -e dev_mysql -C config/sidekiq.yml
it works (in other words it sends email successfully).

/config/sidekiq.yml

https://github.com/mperham/sidekiq/wiki/Logging

http://stackoverflow.com/questions/15260634/sidekiq-configuration-for-multiple- environments

---

:verbose: true
:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:concurrency: 25
I have just started using Sidekiq and am novice to it.So please bear with me if I there is some concept which I misunderstood or unaware about.

I need to get rid of the mentioned problem so that my worker gets assigned to the desired queue and also processed successfully.

Thanks,
Jignesh

—
Reply to this email directly or view it on GitHub.

[JigneshGohel-BoTreeConsulting]

@mperham Thanks for your prompt response.So can you guide me with the solution on what is required to to make the implementation behave in desired manner as mentioned in my first post? Will it be correct if I do not use delay extension and consider the fact that BulkEmailWorker#perform_async should move the email sending task to background?

Update:

Replacing

      BulkMailer.delay.general_mail(mail_template.id)

WITH

      BulkMailer.general_mail(mail_template.id)

and invoking

    BulkEmailWorker.perform_async

doesn't even enqueue the task in any of the queues.I tried even with the default queue.

  1.9.3dev :001 > stats = Sidekiq::Stats.new
  => #<Sidekiq::Stats:0xa0e06b4> 
  1.9.3dev :002 > stats.processed
  2013-04-25T20:02:59Z 5991 TID-1c1sty INFO: Sidekiq client using redis://127.0.0.1:6379 with options {:namespace=>"wljk"}
   => 1 
  1.9.3dev :003 > stats.failed 
   => 0 
  1.9.3dev :004 > stats.queues 
   => {"default"=>0} 
  1.9.3dev :005 > stats.en
  stats.enable_warnings  stats.enqueued         stats.enum_for         
  1.9.3dev :005 > stats.enqueued 
   => 0 
  1.9.3dev :006 > 

Thanks,
Jignesh

[JigneshGohel-BoTreeConsulting]

My requirement, in-case it helps in understanding what I am trying to do, is like from my application a user can create Mail Templates with recipients wherein s/he can schedule the mails too at a specific date-time and also define its recurring interval.Once the mail template is created in the database, at the schedule date-time and if recurring later at recurring schedules the application should send emails in background.I am trying to achieve this using Sidekiq.

Thanks,
Jignesh

[mperham]

Everything works fine if you remove the custom queue name. Just use the default queue.

[JigneshGohel-BoTreeConsulting]

Sorry it was my misunderstanding due to which I posted the comment #872 (comment). Actually the worker did processed the task as can be seen from the Stats.processed value.

  1.9.3dev :002 > stats.processed
    2013-04-25T20:02:59Z 5991 TID-1c1sty INFO: Sidekiq client using redis://127.0.0.1:6379 with options    {:namespace=>"wljk"}
     => 1 

However I misunderstood that it was not processed in the sense that I didn't received any email after replacing

  BulkMailer.delay.general_mail(mail_template.id)

WITH

  BulkMailer.general_mail(mail_template.id)

But it was obvious that I didn't received any mail because I didn't invoked deliver on BulkMailer.general_mail(mail_template.id) after removing delay from my ActionMailer class.

I did that change i.e.

         BulkMailer.general_mail(mail_template.id).deliver

and I am getting the mails being sent in the background triggered by the worker and it is being assigned to the custom queue too if I use a custom queue name.

My apologies once again for the inconvenience caused due to my own misunderstandings and @mperham appreciate your prompt help.

Thanks,
Jignesh