Using sneakers - Rails 5 - Active Job

[nazarhussain]

I am using sneakers with Rails 5 and ActiveJob with following setup:

Under application.rb I have

config.active_job.queue_adapter = :sneakers
config.autoload_paths += %W(#{Rails.root}/app/jobs)

Sneakers configurations is under:

Sneakers.configure(
    {
        amqp: 'amqp://guest:guest@192.168.99.100:5672'
        vhost: '/',
        heartbeat: 5,
        handler: Sneakers::Handlers::Maxretry
    })

My job is

class PartnerImportJob < ApplicationJob
  queue_as 'app.import.partner'

  def perform(data)
    puts data.inspect
  end
end

When I run rake sneakers:run it shows following logs

2016-08-19T05:43:54Z p-82837 t-owgygl2yg WARN: Loading runner configuration...
2016-08-19T05:43:54Z p-82837 t-owgygl2yg INFO: New configuration: #<Sneakers::Configuration:0x007fc948236e78 @hash={:runner_config_file=>nil, :metrics=>nil, :daemonize=>false, :start_worker_delay=>0.2, :workers=>4, :log=>#IO:/dev/ttys006, :pid_path=>"sneakers.pid", :amqp_heartbeat=>10, :timeout_job_after=>5, :prefetch=>10, :threads=>10, :share_threads=>false, :ack=>true, :heartbeat=>5, :hooks=>{}, :exchange=>"sneakers", :exchange_options=>{:type=>:direct, :durable=>true, :auto_delete=>false, :arguments=>{}}, :queue_options=>{:durable=>true, :auto_delete=>false, :exclusive=>false, :arguments=>{}}, :amqp=>"amqp:/@192.168.99.100:5672", :vhost=>"/", :handler=>Sneakers::Handlers::Maxretry, :logger=>#<ServerEngine::DaemonLogger:0x007fc945c3e388 @rotate_age=5, @rotate_size=1048576, @logdev_class=Logger::LogDevice, @progname=nil, @Level=1, @default_formatter=#<Logger::Formatter:0x007fc945c3e310 @datetime_format=nil>, @Formatter=Sneakers::Support::ProductionFormatter, @logdev=#IO:/dev/ttys006>, :worker_type=>"process", :worker_classes=>[ActiveJob::QueueAdapters::SneakersAdapter::JobWrapper], :log_stdout=>false, :log_stderr=>false}>

But it does not process any of the job. There are few messages available in that particular queue.

What I am doing wrong here?

[quidproquo]

I noticed that it doesn't automatically create queues and that I have to use the default exchange for anything to work, or setup manual bindings if I use a custom exchange as described in the How To using 'sneakers' as an exchange. This mostly seems to be RabbitMQ-specific, and maybe it was due to newer version of the broker I'm using. After going through this for the past few days it doesn't seem that this gem is a drop-in replacement for resque.

[michaelklishin]

@quidproquo I don't think this gem claims to be a replacement for Resque. RabbitMQ has a different feature set: if you use a custom exchange, in many cases it's up to you to ensure that the bindings you need exist.

Hutch is an example of a project that's highly opinionated and tries to hide the topology from the user but it's also fairly limited for exactly that reason.

[gabrieljoelc]

@jondot do we think this is something that should go into a sneakers-rails gem? if so, should we add this to your "roadmap" records?

[jondot]

Sure! I'm just putting down things on a scratch note, these will be great topics to discuss in our upcoming meeting.
I'll make these into a doc and then you can just add as you wish

[ilyakatz]

@gabrieljoelc @jondot so what is the current way to get this to work?

[gabrieljoelc]

@ilyakatz I did find a blog post with someone that got it working. They call the ActiveJob::Base.execute() method inside a "primary" queue/worker:

class PrimaryWorker
  include Sneakers::Worker
  from_queue :primary

  def work(msg)
    job_data = ActiveSupport::JSON.decode(msg)
    ActiveJob::Base.execute job_data
    ack!
  end
end

and then enqueue messages against your ActiveJob subtype:

MyActiveJobSubTypeThatDoesStuffForMyUsers.perform_later()

the author finishes with handling errors and retries.

We don't need to add anything specific to the library. We'd love contributions to the wiki (for example). I'll leave this open until that's done.

[sharshenov]

This adapter handles it https://github.com/veeqo/advanced-sneakers-activejob

[mengu]

hi all,

just wanted to add my 2 cents. we have a heavy workload and currently using sidekiq. i wanted to evaluate how rabbitmq & sneakers would help us there and i'm kind of happy with the overall result. there are three pain points for us here:

• sneakers doesn't respect activejob's queue_as:
  • sneakers doesn't create the queues and bind them to the configured exchange if the queue is not default.
  • sneakers publishes the messages when perform_later is called.
  • sneakers doesn't listen the queue for messages so they are not consumed.
• sneakers doesn't support delayed jobs. we upload some private reports to our cdn to be deleted in 7 days, for example.
• there's no way to route the messages as we can with sneakers workers.

i think sneakers should only handle the first point as i think its rabbitmq's job to support delayed queues. also i don't think rails devs would let us pass around exchange options or routing options.

we don't have any of these problems if we just don't use ActiveJob and adapt Sneakers::Worker.

p.s. this is not a complaint, just a reference point for people who'd like to evaluate sneakers & activejob in the future.

[sharshenov]

@mengu have you tried advanced sneakers adapter for activejob? It covers all problems you've listed

[bobf]

Very early days but can I vouch that @sharshenov 's gem has got me off the ground with ActiveJob - I was a bit stuck with the standard Sneakers backend. I'll update my comment if I run into any issues. Thanks for the time & effort, @sharshenov .