can't make autoscaler to work with multiple queues

[friendlycredit]

Hi,
I have a Heroku application with 3 Sidekiq workers on 3 different queues and I'm trying to make it work with this great gem.

My workers are configured to use the following queues: "enrich", "mass_enrich" and "import" (all given as strings and not symbols)

That's my sidekiq.rb conf file:

require 'sidekiq'
require 'autoscaler/sidekiq'
require 'autoscaler/heroku_scaler'

heroku = nil
if ENV['HEROKU_APP']
  heroku = {}
  scaleable = %w[mass_enrich enrich import] - (ENV['ALWAYS'] || '').split(' ')
  scaleable.each do |queue|
    heroku[queue] = Autoscaler::HerokuScaler.new(
      queue,
      ENV['HEROKU_API_KEY'],
      ENV['HEROKU_APP'])
  end
end

Sidekiq.configure_client do |config|
  if heroku
    config.client_middleware do |chain|
      chain.add Autoscaler::Sidekiq::Client, heroku
    end
  end
end

# define HEROKU_PROCESS in the Procfile:
#
#    default: env HEROKU_PROCESS=default bundle exec sidekiq -r ./background/boot.rb
#    import:  env HEROKU_PROCESS=import bundle exec sidekiq -q import -c 1 -r /background/boot.rb

Sidekiq.configure_server do |config|
  config.server_middleware do |chain|
    if heroku && ENV['HEROKU_PROCESS'] && heroku[ENV['HEROKU_PROCESS']]
      p "Setting up auto-scaledown"
      chain.add(Autoscaler::Sidekiq::Server, heroku[ENV['HEROKU_PROCESS']], 60, [ENV['HEROKU_PROCESS']])
    else
      p "Not scaleable"
    end
  end
end

and that's my Procfile:

web: bundle exec thin start -R config.ru -e $RAILS_ENV -p $PORT
mass_enrich: env HEROKU_PROCESS=mass_enrich bundle exec sidekiq -c 1 -q mass_enrich
enrich: env HEROKU_PROCESS=enrich bundle exec sidekiq -c 25 -q enrich
import: env HEROKU_PROCESS=import bundle exec sidekiq -c 1 -q import

My user can only send jobs to the mass_enrich queue, This worker in turn, after performing some filtering and normalization will send some enrich jobs (can be hundreds or thousands). The EnrichWorker will perform the enrichment and might send an import job.
I've separated to 3 queues because I wanted to ensure that mass_enrichment requests are not stuck behind a lot of enrichments, and that import will always happen at a slow pace without hurting enrichment capacity.

What happens is that I send the mass_enrichment job and autoscaler starts the mass_enrich process. It handles the job and I see that Sidekiq now has waiting jobs on the enrich queue but nothing happens then.
When I manually (via rails console) send a dummy enrich job, the enrich process begins and handles all of them but then the same happens to the import job.

Is there any problem with workers sending other workers? Any other approaches to solve my problem?

Many thanks, and if I can help in any way, please feel free to ask. I'm not a ruby/rails expert and don't know the middleware layer very well but will be happy to try.

Zach

[pboling]

I have a setup that is very similar. I am also unable to get autoscaler to start my workers, but it does shut them down just fine.

My Profile looks like this:

web: bundle exec rails server puma -p $PORT -e $RACK_ENV
critical: env HEROKU_PROCESS=critical bundle exec sidekiq -c 4 -q critical,1
default: env HEROKU_PROCESS=default bundle exec sidekiq -c 2 -q default,1

I have no idea what else to try. I'm considering switching back to HireFireApp.

[JustinLove]

@friendlycredit , I think the issue is that you have background jobs trigger jobs in a different worker - my application doesn't do this (my import jobs are triggered manually or by the scheduler) As a consquence, the example code doesn't configure scale-up on the background jobs. I believe that you need to add config.client_middleware to your Sidekiq.configure_server block.

[JustinLove]

@pboling can you open a separate issue to save us some confusion?

Please provide your sidekiq middleware configuration.

Another good place to start would be to try and get https://github.com/JustinLove/autoscaler_sample working, and make a fork with you configuration that demonstrates the problem.

[pboling]

@JustinLove My issue seems to be literally the same one @friendlycredit, so I am trying you're advice to him now. Our jobs schedule other jobs too.

[pboling]

@JustinLove Just to verify there wasn't a typo - you are saying add config.clientmiddleware to the Sidekiq.configureserver block?

I already have a config.servermiddleware block in my Sidekiq.configureserver block:

  config.server_middleware do |chain|
      chain.add(Autoscaler::Sidekiq::Server, heroku[ENV['HEROKU_PROCESS']], 120, [ENV['HEROKU_PROCESS']])
  end

Are you saying I need the config.client_middleware in addition to, or in place of the config.server_middleware I have there now, or was it a typo, and what I have is correct?

[pboling]

@JustinLove Here is my complete sidekiq initializer and Procfile.

[JustinLove]

Middleware and configuration are two separate things, that happen to often be entangled.

Configurations are alternates depending on how it's run. Server runs when you start the Sidekiq CLI, Client runs any other time. The configures are convenience methods to streamline setting the configuration for each case. Their execution is mutually exclusive - one has yield if server? and the other yield unless server?

Middleware exists in two chains; the chains are disjoint, but both can be configured and running in the same process. The client chain is executed when pushing a job, and the server chain is executed when running a job.

So: when running a worker, the configure_server block is run, the configure_client block is not. Any process that schedules jobs for a worker that is not itself needs a client_middleware with the autoscaler client installed.

If the duplication offends you, the version of Sidekiq I'm looking at passes Sidekiq as the config object, so you could set up the client_middleware outside of the configure blocks to have it run every time.

See also:

• client middleware missing from processors sidekiq/sidekiq#175
• Middleware documentation clarification sidekiq/sidekiq#545
• https://github.com/mperham/sidekiq/wiki/Middleware

@pboling Are you no longer using low? It's in your sidekiq config, but not in Procfile.

[pboling]

@JustinLove That is correct - I removed low from the Procfile. I guess that could be my problem? It'll take me awhile to digest your explanation. I'll take a look at those links. Thanks for the help!

[pboling]

@JustinLove does autoscaler scale workers when running foreman locally? Why, or why not? Is there anyway to test that it is capable of scaling up without redeploying each time I want to try a new tweak?

I am still unable to scale up, but down is working fine. I have updated my gist with the current code.

UPDATE: I've been watching my queues. I'm not sure whendefault started working, but I just deployed a switch to github master gem 'sidekiq', github: 'mperham/sidekiq' and I have seen my default scale up. So it is working partially. I am still having to manually start the critical queue. The low queue hasn't been empty yet so I can't tell if it is being managed. Also it seems as though the empty queues are no longer scaling down.

I think I may have something bad in my sidekiq commands in the Procfile, so reading up on that now.

[JustinLove]

Right now it only interacts with the Heroku API. The autoscaler_sample project has some notes on remote-controlling a heroku instance from your local machine for testing purposes. I made HerokuScaler a separate object, so in principal you could write a scaler that spawned and killed processes locally.

I'll look at the gist later.

I'll have to review this, but I believe that there is still some spooky action at a distance because only one active flag is being used, if your low queue is constantly cycling, it may be keeping the others up.

[JustinLove]

I made a sample configuration using your gist.

https://github.com/JustinLove/autoscaler_sample/tree/pboling

Things work fine in the basic case, so there are no fundamental configuration errors. I removed the final call to ActiveRecord since the sample has no database, and I didn't try running it with Puma.

I was able to prevent other processes from scaling down by hitting 'low' occasionally, so the theoretical entanglement has been actually seen. I was also seeing spurious scale-to-1 for the low process, so something is off there.

[JustinLove]

I just pushed 0.2.1 which should eliminate the known crosstalk issues.

There is also a StubScaler that can be used for local testing. The only thing it does is print a message, but you can check whether scaling is being triggered without a heroku puppet.

[pboling]

Great! I'll give the new version a shot, and report back. On the previous version I did also see strange random scale-to-1 messages when I could have sworn it was already scaled to 1, which, I think is the same as you mentioned.