Error when scaling down - Unable to create thread

[espen]

WARN: ThreadError: can't create Thread: Resource temporarily unavailable 2015-12-23T15:51:14.908914+00:00 app[worker.1]: 3 TID-owpwn1y0o WARN: /app/vendor/bundle/ruby/2.2.0/gems/autoscaler-0.11.0/lib/autoscaler/sidekiq/thread_server.rb:32:in initialize'``

Occurred when reaching the timeout value and trying to figure out whether to scale down. More jobs to process.

I am unable to replicate this now. I did some other config changes so restarted the dynos a few times. I also noticed that a similar error was logged from the Librato gem. Anyways just wanted to leave this here despite not having more info or unable to replicate at the moment.

[JustinLove]

I'd suspect a race condition between shutdown and a new job coming in. Perhaps it would be useful to explicitly catch that exception, though I'd like to understand it a bit better first.

[espen]

Just experienced this error in production now. Was a few hundred jobs to be processed. Restarted the worker and the same error occurred again after having successfully processed half of the jobs. After processing all jobs it now seems to be stable with new incoming jobs and processing them.

[pedrobachiega]

Maybe you are reaching the machine thread limits.
I did the below steps to see that I should use more workers:

1. first, see the machine threads capacity running at rails console from same size of your worker:

1.upto(10_000) do |i|
  Thread.new { sleep }
  puts i
end

Example: to heroku 1x worker, it reaches 254, so, if you have more than 250 jobs and only one worker, you should see the ThreadError: can't create Thread
2) second, I did another client with scaling strategy: #61

Maybe this could help.

[pedrobachiega]

After more hours on this problem, I discovered that the reason was the ThreadServer.
So, I changed the solution to #62

[JustinLove]

I think I've found the issue. Server middleware are run per-thread, with a unique instance per thread. So the thread management information isn't shared and it's creating (at least) one monitor per worker. If it's doing unique middleware per job it would be more.

[pedrobachiega]

Nice Justin!
I'm not an expert about threads, so I didn't figure out.
Dou you plan to make a fix?

[JustinLove]

Is anybody experiencing this reliably enough that you could say if the branch https://github.com/JustinLove/autoscaler/tree/single-thread is a solution?

All I've got is a locally modified autoscaler_sample that does 100 jobs per click.

[pedrobachiega]

Justin, I will try it on weekend.

[JustinLove]

No feedback, but I think I found a problem. I'll be releasing the above branch soon if there are no objections.

[JustinLove]

master updated to single-thread branch and released as 0.12.0