Dynamic scale down #2

Closed
wants to merge 6 commits into
from

Conversation

Projects
None yet
3 participants
@davidakachaos
Owner

davidakachaos commented Apr 11, 2013

Scale down the workers when no more are needed...

This may result in a lot of queries, but it reduces the costs of the workers on Heroku, when 1 long running proces keeps a lot of workers up...

davidakachaos added some commits Feb 27, 2013

Adding Coveralls support
This is to see the spec/test coverage
Merge branch 'mongoid_fix'
* mongoid_fix:
  Mongoid fix
Dynamic downscale
Downscale the workers when they are no longer needed.
This will increase the hits on the heroku api, but let just if that's a real problem?
@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Apr 11, 2013

Coverage remained the same when pulling 2ffc67b on dynamic_scale_down into 892a9a9 on master.

View Details

Coverage remained the same when pulling 2ffc67b on dynamic_scale_down into 892a9a9 on master.

View Details

@jkotchoff

This comment has been minimized.

Show comment
Hide comment
@jkotchoff

jkotchoff May 27, 2013

Hey, this is a really nice fork.

You may however, want to document in this fork/pull request that when setting WORKLESS_MAX_WORKERS to a value greater than one for heroku cedar, Delayed::Job needs to be configured with:

Delayed::Worker.raise_signal_exceptions = :term

otherwise, a race condition occurs whereby if you have multiple workers running and the first worker finishes but a second worker is still going, the call to ::Heroku::API.post_ps_scale is always incorrectly killing the last worker.

This results in a Delayed::Job record that still thinks it's locked to a worker process, however that worker process has been SIGKILL'ed by heroku.

Because this hanging job remains locked (even though it's not being processed), workless doesn't spin down the remaining worker (which isn't doing anything) and you get a race condition with a job that doesn't finish and a worker that doesn't spin down.

Hey, this is a really nice fork.

You may however, want to document in this fork/pull request that when setting WORKLESS_MAX_WORKERS to a value greater than one for heroku cedar, Delayed::Job needs to be configured with:

Delayed::Worker.raise_signal_exceptions = :term

otherwise, a race condition occurs whereby if you have multiple workers running and the first worker finishes but a second worker is still going, the call to ::Heroku::API.post_ps_scale is always incorrectly killing the last worker.

This results in a Delayed::Job record that still thinks it's locked to a worker process, however that worker process has been SIGKILL'ed by heroku.

Because this hanging job remains locked (even though it's not being processed), workless doesn't spin down the remaining worker (which isn't doing anything) and you get a race condition with a job that doesn't finish and a worker that doesn't spin down.

@davidakachaos

This comment has been minimized.

Show comment
Hide comment
@davidakachaos

davidakachaos May 27, 2013

Owner

Very nice catch! I didn't know about the 'raise_signal_exceptions' option! Maybe this can solve another issue we have on the lostboy branch right now that the workers aren't scaled down! Thanks again!

Owner

davidakachaos commented May 27, 2013

Very nice catch! I didn't know about the 'raise_signal_exceptions' option! Maybe this can solve another issue we have on the lostboy branch right now that the workers aren't scaled down! Thanks again!

@davidakachaos davidakachaos referenced this pull request in lostboy/workless May 27, 2013

Closed

Workless scaling up but not scaling down! #19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment