Dynamic scale down #46

Open
wants to merge 6 commits into
from

Projects

None yet

3 participants

@davidakachaos

This is a feature I've been thinking about. Can you take a look at it @lostboy ? Just to see if you would want this feature.

davidakachaos added some commits Feb 27, 2013
@davidakachaos davidakachaos Adding Coveralls support
This is to see the spec/test coverage
cbc8304
@davidakachaos davidakachaos SimpleConv to check coverage f131d0f
@davidakachaos davidakachaos Merge branch 'mongoid_fix'
* mongoid_fix:
  Mongoid fix
892a9a9
@davidakachaos davidakachaos Dynamic downscale
Downscale the workers when they are no longer needed.
This will increase the hits on the heroku api, but let just if that's a real problem?
facd2a3
@davidakachaos davidakachaos Fixed specs 41025f6
@davidakachaos davidakachaos This works, so remove the pending statement. 2ffc67b
@lostboy
Owner

Hi David, 6 months is a while to wait for feedback, sorry about that. Is this a feature you're still interested in and can you expand on its purpose?

@davidakachaos

Well, let me first explain what 'issue' I'm trying to solve here.

Let's say we are running a cron job every hour that checks if there are file to be imported from a ftp server. When there are such files, for each file a Delayed Job is created to import the file. Now when a file is imported, sometimes 1 or more emails are send out by the system or another task needs to be done like export a small file to another server or things like that. Of course, these are all Delayed Jobs.

So no worries, we have workless and we can scale up when we need to right; for argument sake let's say the max is 500 workers (a lot! I know right!) but the min is 0.

So we queue up the work, and the jobs fill up the ratio quickly, so workless scales the workers up and up. Great! But now the following happens; 250 workers are up, and the queue is being emptied fast. Pretty soon the queue of jobs is 'empty'. That is to say; 2 or 3 files take a long time to be imported. So 3 workers are busy, while 247 workers are hanging around, playing poker or what not...

This PR is trying to solve this problem by scaling down the workers after a job is done. So when no more than 3 workers are needed, 247 workers could be fired....

The thing I don't know, if this is still needed as of the present version of the gem. Could you share your thoughts on this as well?

@jkotchoff

I think something like this would be nice. I just tried workless 1.2.3 and experienced the scenario david mentioned
ie.
1/ Job A and Job B queued in Delayed Job
2/ Workless kicks off a worker for each job (2 workers)
3/ Job A finishes very quickly, Job B takes a lot longer

  • instead of scaling the workers from 2 back to 1 after Job A finishes, workless is waiting until both jobs are complete before scaling down (back to 0) thereby consuming more worker time then is neccessary.

caveat: We are using david's dynamic_scale_down fork in production however I have noted that sometimes our workers seem to 'hang' with this fork ie. there is a job which isn't doing anything but thinks it is running with an associated idled worker.

I haven't been able to diagnose why yet. Possibly because post_ps_scale always kills the last worker that was created when sometimes it needs to be killing a different worker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment