When creating many jobs in a transaction, workless scales up workers before transaction is complete #34

Closed
rhomeister opened this Issue Dec 4, 2012 · 7 comments

Comments

Projects
None yet
3 participants

Hi,

I've got a very long and complex transaction, which creates a large number of Delayed Jobs. Workless seems to scale up the number of workers to the maximum number before the transaction is complete. Sometimes, this transaction takes a full hour to complete (it needs to fetch several images, etc.). In such cases, I've got 10 workers idling for a full hour each time.

Wouldn't it be better to wait until there are actual jobs in the database before starting workers?

Thanks

Collaborator

davidakachaos commented Dec 7, 2012

Maybe Workless can switch to a after commit hook instead of a after create hook?

davidakachaos added a commit to davidakachaos/workless that referenced this issue Dec 7, 2012

Scale the workers after_commit
This will scale the workers after committing the changes
to the database. This will fix issue #34.

Main point of this is to relax the scaling to happen when
the jobs are in the database. This may also have an effect 
on issue #33.
Collaborator

davidakachaos commented Dec 8, 2012

Could you test my adjustments in my fork to review the changes I made?

Hi David,

It seems to work. Thanks!

In detail: I tried the following code:

Delayed::Job.transaction do 200.times {"a".delay.capitalize} sleep 120 end

With the current master branch, workless starts the maximum number of workers after the second line has completed. With your modifications, the workers (correctly) start after the transaction is complete. (I tested this on Heroku Cedar)

One more problem that I noticed after running this code is the following. After the transaction commits, a query is executed for every single created Delayed::Job. For example, for the snippet above, I got the following console output 200 times:

Delayed::Backend::ActiveRecord::Job Load (6.3ms) SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE "delayed_jobs"."failed_at" IS NULL

This is horribly slow (much slower than the 6ms reported here). The control does not return from this snippet until these queries have finished.

Any idea what causes this?

Ruben

Collaborator

davidakachaos commented Dec 8, 2012

Hi Ruben,

I suspect this has to do with the fact the all the queries are 'on hold' till the transaction commits.... This of cource is caused because of the 'after commit' hooks I changed it to... Now to solve this, I need to dive a bit deeper into active record and see if I can prevent the query being executed 200 times... Needs a bit more work. Maybe @lostboy has an idea on howto fix this... But I'll try to find a solution to this. If you can find one @rhomeister that would be great of course =D

Collaborator

davidakachaos commented Dec 8, 2012

Quick question, any chance you can rewrite the code you're using to make the commits happen more often (just reffering to http://stackoverflow.com/questions/9510586/when-should-an-activerecord-transaction-be-used and http://stackoverflow.com/questions/2509320/saving-multiple-objects-in-a-single-call-in-rails) as it seems to me, a transactions that runs over an hour (!!!) is a bit extreme?

The operation I'm performing has transaction semantics in principle, but
these could be relaxed a little. A rewrite would therefore be possible, but
not desired. The problem is that the transaction is slow only in production
on Heroku, because of the aforementioned very slow queries. As a result, it
is much faster in development (order of magnitude easily).

I must add that the situation is bearable with your current fix. I simply
have a worker operational for an hour now, instead of 11 workers.

On 8 December 2012 23:46, David Westerink notifications@github.com wrote:

Quick question, any chance you can rewrite the code you're using to make
the commits happen more often (just reffering to
http://stackoverflow.com/questions/9510586/when-should-an-activerecord-transaction-be-usedand
http://stackoverflow.com/questions/2509320/saving-multiple-objects-in-a-single-call-in-rails)
as it seems to me, a transactions that runs over an hour (!!!) is a bit
extreme?


Reply to this email directly or view it on GitHubhttps://github.com/lostboy/workless/issues/34#issuecomment-11164735.

Owner

lostboy commented Aug 19, 2013

Closing due to inactivity and a merged PR

@lostboy lostboy closed this Aug 19, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment