Horizontal scaling job concurrency issue

[jaredm4]

We wanted to test for the possibility that there's too many jobs for a single instance to handle, and add more instances to tackle more jobs. From initial tests, this is what looks like is happening:

• Both instance's job runners start and grab the same "pending" job row.
• Both set the state to "running", and continue to both run the job.

This is bad for obvious reasons but we also have our jobs create new jobs as needed after they complete. So this compounds the issue by adding new duplicate jobs at the end.

Can Innodb row-locking solve this issue? Has anyone else solved it? Scaling vertically is an option but only for so long. I foresee too many jobs for a single instance.

[schmittjoh]

Right now, this bundle only supports vertical scaling. For horizontal scaling, we would need some sort of locking/coordination between runners, or use some other scheme for distributing work like only running even job ids in one runner and odd ids in another runner.

[JeffreyDD]

But if the max concurrent jobs goes over 60 you're in trouble, cause the runner waits at least 1 second before starting the next job. After some modifications to this bundle, I'm currently able to run 200 processes simultaneously, but the machine load goes up way beyond 15. And no, the scripts are not causing that load. Running the executed command 300 times simultaneously from a simple bash shell causes a load of ~1.7, running the same tasks with a max simultaneous jobs set to 150 causes a load between 15 and 18. So, how are we gonna solve this? Even when the node is capable of running that many tasks at once, the bundle simply can't keep up, causing awful issues like tremendous high loads . . .

[schmittjoh]

This sounds like a separate issue, and you'll need to provide more information about where the time is spent.

[ghost]

@JeffreyDD can you provide a public blackfire.io profile so we can try to find the bottleneck (if any)?

[schmittjoh]

Closing as support for running multiple instances of jms-job-queue:run has been added a while ago.