Limit Number of PHP processes

[codivist]

It is possible to limit the number of php processes that get spawned?

I'm using this package to copy huge amounts of data after an upgrade takes place of my web app, and I don't mind if it takes 1 hour or 24 hours or more. However if there are millions of rows of data that need to get copied, the queue spawns over 100 php processes and CPU usages sits at 100% until most of the queue jobs are down.
Using ->delay(30) still spawns the php processes, but just delays them running, so the CPU still gets pegged.

Is there a way to keep the number of php process spawned to 5 or any number I choose? i.e. ->limitProc(5)

[hopewise]

Did you have any success of this? also, how would I expect the time at which I can get the job finally executed? how would I know that it was just executed?

[codivist]

I was able to process just over 10 million records, but as far as I could find there was no way built into this package to limit the number of processes.
I was able to control how many php processes were spawned by creating a queue job that called the actual queue job that did the database copying. Seems redundant, but it worked for what I needed.

In the first job I look at total records that need to be copied, split it into chunks of about 5000 to 6000 (or whatever works best on your hardware), then I loop through each chunk. When one chunk is put on to the queue, I just use sleep(2); to have php wait for two seconds, before putting the next chunk on.
I found that 2 seconds was enough time to process a chunk of about 6000 database records, so as the current queue job was finishing another queue job was starting. I never had more than 4 or 5 php processes running at one time for this.

How long your total job takes really depends on how many records you need to copy. I was able to copy over a million records in less than two minutes. To process over 10 million took several hours, but only because about 3/4 of the way through it slowed way down for some unknown reason.
To find out how long to expect the job to finish executing, you would have to run a series of tests and take the average.

The queue_jobs table will show you which jobs are in line to be run, when they are off the queue they are done executing.

[barryvdh]

To be honest, if you're using this much jobs, don't use this library, bit use an actual queue system..

[codivist]

I would also recommend using an actual queue system, but the software we're writing is being used on a closed (no outside internet connection) secure system, windows, so accessing SQS or other solid queues are not a viable option.

I am looking at something like beanstalkd locally installed, but as of right now I've been told no.

This queue driver works awesome though, even with that many jobs 👍

[mauricioA0]

Hi

I was able to solve with rabbitmq.

I can sent 1.8 million process by hour is ec2 mini.