Pop multiple jobs from a queue at once?

[mfn]

I was wondering, how would you solve the following:

• you push a lot of jobs to a queue, assume each job contains very primitive payloads like just an ID of something
• having n workers on the queue to process them in parallel works
  but
  what I have a more efficient way by collecting/aggregating those IDs and performing a certain "in batch" on the server side
• note that I do not know nor care about the actual jobs/IDs being generated upfront, that is: I cannot use chained or batched jobs

In my particular case I would need something like:

• many different parts of the system push jobs to a queue, independently
• instead of a worker polling one job off the queue
• I want him to poll of n jobs (e.g. 8, 16) at once
  • so within the job (?) I can aggregate the IDs and perform efficient processing them (SQL WHERE IN or some remote http batch processing API, etc.)

I do understand this is not the same anymore as a job, which does one thing and starts over with the next payload from the queue.

Any idea how this could be made work with the current Laravel queue system? Or at least with minimal changes maybe?

Thank you :)

[henzeb]

You should look into Laravel Horizon.

[mfn]

How so?

I've used Laravel Horizon, but it's a different approach how to manage workers (instead of supervisor directly managing them), it's not concerned with how to handle individual jobs and queues?

[Radiergummi]

@mfn Hi Markus! Did you find a good way to deal with this? I have the same use case basically, which is collecting tasks to dispatch to an external API that has a batch processing endpoint - so waiting until I have kind of a "critical mass"/sufficient pressure before dispatching the request would greatly improve performance. From what I know about Laravel's queue system, this isn't really possible still.

ps: we were briefly working together in a previous life when I was building the unified messaging API for MessengerPeople with Max. Hi! :)

[mfn]

Hi 😅

Did you find a good way to deal with this?

Not with any Laravel/3rd package, no.

We implemented our own "daemon" concept, closely based on the worker concept of Laravel: not as generic, but purposefully written for the one task we needed it (indexing to ElasticSearch). The custom use of blpop and lpop to fetch indexable IDs from Redis was much more efficient than anything else.

Even today looking at https://laravel.com/docs/11.x/queues#job-batching , my understanding is that it's something else.

(I try to remember why I wrote this message here in 2022: that daemon I'm talking about was written in 2017 and still runs 😆 I guess I wanted to find out if there's something off-the-shelve. I guess we know that answer now 😅)