How to limit concurrency with async?

[maximejobin]

I have a task to do which requires to loop thousands of time to launch thousands of async subtasks.

I'd like these subtasks to be launched with a limitation of 10 at the time. What is the best way to achieve that using the async library?

Thank you!

[WyriHaximus]

Have a look at https://github.com/clue/reactphp-mq that should be able to help you. Otherwise you, if you need something more persistent than in process memory, you could check out https://github.com/jakubkulhan/bunny/

[SimonFrings]

@maximejobin good question 👍

I agree with what @WyriHaximus said, just note that when using clue/reactphp-mq you'll queue your (thousands) operations in-memory and I'm not sure if this is something you want to do (the more operations, the more memory you're using).

You might also want to take a look at clue/reactphp-flux as this works similar to clue/reactphp-mq without keeping you operations inside your memory, so this might be a good fit for the amount of operations you have. Taken from the flux documentation:

It uses ReactPHP's standard streaming interfaces which allow to process huge inputs without having to store everything in memory at once and instead allows you to efficiently process its input in small chunks.

Note that clue/reactphp-flux uses a streaming approach which is a bit more complicated tho.

For the future, we're planning to combine clue/reactphp-mq and clue/reactphp-flux and merge them with our reactphp/async component. This will allow us to have a single project capable of handling both a few and a large number of operations, while keeping it user-friendly. Please note that these are currently just ideas, so I cannot provide an exact timeline for when we will be able to work on this.

[maximejobin]

Wow @WyriHaximus and @SimonFrings ! This is so much more information than what I expected.

Thank you very much for taking the time to write all this. This is exactly what I needed. More than 'good enough' for now!! I will keep an eye on the future combination in reactphp/async.

Have an awesome day!