Spawning Async Tasks resulting in Runtime blocked

[manuelgdlvh]

Hi mates, i have a several questions about how internally runtime of tokio works.
To put a little bit in context i am using an event driven architecture using commands as operation unit and i am using isolated runtime to process commands but there is something that i am not understanding.

       runtime.block_on(async move {
            while let Some(task) = rx.recv().await {
                tokio::spawn(command_request_processor(task))
            }
        })

Spawning async tasks the runtime get collapsed very early even controlling the maximum number of concurrently tokio tasks spawns using semaphores.

Command requests by clients obtained from Redis PubSub and pushing tasks to command processor runtime shown before
Producing 30 ops / second

Command Request Receiver obtaining tasks through the previously shown runtime and generating tasks

Processing 5 tasks / second and getting blocked.

If I don't spawn Tokio tasks, it processes correctly, but tasks keep accumulating to be performed. The runtime doesn't get blocked, and it works correctly (but with zero concurrency).
The tasks processed internally are a mix of IO and CPU tasks but of low intensity.

What could be happening? Feel free to ask for any necessary information.

Thanks in advance!

[Darksonn]

Are you blocking the thread? That is, are any of your tasks spending a lot of time without reaching an .await?

To check this, you can use tokio-console. Blocking tasks show up as a busy duration that keeps going up, but the poll count does not change.

[manuelgdlvh]

Good evening from Spain. After a few days, I have finally identified the issue I was facing. In case someone else encounters it, it could be summarized as 'do not keep references to locks if there is any .await involved.' That was exactly what was happening to me. GET_LOCK AWAIT DROP_LOCK was generating a deadlock within the runtime. When we await, it yields(), and since we are holding a reference to a critical section, any thread executing that part will be blocked, leaving the runtime with all threads blocked.
Thanks for all, your answer opened my mind :)

The performance is not bad ;)
This code was rewritten from java (soon in production as mobile app) and the performance is around 2.5x (with an large margin of improvement).
Thanks for all again!