Why use "sync" instead of "async" when serving ML models?

[Kludex]

Description

On the lecture of July 6th Sebastián was talking about FastAPI and ML models, and he said that is better to use sync notation instead of async when serving ML models. I didn't understand the motive, could someone explain it?

Thank you!

[rkbeatss]

He recommended the use of sync rather than async functions when serving ML models because most ML operations are CPU intensive and the program would benefit from being able to do more computations in parallel. This means that in most cases the time will be spent actually doing this work rather than waiting around, rendering theasync notation less useful in speeding up the program.

If I understood correctly, I think he suggested running several processes in parallel for CPU intensive ML models, which would allow to generate several predictions at the same time for different requests.

[Kludex]

Makes sense, thank you @rkbeatss

[phy25]

You can close the issue if you don't have other questions.

[tiangolo]

Thanks for the help here @rkbeatss and @phy25 ! 👏 🙇

Thanks for reporting back and closing the issue @Kludex 👍

---

It's mainly because:

Blocking (CPU-bound) code should not be called directly. For example, if a function performs a CPU-intensive calculation for 1 second, all concurrent asyncio Tasks and IO operations would be delayed by 1 second.

Ref: https://docs.python.org/3/library/asyncio-dev.html#running-blocking-code

By using normal def functions FastAPI runs them in a threadpool with loop.run_in_executor().

[Kludex]

Three years ago I asked this question because the operations were CPU bound, and running in a thread pool wasn't suppose to help...

The link on the Python documentation shows to use an executor, but doesn't say which kind.

The thing here is that the computations escape the GIL, and for that reason they can run in the thread executor. I didn't know about this at the time.

[dummyuser-123]

I have also made an ml model api with sync (def) functions. And it is will giving me the outputs in parallel. But I have getting an problem here. Like if I make single requests then it takes 3 second to give response but if I make 5 parallel requests then it is taking 15 seconds for each response. Here, you can see all the request are waiting for the completion rather than running separately. Means, response time increase as the number of requests are increases. So, can you give any advice regarding this problem to solve it.

I have also followed this article but didn't get any success. Instead of it all the request are running in sequential manner.