[Feature] [Serve] Threading for Ray Serve

[cin-duke]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Currently, Ray Serve only supports asyncio which will block the event loop when there are some computation heavy tasks. Which is not ideal for handling concurrent requests.
It'd be great if we can support threading for Ray Serve deployment. It will be easier to process concurrent requests.

Use case

Allow Ray Serve deployment to process some computation heavy tasks while waiting for results on other threads.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[cin-duke]

Hi @simon-mo, could you take a look at this feature. Thank you 😃

[simon-mo]

Thanks for posting this @cin-duke. cc @jiaodong @edoakes from the Serve team.

Actually after some thoughts, this is more nuanced than I thought.

When you want concurrent requests but it is CPU bounded, for example a single call using 100% CPU, you should use replicas instead of threading to ensure that each process. Replicas are easier to manage and performance is more predictable.

When you have a call that only uses say 20% CPU but it is blocking call (no async option), threading might still make sense. However in this case replicas are still preferred because you can do YourDeployment.options(ray_actor_options={"num_cpus": 0.2}, num_replicas=10) to scale out.

The only case where threading would be useful is to use lower number of replicas due to process overhead. In this case you can use a Python threadpool and orchestrate it with asyncio. https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor

In summary, I think supporting threaded actor in max_concurrent_quries might be challenging. Let me know whether this make sense!

[cin-duke]

Thanks for your detailed explanation @simon-mo

Here is some of my comments:

When you have a call that only uses say 20% CPU but it is blocking call (no async option), threading might still make sense. However in this case replicas are still preferred because you can do YourDeployment.options(ray_actor_options={"num_cpus": 0.2}, num_replicas=10) to scale out.

In this usecase, thread would be useful because spawning multiple workers will take lots of system RAM, and if the deployment uses GPU RA, it will be very hard to scale out.

The only case where threading would be useful is to use lower number of replicas due to process overhead. In this case you can use a Python threadpool and orchestrate it with asyncio. https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor

I will look into this approach. However, it'd be easier for users if Ray implement the threadpool internally. User only needs to specify the argument to turn on thread, and set max concurrency.

[simon-mo]

In this use case, thread would be useful because spawning multiple workers will take lots of system RAM, and if the deployment uses GPU RA, it will be very hard to scale out.

This is a great point! For the thread pool approach, if you can try to prototype it in your application and see it working, we can integrate it into Ray Serve :D.

[cin-duke]

Okay, we will experiment with it and let you know the result.

[jiaodong]

Hi @cin-duke just revisiting this issue as this topic was brought up by other community users as well. Any updates on your experiment or help needed ?

[cin-duke]

Hi @jiaodong, @simon-mo
Sorry for late response, I don't have time to investigate it. My simple solution is: creating a function to call the deployment, then create a thread pool to run that function on separated thread.