Concurrent Model Execution

[calycekr]

Feature request

Allow multiple models and/or multiple instances of the same model to execute in parallel on the same system.

Motivation

Maximize the use of GPU resources at idle to improve performance in multi GPUs system.

Your contribution

None

Comment

Is there already a way to improve performance by loading the same model on each GPU?

[Narsil]

You can shard the model using --num-shard 2 to use both GPUs for the same model, leading to better latencies.

Loading the same model with different copies is not and will not be done by this crate, you can simply launch 2 copies of tgi directly with CUDA_VISIBLE_DEVICES

[comaniac]

I have the same requirement and thus found this issue. Although sharding a model could make use of multiple GPUs, it may not be the most efficient approach especially when laking of NVLink.

I have tried to launch multiple TGI endpoints and used a Flask server to dispatch requests, but as you can imagine, Flask is not as efficient as Rust, so even this solution worked well, the scalability becomes poor when traffic is high. If supporting multiple instances is not on the TGI roadmap, would you please recommend a better dispatcher approach/framework that we could build on top of it? Thanks.

[Narsil]

Just use a better LB.

There are plenty... ngrok comes to mind.

We have no plan yet to implement any such logic, and if we did it would be to do something which other LB cannot do.

[OlivierDehaene]

You could use NGINX to load balance between the replicas.

[comaniac]

Thanks for the suggestion. I'll give a shot and probably setup a docker-compose to make it easy to use.

[thohag]

While running multiple instances of tgi in the same environment I get this error:
thread 'tokio-runtime-worker' panicked at 'ID not found in entries. This is a bug.', router/src/infer.rs:487:14
Using tgi version 1.1.1

[Narsil]

Could you open an new issue with all the asked information, and maybe try and provide a reproducer script/bash commands ?

If you are launching outside of docker you need to specify the file used for communication between pytorch processes. https://huggingface.co/docs/text-generation-inference/basic_tutorials/launcher#shardudspath

Maybe this one too: https://huggingface.co/docs/text-generation-inference/basic_tutorials/launcher#masterport

[thohag]

Thank you for the suggestion.
I was launching 4 instances of tgi inside a single docker container.

Setting --shard-uds-path to unique paths for each instance solved my problem.
In example:
--shard-uds-path /tmp/text-generation-server-0
--shard-uds-path /tmp/text-generation-server-1
--shard-uds-path /tmp/text-generation-server-2
--shard-uds-path /tmp/text-generation-server-3

Setting --master-port to unique values had no effect.

Before finding this fix I observed that 2 instances would run fine but 3 or more would fail at increasing rates.