Feature request: support multiple model instances on TensorRT LLM triton backend.

[wengsnow]

I used Baichuan2 13B model weight only int 8 and launch a triton server on single GPU. Now I have a node has 2 GPUs and want to multiple model instances to a instance for each GPUs to improve concurrent request throughput .

For the default config: "kind_cpu count=1", the gpu usage as follows:

Just like fastertransformer, I set "kind_cpu count=2" (https://github.com/triton-inference-server/fastertransformer_backend/blob/c6936c39be6fc57c161db34cfe5c45d1bef81ee2/README.md?plain=1#L48) But it can not work, I get such error:

When this error happened, the first GPU memory was increased, the second GPU memory was not changed.

When I follow the triton guide to set "kind_gpu count=1", get the same error.
(https://github.com/triton-inference-server/server/blob/0f1f9246ee8257db6196da65950205be79379401/docs/user_guide/model_configuration.md?plain=1#L600)

I want to know how to use Multiple Model Instances worked for tensorrtllm, which is a important feature for triton.

[byshiue]

TensorRT-LLM does not support multiple model instances now. I have changed the title to request to support this feature.

[wengsnow]

Thanks for your reply. Could you please tell me how to improve concurrent request throughput, when I have a node with two A100 40G GPUs. As shown in the figure 1, only the first GPUs be used for inference, I hope both two GPUs can work.

[byshiue]

Thanks for your reply. Could you please tell me how to improve concurrent request throughput, when I have a node with two A100 40G GPUs. As shown in the figure 1, only the first GPUs be used for inference, I hope both two GPUs can work.

If you only hope to run one instance per GPU, you can run

CUDA_VISIBLE_DEVICES=0 <launch server>
CUDA_VISIBLE_DEVICES=1 <launch server>

to run two server on two GPUs. Note that you need to adjust the port to prevent using same port.

[wengsnow]

Thanks for your reply. Could you please tell me how to improve concurrent request throughput, when I have a node with two A100 40G GPUs. As shown in the figure 1, only the first GPUs be used for inference, I hope both two GPUs can work.

If you only hope to run one instance per GPU, you can run

CUDA_VISIBLE_DEVICES=0 <launch server>
CUDA_VISIBLE_DEVICES=1 <launch server>

to run two server on two GPUs. Note that you need to adjust the port to prevent using same port.

This method will cause two different urls to inference, because of different ports. Has any method can use one url to query these two server?

[byshiue]

The answer is no at the moment.

[konodyuk]

@byshiue @ncomly-nvidia
Could you provide an estimated timeline for implementing multi-instance support? This is a crucial concern for those transitioning from the Fastertransformer backend to TensorRT-LLM.

[sadreamer]

Same here. I would also kindly ask for an estimation of when multi-instance will be supported. It's indeed very important for some use cases.
NB in the FT backend there was a similar issue with multi-instance, though: triton-inference-server/fastertransformer_backend#63