Does TRTIS support model parallelism?

[vilmara]

Hi all, does TRTIS support model parallelism?, I mean, if a single model is copied in several GPUs to maximize inference throughput. Thanks,

[deadeyegoodwin]

Yes, you can serve multiple different models, multiple instances of the same model, or multiple instances of multiple models, on one or more CPUs and GPUs, simultaneously.

The docs discuss it here: https://docs.nvidia.com/deeplearning/sdk/tensorrt-inference-server-master-branch-guide/docs/model_configuration.html?highlight=batching#instance-groups

As does this blog post (in the Performance section): https://devblogs.nvidia.com/nvidia-serves-deep-learning-inference/

[vilmara]

@deadeyegoodwin, thanks for your prompt reply

[BDHU]

@deadeyegoodwin for a large model that doesn't fit on a single GPU, how does triton split the model onto mutlple GPUs then?

[shanshanpt]

@deadeyegoodwin for a large model that doesn't fit on a single GPU, how does triton split the model onto mutlple GPUs then?

Triton may not support this case based on my experience.