[Fastpitch/Pytorch] Multi GPU inferencing in a single triton server issue

[Shashika007]

Related to Fastpitch/Pytorch(s)
(https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch)

Describe the bug

We are using FastPitch to generate the Melspectrogram for Thai Language. We have an inference server that occupies two T4 GPUs. Our approach is to use both GPUs in one triton server as follows,

When we run only one GPU per triton server, TTS model could generate Mel-Spectrogram without an issue. But once we use two GPUs per triton server we had to face the following issue.

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

To Reproduce
Steps to reproduce the behavior:

1. Follow the Fastpitch triton example to generate a triton model and the config.

2. Copy the model folder into specific location (example ~/models)

3. Edit the triton model configuration config.pbxt (two instance groups)
   instance_group[ { count: 2 kind: KIND_GPU gpus: [0] }, { count: 2 kind: KIND_GPU gpus: [1] } ]

4. Pull the triton server 21.05 and run the Triton server as follows with CUDA_VISIBLE_DEVICES=0,1 environment.
   docker run -it --rm --runtime=nvidia --env CUDA_VISIBLE_DEVICES=0,1 --gpus=all -v ~/models:/models -p 8000:8000 -p 8001:8001 -p 8002:8002 nvcr.io/nvidia/tritonserver:21.05-py3 tritonserver --model-repository=/models --strict-model-config=false

5. Modify and run a simple client from Client libraries (HTTP, GRPC) to infer the model.

Expected behavior

Runtime Error will show up in the triton client-side log as follows,

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Code that relates the issue

Environment

• Container version: nvcr.io/nvidia/tritonserver:21.05-py3 (Triton server image)
• GPUs in the system: 2x Tesla T4-16GB
• CUDA driver version: Driver Version: 515.65.01

[nwalker-nvidia]

I assume you're following this: https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch/triton

Please use only one CUDA device with Triton for the initial setup steps (I.e. convert model) .. and multi-gpu only when you run triton server and load model/instances.

for this step:
python3 triton/convert_model.py

do:

CUDA_VISIBLE_DEVICES=0 python3 triton/convert_model.py

then multi-GPU is fine with model-navigator and running the server

[Shashika007]

Thank you @nwalker-nvidia.

I have tried your approach but the result was the same.

I used CUDA_VISIBLE_DEVICES=0 in the model conversion and model config generation scripts as follow,

In model conversion,

In model config generation,

I used multi GPUs ( 0,1 ) when I ran the triton server.

But, the issue repeats as follow,

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

[Shashika007]

I tried to convert the model into torch scripted. And it works fine. Previously, the model converted into torch traced.