Repository navigation
CUDA cudaMemcpyAsync invalid argument error when using Triton instance_group multi-instance load balancing with TensorRT-LLM BLS model on RTX 5090 #15476
Replies: 2 comments 2 replies
|
A couple of things to separate here. First, the instance_group [
{ kind: KIND_CPU, count: 2 }
]
parameters: {
key: "gpu_device_ids"
value: { string_value: "0;1" }
}For a TP=2 model on four GPUs it would look like The part I would be suspicious of is the BLS + auxiliary GPU models + Triton load balancing combination. With Triton core load balancing, the BLS request does not necessarily control which TensorRT-LLM instance receives the request. If an auxiliary model produces tensors on GPU 0 and Triton routes the downstream TensorRT-LLM request to the instance on GPU 1, a later I would debug it in this order:
For production, I would lean toward two explicit pipelines or a leader/orchestrator setup with an external load balancer rather than letting one BLS model fan into an implicitly load-balanced TRT-LLM model, unless all tensors crossing that boundary are CPU materialized. So my short answers would be:
|
|
The failure pattern looks more like a device ownership / routing problem than a For BLS, the safest debugging step is to force a CPU boundary between auxiliary models and TensorRT-LLM. In the Python backend, convert auxiliary outputs to CPU / NumPy before passing them into the LLM request. If that fixes it, the issue is almost certainly cross-GPU tensor handoff rather than the engine itself. If you want to keep GPU tensors, make the routing explicit. Common pattern is to create per-GPU model names, for example: |
Uh oh!
There was an error while loading. Please reload this page.
Description
When deploying a TensorRT-LLM model with Triton's instance_group multi-instance load balancing on NVIDIA RTX 5090 GPUs, a CUDA runtime error occurs. The setup involves a BLS (Business Logic Scripting) model where the LLM (TensorRT-LLM) depends on results from auxiliary models (e.g., ASR encoder, adaptor, CTC, speaker model, etc.).
Environment
GPU: NVIDIA RTX 5090 (2x GPUs)
TensorRT-LLM: (1.0.0)
Triton Inference Server: (2.62)
CUDA: (13.1)
Model: Speech LLM with DP=2 TP=1
Python backend: BLS model orchestrating auxiliary models + TensorRT-LLM
Configuration
The tensorrt_llm model config (config.pbtxt) uses instance_group with KIND_CPU and gpu_device_ids to enable multi-instance load balancing, as described in the Triton documentation:
The BLS model (infer_bls) also has multi-instance configuration:
Auxiliary models (encoder, adaptor, etc.) are each assigned to specific GPUs via:
Error Message
Reproduction Steps
Questions
Workaround Attempts:
Single instance without instance_group multi-instance: works correctly
Setting CUDA_VISIBLE_DEVICES to restrict to a single GPU set: works but loses load balancing benefit
All reactions