unable to launch model with tensorrt_llm

[janpetrov]

System Info

Built tensorrtllm_backend from source using dockerfile/Dockerfile.trt_llm_backend
version 0.13.0.dev2024081300 (the same issue an earlier 0.12.0dev....)
Driver Version: 545.23.08 CUDA Version: 12.3 (the same issue for Driver Version: 555.42.06 and CUDA Version: 12.5)
H100-PCI (the same issue for H100-SXM)

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

We compiled and attempted to launchvarious 7b / 8b models, but when starting the triton server we always get:

[TensorRT-LLM][WARNING] gpu_device_ids is not specified, will be automatically set
I0821 08:15:32.609876 2817 libtensorrtllm.cc:114] "TRITONBACKEND_ModelInitialize: tensorrt_llm (version 1)"
Floating point exception (core dumped)
root@honzap-ct121:/mnt/ng/triton# I0821 08:15:33.549090 2830 pb_stub.cc:2049] Non-graceful termination detected.
I0821 08:15:33.559462 2831 pb_stub.cc:2049] Non-graceful termination detected.

Expected behavior

the triton server would load and start to serve

actual behavior

I0821 08:15:32.609876 2817 libtensorrtllm.cc:114] "TRITONBACKEND_ModelInitialize: tensorrt_llm (version 1)"
Floating point exception (core dumped)
root@honzap-ct121:/mnt/ng/triton# I0821 08:15:33.549090 2830 pb_stub.cc:2049] Non-graceful termination detected.
I0821 08:15:33.559462 2831 pb_stub.cc:2049] Non-graceful termination detected.

additional notes

everything ran fine up to the tagged version 0.11.0

The issue seems to be specific for tensorrt llm backend as the following script that loads the very same triton model runs fine

from tensorrt_llm.runtime import ModelRunner, SamplingConfig
from transformers import AutoTokenizer

tokenizer_path = "{SOME_PATH}"
dir_path = "{SOME_PATH}"

tokenizer = AutoTokenizer.from_pretrained(tokenizer_path)
runner = ModelRunner.from_dir(dir_path)
sc = SamplingConfig(pad_id=tokenizer.eos_token_id, end_id=tokenizer.eos_token_id)

tokenized = [tokenizer("Hello, my dog is cute", return_tensors="pt")["input_ids"][0]]
res = runner.generate(batch_input_ids=tokenized, sampling_config=sc)
print(tokenizer.decode(res[0][0], skip_special_tokens=True))

[thanhlt998]

@janpetrov I faced the same problem mpirun noticed that process rank 0 with PID 0 on node 769e1c33621a exited on signal 8 (Floating point exception) while try the fixed at PR.

[janpetrov]

@thanhlt998

Thank you. Our model runs on the single GPU (just a single process). We experience the issue both when we run via mpi or when we start the triton server directly.

Please, could you comment on "while try the fixed at NVIDIA/TensorRT-LLM#2130"? Do you mean that you encountered the error when using this PR or do you suggest that the error is already corrected in the PR?

[thanhlt998]

@janpetrov I tried the version before the fixed PR that used triton 24.05, it can launch model normally. After the fixed PR, it changes into triton 24.07 but I can run the engines inside the container environment successfully. Maybe the triton version 24.07 cause this problem?

[janpetrov]

@thanhlt998 thank you, this sounds reasonable, we see the same pattern that the error occurs with triton version 24.07

[thanhlt998]

@janpetrov I just try to downgrade triton 24.07 -> 24.05 and rebuilt the docker image. But the same error is raised when I launch the model. So the cause may not be from the triton version.

[jihyeonl-nvda]

Same error of mpirun noticed that process rank 0 with PID 0 on node dgxh100 exited on signal 8 (Floating point exception). with triton 24.07.

[skyCreateXian]

I encountered the same problem with the code branch 8aaf89b

I solved the problem by modifying tensortllm/config. pbtxt gpu_device_ids to 0

[janpetrov]

I encountered the same problem with the code branch 8aaf89b

I solved the problem by modifying tensortllm/config. pbtxt gpu_device_ids to 0

Thank you! Works for us.