TensorRT-LLM gets stuck with trtllm-serve

[mpaulazamin]

My system information:

I'm trying to serve Qwen2.5-3B-Instrut with tensor parallelism 4 and trtllm-serve, but the process gets stuck:

trtllm-serve Qwen2.5-3B-Instruct --tp_size 4
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
2025-07-02 03:34:03,877 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/home/shareduser/miniconda3/envs/mz-test-tensorrt-llm-llama/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[07/02/2025-03:34:04] [TRT-LLM] [W] Overriding LlmArgs.max_input_len (annotation=int required=False default=1024 description='The maximum input length.') with build_config.max_input_len (1024).
[07/02/2025-03:34:04] [TRT-LLM] [W] Overriding LlmArgs.max_seq_len (annotation=Union[int, NoneType] required=False default=None description='The maximum sequence length.') with build_config.max_seq_len (None).
[07/02/2025-03:34:04] [TRT-LLM] [W] Overriding LlmArgs.max_beam_width (annotation=int required=False default=1 description='The maximum beam width.') with build_config.max_beam_width (1).
[07/02/2025-03:34:04] [TRT-LLM] [I] Compute capability: (8, 9)
[07/02/2025-03:34:04] [TRT-LLM] [I] SM count: 142
[07/02/2025-03:34:04] [TRT-LLM] [I] SM clock: 3105 MHz
[07/02/2025-03:34:04] [TRT-LLM] [I] int4 TFLOPS: 902
[07/02/2025-03:34:04] [TRT-LLM] [I] int8 TFLOPS: 451
[07/02/2025-03:34:04] [TRT-LLM] [I] fp8 TFLOPS: 451
[07/02/2025-03:34:04] [TRT-LLM] [I] float16 TFLOPS: 225
[07/02/2025-03:34:04] [TRT-LLM] [I] bfloat16 TFLOPS: 225
[07/02/2025-03:34:04] [TRT-LLM] [I] float32 TFLOPS: 112
[07/02/2025-03:34:04] [TRT-LLM] [I] Total Memory: 47 GiB
[07/02/2025-03:34:04] [TRT-LLM] [I] Memory clock: 10001 MHz
[07/02/2025-03:34:04] [TRT-LLM] [I] Memory bus width: 384
[07/02/2025-03:34:04] [TRT-LLM] [I] Memory bandwidth: 960 GB/s
[07/02/2025-03:34:04] [TRT-LLM] [I] PCIe speed: 2500 Mbps
[07/02/2025-03:34:04] [TRT-LLM] [I] PCIe link width: 16
[07/02/2025-03:34:04] [TRT-LLM] [I] PCIe bandwidth: 5 GB/s
[07/02/2025-03:34:04] [TRT-LLM] [I] Specified dtype 'auto'; inferred dtype 'bfloat16'.
[07/02/2025-03:34:04] [TRT-LLM] [W] Implicitly setting QWenConfig.seq_length = 8192
[07/02/2025-03:34:04] [TRT-LLM] [W] Implicitly setting QWenConfig.qwen_type = qwen2
[07/02/2025-03:34:04] [TRT-LLM] [W] Implicitly setting QWenConfig.moe_intermediate_size = 0
[07/02/2025-03:34:04] [TRT-LLM] [W] Implicitly setting QWenConfig.moe_shared_expert_intermediate_size = 0
[07/02/2025-03:34:04] [TRT-LLM] [W] Implicitly setting QWenConfig.tie_word_embeddings = True
[07/02/2025-03:34:04] [TRT-LLM] [I] Set nccl_plugin to None.
[07/02/2025-03:34:04] [TRT-LLM] [I] start MpiSession with 4 workers
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1296: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package optimum. Picked distribution: optimum
Multiple distributions found for package optimum. Picked distribution: optimum
2025-07-02 03:34:10,672 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
2025-07-02 03:34:10,702 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/home/shareduser/miniconda3/envs/mz-test-tensorrt-llm-llama/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
2025-07-02 03:34:10,714 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
2025-07-02 03:34:10,729 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/home/shareduser/miniconda3/envs/mz-test-tensorrt-llm-llama/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/home/shareduser/miniconda3/envs/mz-test-tensorrt-llm-llama/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
/home/shareduser/miniconda3/envs/mz-test-tensorrt-llm-llama/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[TensorRT-LLM] TensorRT-LLM version: 0.20.0
[TensorRT-LLM] TensorRT-LLM version: 0.20.0

I tried with different models, and it's always stuck on that part. Yesterday it was working fine.

[SimengLiu-nv]

Could be the same issue as #5665 .
@mpaulazamin Can you export TLLM_LLM_ENABLE_DEBUG=1, it should dump more logs and help investigate.

[rnik12]

With TLLM_LLM_ENABLE_DEBUG=1

Python 3.12.11 (main, Aug 14 2025, 16:21:23) [Clang 20.1.4 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from tensorrt_llm import LLM, SamplingParams
Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

Authorization required, but no authorization protocol specified

[2025-08-23 15:27:24] INFO config.py:54: PyTorch version 2.7.1+cu128 available.
/root/.pyenv/versions/3.12.11/lib/python3.12/site-packages/modelopt/torch/__init__.py:36: UserWarning: transformers version 4.55.0 is incompatible with nvidia-modelopt and may cause issues. Please install recommended version with `pip install nvidia-modelopt[hf]` if working with HF models.
  _warnings.warn(
LLM debug mode enabled.
2025-08-23 15:27:27,254 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/root/.pyenv/versions/3.12.11/lib/python3.12/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  warnings.warn(
[08/23/2025-15:27:27] [TRT-LLM] [I] Starting TensorRT-LLM init.
[08/23/2025-15:27:27] [TRT-LLM] [I] TensorRT-LLM inited.
[TensorRT-LLM] TensorRT-LLM version: 1.1.0rc1
>>> 
>>> llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")
[08/23/2025-15:27:35] [TRT-LLM] [W] Using default gpus_per_node: 1
[08/23/2025-15:27:35] [TRT-LLM] [I] Set nccl_plugin to None.
[08/23/2025-15:27:35] [TRT-LLM] [I] neither checkpoint_format nor checkpoint_loader were provided, checkpoint_format will be set to HF.
LLM.args.mpi_session: None

[08/23/2025-15:27:36] [TRT-LLM] [I] PyTorchConfig(extra_resource_managers={}, use_cuda_graph=True, cuda_graph_batch_sizes=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 64, 128], cuda_graph_max_batch_size=128, cuda_graph_padding_enabled=False, disable_overlap_scheduler=False, moe_max_num_tokens=None, moe_load_balancer=None, attention_dp_enable_balance=False, attention_dp_time_out_iters=50, attention_dp_batching_wait_iters=10, batch_wait_timeout_ms=0, attn_backend='TRTLLM', moe_backend='CUTLASS', moe_disable_finalize_fusion=False, enable_mixed_sampler=False, sampler_type=<SamplerType.auto: 'auto'>, kv_cache_dtype='auto', mamba_ssm_cache_dtype='auto', enable_iter_perf_stats=False, enable_iter_req_stats=False, print_iter_log=False, torch_compile_enabled=False, torch_compile_fullgraph=True, torch_compile_inductor_enabled=False, torch_compile_piecewise_cuda_graph=False, torch_compile_piecewise_cuda_graph_num_tokens=None, torch_compile_enable_userbuffers=True, torch_compile_max_num_streams=1, enable_autotuner=True, enable_layerwise_nvtx_marker=False, load_format=<LoadFormat.AUTO: 0>, enable_min_latency=False, allreduce_strategy='AUTO', stream_interval=1, force_dynamic_quantization=False, _limit_torch_cuda_mem_fraction=True)
create pool session ...
rank 0 using MpiPoolSession to spawn MPI processes
Server [proxy_request_queue] bound to tcp://127.0.0.1:34729 in PAIR
[08/23/2025-15:27:36] [TRT-LLM] [I] Generating a new HMAC key for server proxy_request_queue
Server [worker_init_status_queue] bound to tcp://127.0.0.1:41747 in PAIR
[08/23/2025-15:27:36] [TRT-LLM] [I] Generating a new HMAC key for server worker_init_status_queue
Server [proxy_result_queue] bound to tcp://127.0.0.1:43893 in PAIR
[08/23/2025-15:27:36] [TRT-LLM] [I] Generating a new HMAC key for server proxy_result_queue
Server [proxy_stats_queue] bound to tcp://127.0.0.1:41601 in PAIR
[08/23/2025-15:27:36] [TRT-LLM] [I] Generating a new HMAC key for server proxy_stats_queue
Server [proxy_kv_cache_events_queue] bound to tcp://127.0.0.1:42257 in PAIR
[08/23/2025-15:27:36] [TRT-LLM] [I] Generating a new HMAC key for server proxy_kv_cache_events_queue

[karljang]

In the #5665 issue, someone reported that setting the OMPI settings in this comment resolved the hang problem.
Here’s the link to the comment: #5665 (comment)

I’m closing this issue due to its prolonged inactivity. If the problem persists in the latest release, please feel free to open a new issue.

Thank you!