baichuan2-13b exec error

[zhanglv0209]

root@GPU-26:/tensorrtllm_backend/tensorrtllm_backend/tensorrtllm_backend# CUDA_VISIBLE_DEVICES=0,3 python3 ./scripts/launch_triton_server.py --world_size=2 --model_repo=/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo
root@GPU-26:/tensorrtllm_backend/tensorrtllm_backend/tensorrtllm_backend# I1030 09:08:30.450984 1398 pinned_memory_manager.cc:241] Pinned memory pool is created at '0x7f3edc000000' with size 268435456
I1030 09:08:30.454704 1398 cuda_memory_manager.cc:107] CUDA memory pool is created on device 0 with size 67108864
I1030 09:08:30.454713 1398 cuda_memory_manager.cc:107] CUDA memory pool is created on device 1 with size 67108864
I1030 09:08:30.472889 1399 pinned_memory_manager.cc:241] Pinned memory pool is created at '0x7ff556000000' with size 268435456
I1030 09:08:30.492939 1399 cuda_memory_manager.cc:107] CUDA memory pool is created on device 0 with size 67108864
I1030 09:08:30.492958 1399 cuda_memory_manager.cc:107] CUDA memory pool is created on device 1 with size 67108864
I1030 09:08:30.796900 1398 model_lifecycle.cc:461] loading: tensorrt_llm:2
I1030 09:08:30.796944 1398 model_lifecycle.cc:461] loading: preprocessing:1
I1030 09:08:30.796963 1398 model_lifecycle.cc:461] loading: postprocessing:1
I1030 09:08:30.815125 1399 model_lifecycle.cc:461] loading: tensorrt_llm:2
I1030 09:08:30.815165 1399 model_lifecycle.cc:461] loading: preprocessing:1
I1030 09:08:30.815184 1399 model_lifecycle.cc:461] loading: postprocessing:1
[TensorRT-LLM][WARNING] max_tokens_in_paged_kv_cache is not specified, will use default value
[TensorRT-LLM][WARNING] Cannot find parameter with name: batch_scheduler_policy
[TensorRT-LLM][WARNING] kv_cache_free_gpu_mem_fraction is not specified, will use default value of 0.85 or max_tokens_in_paged_kv_cache
[TensorRT-LLM][WARNING] max_num_sequences is not specified, will be set to the TRT engine max_batch_size
[TensorRT-LLM][WARNING] enable_trt_overlap is not specified, will be set to true
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be number, but is null
[TensorRT-LLM][WARNING] Optional value for parameter max_num_tokens will not be set.
[TensorRT-LLM][INFO] Initializing MPI with thread mode 1
I1030 09:08:30.880197 1398 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: postprocessing_0_0 (CPU device 0)
I1030 09:08:30.880584 1398 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: preprocessing_0_0 (CPU device 0)
[TensorRT-LLM][WARNING] max_tokens_in_paged_kv_cache is not specified, will use default value
[TensorRT-LLM][WARNING] Cannot find parameter with name: batch_scheduler_policy
[TensorRT-LLM][WARNING] kv_cache_free_gpu_mem_fraction is not specified, will use default value of 0.85 or max_tokens_in_paged_kv_cache
[TensorRT-LLM][WARNING] max_num_sequences is not specified, will be set to the TRT engine max_batch_size
[TensorRT-LLM][WARNING] enable_trt_overlap is not specified, will be set to true
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be number, but is null
[TensorRT-LLM][WARNING] Optional value for parameter max_num_tokens will not be set.
[TensorRT-LLM][INFO] Initializing MPI with thread mode 1
I1030 09:08:30.897152 1399 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: postprocessing_0_0 (CPU device 0)
I1030 09:08:30.897646 1399 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: preprocessing_0_0 (CPU device 0)
[TensorRT-LLM][INFO] MPI size: 2, rank: 0
[TensorRT-LLM][INFO] MPI size: 2, rank: 1
I1030 09:08:31.500529 1399 pb_stub.cc:325] Failed to initialize Python stub: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize

I1030 09:08:31.503903 1398 pb_stub.cc:325] Failed to initialize Python stub: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize

E1030 09:08:31.669733 1398 backend_model.cc:634] ERROR: Failed to create instance: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize

E1030 09:08:31.669950 1398 model_lifecycle.cc:621] failed to load 'postprocessing' version 1: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize

I1030 09:08:31.669989 1398 model_lifecycle.cc:756] failed to load 'postprocessing'
E1030 09:08:31.686577 1399 backend_model.cc:634] ERROR: Failed to create instance: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize

E1030 09:08:31.686780 1399 model_lifecycle.cc:621] failed to load 'postprocessing' version 1: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize

I1030 09:08:31.686815 1399 model_lifecycle.cc:756] failed to load 'postprocessing'
I1030 09:08:32.939433 1398 pb_stub.cc:325] Failed to initialize Python stub: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/preprocessing/1/model.py(69): initialize

I1030 09:08:32.948539 1399 pb_stub.cc:325] Failed to initialize Python stub: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/preprocessing/1/model.py(69): initialize

E1030 09:08:33.484284 1399 backend_model.cc:634] ERROR: Failed to create instance: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/preprocessing/1/model.py(69): initialize

E1030 09:08:33.484479 1399 model_lifecycle.cc:621] failed to load 'preprocessing' version 1: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/preprocessing/1/model.py(69): initialize

I1030 09:08:33.484529 1399 model_lifecycle.cc:756] failed to load 'preprocessing'
E1030 09:08:33.497454 1398 backend_model.cc:634] ERROR: Failed to create instance: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/preprocessing/1/model.py(69): initialize

E1030 09:08:33.497550 1398 model_lifecycle.cc:621] failed to load 'preprocessing' version 1: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/preprocessing/1/model.py(69): initialize

I1030 09:08:33.497571 1398 model_lifecycle.cc:756] failed to load 'preprocessing'
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 2
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 1
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 1
[TensorRT-LLM][INFO] Loaded engine size: 7653 MiB
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 2
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 1
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 1
[TensorRT-LLM][INFO] Loaded engine size: 7653 MiB
[TensorRT-LLM][INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 8683, GPU 26856 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 8685, GPU 26866 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 8683, GPU 56434 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] Init cuDNN: CPU +2, GPU +10, now: CPU 8685, GPU 56444 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +7649, now: CPU 0, GPU 7649 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +7649, now: CPU 0, GPU 7649 (MiB)
E1030 09:08:44.006947 1398 backend_model.cc:634] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
E1030 09:08:44.007090 1398 model_lifecycle.cc:621] failed to load 'tensorrt_llm' version 2: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
I1030 09:08:44.007124 1398 model_lifecycle.cc:756] failed to load 'tensorrt_llm'
E1030 09:08:44.007384 1398 model_repository_manager.cc:563] Invalid argument: ensemble 'ensemble' depends on 'postprocessing' which has no loaded version. Model 'postprocessing' loading failed with error: version 1 is at UNAVAILABLE state: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize
;
I1030 09:08:44.007619 1398 server.cc:592]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1030 09:08:44.007837 1398 server.cc:619]
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| tensorrtllm | /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-comput |
| | | e-capability":"6.000000","default-max-batch-size":"4"}} |
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-comput |
| | | e-capability":"6.000000","shm-region-prefix-name":"prefix0_","default-max-batch-size":"4"}} |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+

I1030 09:08:44.008106 1398 server.cc:662]
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Model | Version | Status |
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| postprocessing | 1 | UNAVAILABLE: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported. |
| | | |
| | | At: |
| | | /usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained |
| | | /tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize |
| preprocessing | 1 | UNAVAILABLE: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported. |
| | | |
| | | At: |
| | | /usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained |
| | | /tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/preprocessing/1/model.py(69): initialize |
| tensorrt_llm | 2 | UNAVAILABLE: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and pa |
| | | ged KV cache. |
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+

E1030 09:08:44.013076 1399 backend_model.cc:634] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
E1030 09:08:44.013151 1399 model_lifecycle.cc:621] failed to load 'tensorrt_llm' version 2: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
I1030 09:08:44.013168 1399 model_lifecycle.cc:756] failed to load 'tensorrt_llm'
E1030 09:08:44.013322 1399 model_repository_manager.cc:563] Invalid argument: ensemble 'ensemble' depends on 'postprocessing' which has no loaded version. Model 'postprocessing' loading failed with error: version 1 is at UNAVAILABLE state: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported.

At:
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained
/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize
;
I1030 09:08:44.013473 1399 server.cc:592]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1030 09:08:44.013625 1399 server.cc:619]
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| tensorrtllm | /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-comput |
| | | e-capability":"6.000000","default-max-batch-size":"4"}} |
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-comput |
| | | e-capability":"6.000000","shm-region-prefix-name":"prefix1_","default-max-batch-size":"4"}} |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+

I1030 09:08:44.013837 1399 server.cc:662]
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Model | Version | Status |
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| postprocessing | 1 | UNAVAILABLE: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported. |
| | | |
| | | At: |
| | | /usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained |
| | | /tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/postprocessing/1/model.py(65): initialize |
| preprocessing | 1 | UNAVAILABLE: Internal: ValueError: Tokenizer class BaichuanTokenizer does not exist or is not currently imported. |
| | | |
| | | At: |
| | | /usr/local/lib/python3.10/dist-packages/transformers/models/auto/tokenization_auto.py(748): from_pretrained |
| | | /tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/preprocessing/1/model.py(69): initialize |
| tensorrt_llm | 2 | UNAVAILABLE: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and pa |
| | | ged KV cache. |
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1030 09:08:44.073052 1398 metrics.cc:817] Collecting metrics for GPU 0: NVIDIA A800 80GB PCIe
I1030 09:08:44.073087 1398 metrics.cc:817] Collecting metrics for GPU 1: NVIDIA A800 80GB PCIe
I1030 09:08:44.076025 1398 metrics.cc:710] Collecting CPU metrics
I1030 09:08:44.076206 1398 tritonserver.cc:2458]
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.39.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_me |
| | mory binary_tensor_data parameters statistics trace logging |
| model_repository_path[0] | /tensorrtllm_backend/tensorrtllm_backend/triton_model_repo |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| cuda_memory_pool_byte_size{1} | 67108864 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
| cache_enabled | 0 |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+

I1030 09:08:44.076217 1398 server.cc:293] Waiting for in-flight requests to complete.
I1030 09:08:44.076221 1398 server.cc:309] Timeout 30: Found 0 model versions that have in-flight inferences
I1030 09:08:44.076228 1398 server.cc:324] All models are stopped, unloading models
I1030 09:08:44.076234 1398 server.cc:331] Timeout 30: Found 0 live models and 0 in-flight non-inference requests
I1030 09:08:44.077887 1399 metrics.cc:817] Collecting metrics for GPU 0: NVIDIA A800 80GB PCIe
I1030 09:08:44.077913 1399 metrics.cc:817] Collecting metrics for GPU 1: NVIDIA A800 80GB PCIe
I1030 09:08:44.078310 1399 metrics.cc:710] Collecting CPU metrics
I1030 09:08:44.078464 1399 tritonserver.cc:2458]
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.39.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_me |
| | mory binary_tensor_data parameters statistics trace logging |
| model_repository_path[0] | /tensorrtllm_backend/tensorrtllm_backend/triton_model_repo |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| cuda_memory_pool_byte_size{1} | 67108864 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
| cache_enabled | 0 |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+

I1030 09:08:44.078473 1399 server.cc:293] Waiting for in-flight requests to complete.
I1030 09:08:44.078477 1399 server.cc:309] Timeout 30: Found 0 model versions that have in-flight inferences
I1030 09:08:44.078484 1399 server.cc:324] All models are stopped, unloading models
I1030 09:08:44.078488 1399 server.cc:331] Timeout 30: Found 0 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
error: creating server: Internal - failed to load all models

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

---

mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[9409,1],0]
Exit code: 1

===========

i use baichuan2-13b

[anxietymonger]

It seems caused by lack of argument trust_remote_code=True when loading BaichuanTokenizer. After manually adding the argument in preprocessing/1/model.py and postprocessing/1/1model.py, the server launched successfully. Could we expect a fix for these? Or there is other smarter workaround (such as adding a command line argument)?

[zhanglv0209]

It seems caused by lack of argument trust_remote_code=True when loading BaichuanTokenizer. After manually adding the argument in preprocessing/1/model.py and postprocessing/1/1model.py, the server launched successfully. Could we expect a fix for these? Or there is other smarter workaround (such as adding a command line argument)?

thanks, it help me slove problems

but other error happen:

CUDA_VISIBLE_DEVICES=0,3 python3 ./scripts/launch_triton_server.py --world_size=2 --model_repo=/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo
root@GPU-26:/tensorrtllm_backend/tensorrtllm_backend/tensorrtllm_backend# I1030 10:36:44.146919 3659 pinned_memory_manager.cc:241] Pinned memory pool is created at '0x7f207a000000' with size 268435456
I1030 10:36:44.147641 3660 pinned_memory_manager.cc:241] Pinned memory pool is created at '0x7f29ee000000' with size 268435456
I1030 10:36:44.152854 3659 cuda_memory_manager.cc:107] CUDA memory pool is created on device 0 with size 67108864
I1030 10:36:44.152865 3659 cuda_memory_manager.cc:107] CUDA memory pool is created on device 1 with size 67108864
I1030 10:36:44.153494 3660 cuda_memory_manager.cc:107] CUDA memory pool is created on device 0 with size 67108864
I1030 10:36:44.153503 3660 cuda_memory_manager.cc:107] CUDA memory pool is created on device 1 with size 67108864
I1030 10:36:44.517834 3660 model_lifecycle.cc:461] loading: tensorrt_llm:2
I1030 10:36:44.517880 3660 model_lifecycle.cc:461] loading: preprocessing:1
I1030 10:36:44.517903 3660 model_lifecycle.cc:461] loading: postprocessing:1
I1030 10:36:44.521580 3659 model_lifecycle.cc:461] loading: tensorrt_llm:2
I1030 10:36:44.521624 3659 model_lifecycle.cc:461] loading: preprocessing:1
I1030 10:36:44.521651 3659 model_lifecycle.cc:461] loading: postprocessing:1
[TensorRT-LLM][WARNING] The batch scheduler policy will be set to guaranteed_no_evictsince the backend operates in decoupled mode
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be number, but is null
[TensorRT-LLM][WARNING] Optional value for parameter max_num_tokens will not be set.
[TensorRT-LLM][INFO] Initializing MPI with thread mode 1
I1030 10:36:44.601752 3660 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: postprocessing_0_0 (CPU device 0)
I1030 10:36:44.602146 3660 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: preprocessing_0_0 (CPU device 0)
[TensorRT-LLM][WARNING] The batch scheduler policy will be set to guaranteed_no_evictsince the backend operates in decoupled mode
[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be number, but is null
[TensorRT-LLM][WARNING] Optional value for parameter max_num_tokens will not be set.
[TensorRT-LLM][INFO] Initializing MPI with thread mode 1
I1030 10:36:44.610881 3659 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: postprocessing_0_0 (CPU device 0)
I1030 10:36:44.611077 3659 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: preprocessing_0_0 (CPU device 0)
[TensorRT-LLM][INFO] MPI size: 2, rank: 0
[TensorRT-LLM][INFO] MPI size: 2, rank: 1
I1030 10:36:45.179615 3660 model_lifecycle.cc:818] successfully loaded 'postprocessing'
I1030 10:36:45.323014 3659 model_lifecycle.cc:818] successfully loaded 'postprocessing'
I1030 10:36:46.691643 3660 model_lifecycle.cc:818] successfully loaded 'preprocessing'
I1030 10:36:46.920485 3659 model_lifecycle.cc:818] successfully loaded 'preprocessing'
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 4096
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 1
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 1
[TensorRT-LLM][INFO] Loaded engine size: 7653 MiB
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 4096
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 1
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 1
[TensorRT-LLM][INFO] Loaded engine size: 7653 MiB
[TensorRT-LLM][INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 8684, GPU 26856 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 8685, GPU 26866 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 8684, GPU 59224 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 8685, GPU 59234 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +7649, now: CPU 0, GPU 7649 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +7649, now: CPU 0, GPU 7649 (MiB)
E1030 10:36:54.952488 3660 backend_model.cc:634] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
E1030 10:36:54.952597 3660 model_lifecycle.cc:621] failed to load 'tensorrt_llm' version 2: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
I1030 10:36:54.952616 3660 model_lifecycle.cc:756] failed to load 'tensorrt_llm'
E1030 10:36:54.952767 3660 model_repository_manager.cc:563] Invalid argument: ensemble 'ensemble' depends on 'tensorrt_llm' which has no loaded version. Model 'tensorrt_llm' loading failed with error: version 2 is at UNAVAILABLE state: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.;
I1030 10:36:54.952951 3660 server.cc:592]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1030 10:36:54.953110 3660 server.cc:619]
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| tensorrtllm | /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-comput |
| | | e-capability":"6.000000","default-max-batch-size":"4"}} |
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-comput |
| | | e-capability":"6.000000","shm-region-prefix-name":"prefix1_","default-max-batch-size":"4"}} |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+

I1030 10:36:54.953280 3660 server.cc:662]
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Model | Version | Status |
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| postprocessing | 1 | READY |
| preprocessing | 1 | READY |
| tensorrt_llm | 2 | UNAVAILABLE: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and pa |
| | | ged KV cache. |
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+

E1030 10:36:54.977825 3659 backend_model.cc:634] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
E1030 10:36:54.977950 3659 model_lifecycle.cc:621] failed to load 'tensorrt_llm' version 2: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
I1030 10:36:54.977986 3659 model_lifecycle.cc:756] failed to load 'tensorrt_llm'
E1030 10:36:54.978209 3659 model_repository_manager.cc:563] Invalid argument: ensemble 'ensemble' depends on 'tensorrt_llm' which has no loaded version. Model 'tensorrt_llm' loading failed with error: version 2 is at UNAVAILABLE state: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.;
I1030 10:36:54.978411 3659 server.cc:592]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+

I1030 10:36:54.978573 3659 server.cc:619]
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+
| tensorrtllm | /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-comput |
| | | e-capability":"6.000000","default-max-batch-size":"4"}} |
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min-comput |
| | | e-capability":"6.000000","shm-region-prefix-name":"prefix0_","default-max-batch-size":"4"}} |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------+

I1030 10:36:54.978713 3659 server.cc:662]
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Model | Version | Status |
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+
| postprocessing | 1 | READY |
| preprocessing | 1 | READY |
| tensorrt_llm | 2 | UNAVAILABLE: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and pa |
| | | ged KV cache. |
+----------------+---------+--------------------------------------------------------------------------------------------------------------------------------------------------------------+

I1030 10:36:55.022033 3660 metrics.cc:817] Collecting metrics for GPU 0: NVIDIA A800 80GB PCIe
I1030 10:36:55.022105 3660 metrics.cc:817] Collecting metrics for GPU 1: NVIDIA A800 80GB PCIe
I1030 10:36:55.025567 3660 metrics.cc:710] Collecting CPU metrics
I1030 10:36:55.025920 3660 tritonserver.cc:2458]
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.39.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_me |
| | mory binary_tensor_data parameters statistics trace logging |
| model_repository_path[0] | /tensorrtllm_backend/tensorrtllm_backend/triton_model_repo |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| cuda_memory_pool_byte_size{1} | 67108864 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
| cache_enabled | 0 |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+

I1030 10:36:55.025942 3660 server.cc:293] Waiting for in-flight requests to complete.
I1030 10:36:55.025954 3660 server.cc:309] Timeout 30: Found 0 model versions that have in-flight inferences
I1030 10:36:55.026117 3660 server.cc:324] All models are stopped, unloading models
I1030 10:36:55.026133 3660 server.cc:331] Timeout 30: Found 2 live models and 0 in-flight non-inference requests
I1030 10:36:55.029244 3659 metrics.cc:817] Collecting metrics for GPU 0: NVIDIA A800 80GB PCIe
I1030 10:36:55.029289 3659 metrics.cc:817] Collecting metrics for GPU 1: NVIDIA A800 80GB PCIe
I1030 10:36:55.029793 3659 metrics.cc:710] Collecting CPU metrics
I1030 10:36:55.029966 3659 tritonserver.cc:2458]
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.39.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_me |
| | mory binary_tensor_data parameters statistics trace logging |
| model_repository_path[0] | /tensorrtllm_backend/tensorrtllm_backend/triton_model_repo |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| cuda_memory_pool_byte_size{1} | 67108864 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
| cache_enabled | 0 |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------+

I1030 10:36:55.029981 3659 server.cc:293] Waiting for in-flight requests to complete.
I1030 10:36:55.029987 3659 server.cc:309] Timeout 30: Found 0 model versions that have in-flight inferences
I1030 10:36:55.030062 3659 server.cc:324] All models are stopped, unloading models
I1030 10:36:55.030068 3659 server.cc:331] Timeout 30: Found 2 live models and 0 in-flight non-inference requests
I1030 10:36:56.026239 3660 server.cc:331] Timeout 29: Found 2 live models and 0 in-flight non-inference requests
Cleaning up...
Cleaning up...
I1030 10:36:56.030173 3659 server.cc:331] Timeout 29: Found 2 live models and 0 in-flight non-inference requests
Cleaning up...
Cleaning up...
I1030 10:36:56.258719 3660 model_lifecycle.cc:603] successfully unloaded 'postprocessing' version 1
I1030 10:36:56.266025 3659 model_lifecycle.cc:603] successfully unloaded 'postprocessing' version 1
I1030 10:36:56.580914 3660 model_lifecycle.cc:603] successfully unloaded 'preprocessing' version 1
I1030 10:36:56.686998 3659 model_lifecycle.cc:603] successfully unloaded 'preprocessing' version 1
I1030 10:36:57.026391 3660 server.cc:331] Timeout 28: Found 0 live models and 0 in-flight non-inference requests
I1030 10:36:57.030332 3659 server.cc:331] Timeout 28: Found 0 live models and 0 in-flight non-inference requests
error: creating server: Internal - failed to load all models
error: creating server: Internal - failed to load all models

Primary job terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

---

mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

Process name: [[12276,1],1]
Exit code: 1

here is tensorrt_llm/config.pbtxt:

name: "tensorrt_llm"
backend: "tensorrtllm"
max_batch_size: 128

model_transaction_policy {
decoupled: True
}

input [
{
name: "input_ids"
data_type: TYPE_INT32
dims: [ -1 ]
},
{
name: "input_lengths"
data_type: TYPE_INT32
dims: [ 1 ]
reshape: { shape: [ ] }
},
{
name: "request_output_len"
data_type: TYPE_UINT32
dims: [ 1 ]
},
{
name: "end_id"
data_type: TYPE_UINT32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "pad_id"
data_type: TYPE_UINT32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "beam_width"
data_type: TYPE_UINT32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "temperature"
data_type: TYPE_FP32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "runtime_top_k"
data_type: TYPE_UINT32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "runtime_top_p"
data_type: TYPE_FP32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "len_penalty"
data_type: TYPE_FP32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "repetition_penalty"
data_type: TYPE_FP32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "min_length"
data_type: TYPE_UINT32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "presence_penalty"
data_type: TYPE_FP32
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "random_seed"
data_type: TYPE_UINT64
dims: [ 1 ]
reshape: { shape: [ ] }
optional: true
},
{
name: "stop"
data_type: TYPE_BOOL
dims: [ 1 ]
optional: true
},
{
name: "streaming"
data_type: TYPE_BOOL
dims: [ 1 ]
optional: true
}
]
output [
{
name: "output_ids"
data_type: TYPE_INT32
dims: [ -1, -1 ]
}
]
instance_group [
{
count: 1
kind : KIND_CPU
}
]
parameters: {
key: "max_beam_width"
value: {
string_value: "1"
}
}
parameters: {
key: "FORCE_CPU_ONLY_INPUT_TENSORS"
value: {
string_value: "no"
}
}
parameters: {
key: "gpt_model_type"
value: {
string_value: "inflight_fused_batching"
}
}
parameters: {
key: "gpt_model_path"
value: {
string_value: "/tensorrtllm_backend/tensorrtllm_backend/triton_model_repo/tensorrt_llm/2/2-gpu"
}
}
parameters: {
key: "max_tokens_in_paged_kv_cache"
value: {
string_value: "1000M"
}
}
parameters: {
key: "batch_scheduler_policy"
value: {
string_value: "max_utilization"
}
}
parameters: {
key: "kv_cache_free_gpu_mem_fraction"
value: {
string_value: "0.85"
}
}
parameters: {
key: "max_num_sequences"
value: {
string_value: "4096"
}
}
parameters: {
key: "enable_trt_overlap"
value: {
string_value: "true"
}
}

[byshiue]

The errors are

E1030 10:36:54.952488 3660 backend_model.cc:634] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
E1030 10:36:54.952597 3660 model_lifecycle.cc:621] failed to load 'tensorrt_llm' version 2: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.

To use inflight-batching, we need to enable gpt attention plugin and paged kv cache. Can you check your argument of building engine or share your building script?

[zhanglv0209]

The errors are

E1030 10:36:54.952488 3660 backend_model.cc:634] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
E1030 10:36:54.952597 3660 model_lifecycle.cc:621] failed to load 'tensorrt_llm' version 2: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.

To use inflight-batching, we need to enable gpt attention plugin and paged kv cache. Can you check your argument of building engine or share your building script?

think you, the problems sloved !

[zhanglv0209]

parameters: {
key: "gpt_model_type"
value: {
string_value: "v1"
}
}

[byshiue]

Thank you for the update. As you find, V1 does not have such requirement.

Close this bug. Feel free to repoen it if needed.

[anxietymonger]

@byshiue Sorry but I'm still wondering that for preprocessing and postprocessing, currently users have to manually add trust_remote_code argument to model.py, is this the expected use case (users have to maintain their own code repository)? Or should I make a PR that adds trust_remote_code=True to model.py?

[byshiue]

We are not sure the effect of adding trust_remote_code=True. We don't encounter such issue when we try to reproduce. I suspect it is related to the way to launch the server. We will take a look.

[anxietymonger]

@byshiue Thank you very much. FYI, trust_remote_code=True is only required for specific models, such as BaichuanTokenizer.

[77h2l]

@byshiue
seems output was not abnormal, I am running the Baichuan2-13B-base model, according the doc the server was successfully enable, but the output seems incorrect,

and my model build command was as follows:
python build.py --model_version v2_13b --model_dir ./Baichuan2-13B-Base/ --dtype float16 --use_gemm_plugin float16 --use_gpt_attention_plugin float16 --use_weight_only --output_dir ./tmp/baichuan_v2_13b/trt_engines/int8_weight_only/1-gpu/

[77h2l]

what's more, when I tried another client script, another error occurs, [StatusCode.UNIMPLEMENTED] ModelInfer RPC doesn't support models with decoupled transaction policy
python3 inflight_batcher_llm/client/inflight_batcher_llm_client.py --request-output-len 200 --tokenizer_dir tensorrt_llm/examples/baichuan/Baichuan2-13B-Base
Got completed request
Received an error from server:
[StatusCode.UNIMPLEMENTED] ModelInfer RPC doesn't support models with decoupled transaction policy
output_ids = []
Input: Born in north-east France, Soyer trained as a
Output:

does anyone have any clues?

[77h2l]

what's more, when I tried another client script, another error occurs, [StatusCode.UNIMPLEMENTED] ModelInfer RPC doesn't support models with decoupled transaction policy python3 inflight_batcher_llm/client/inflight_batcher_llm_client.py --request-output-len 200 --tokenizer_dir tensorrt_llm/examples/baichuan/Baichuan2-13B-Base Got completed request Received an error from server: [StatusCode.UNIMPLEMENTED] ModelInfer RPC doesn't support models with decoupled transaction policy output_ids = [] Input: Born in north-east France, Soyer trained as a Output:

does anyone have any clues?

I have figured out this problem's reason, its due to the decouple param in the tensorrt_llm's config file, it should be set False to disable this problem, but the above problem still remains.