How to load llama. I tried, but I got an error

[UncleFB]

The model is vicuna_13b. I use build.py to generate a 2-gpu model.

E1025 09:17:53.664847 126 backend_model.cc:553] ERROR: Failed to create instance: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
E1025 09:17:53.664952 126 model_lifecycle.cc:622] failed to load 'tensorrt_llm' version 1: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.
I1025 09:17:53.664979 126 model_lifecycle.cc:757] failed to load 'tensorrt_llm'
E1025 09:17:53.665119 126 model_repository_manager.cc:563] Invalid argument: ensemble 'ensemble' depends on 'tensorrt_llm' which has no loaded version. Model 'tensorrt_llm' loading failed with error: version 1 is at UNAVAILABLE state: Internal: unexpected error when creating modelInstanceState: TrtGptModelInflightBatching requires GPT attention plugin with packed input and paged KV cache.;
I1025 09:17:53.665245 126 server.cc:604]

[byshiue]

It looks you don't have installed tensorrt_llm backend successfully. Can you share your building steps and try on the examples demonstrated in readme first?

[UncleFB]

I try on the example. Server is running. But when I curl -X POST localhost:8000/v2/models/ensemble/generate -d '{"text_input": "What is machine learning?", "max_tokens": 20, "bad_words": "", "stop_words": ""}'. Nothing happened.
When I try to use inflight_batcher_llm_client, I got response

[UncleFB]

The model I used is vicuna-13b. I need two gpus.

python build.py --model_dir ./tmp/vicuna/13B/
--dtype float16
--remove_input_padding
--use_gpt_attention_plugin float16
--enable_context_fmha
--use_gemm_plugin float16
--output_dir ./tmp/vicuna/13B/trt_engines/fp16/2-gpu/
--world_size 2
--tp_size 2

config.pbtxt
parameters: {
key: "gpt_model_type"
value: {
string_value: "V1"
}
}
parameters: {
key: "gpt_model_path"
value: {
string_value: "/tensorrtllm_backend/triton_model_repo/tensorrt_llm/1/2-gpu"
}
}
parameters {
key: "tokenizer_dir"
value: {
string_value: "/tensorrtllm_backend/tensorrt_llm/examples/llama/tmp/vicuna/13B"
}
}

when I execute python3 scripts/launch_triton_server.py --world_size=2 --model_repo=/tensorrtllm_backend/triton_model_repo

I got an OOM.

[byshiue]

I try on the example. Server is running. But when I curl -X POST localhost:8000/v2/models/ensemble/generate -d '{"text_input": "What is machine learning?", "max_tokens": 20, "bad_words": "", "stop_words": ""}'. Nothing happened.
When I try to use inflight_batcher_llm_client, I got response

generate endpoint will only be working with Triton 23.10 (https://github.com/triton-inference-server/tensorrtllm_backend/tree/release/0.5.0#query-the-server-with-the-triton-generate-endpoint).

[yunfeng-scale]

@UncleFB I'm seeing the same error with llama. Can you tell me how you fixed it?

[yunfeng-scale]

looks like this is related to how the engine is built. when building needs to specify --use_inflight_batching. these params (gpt_attention_plugin and paged_kv_cache) can be double checked from config.json

[yunfeng-scale]

where is backend_model.cc though? I can't find it in Triton, TRT-LLM or TRT-LLM Triton backend

[UncleFB]

@yunfeng-scale Try this.
python build.py --model_dir /model_dir
--world_size 2
--tp_size 2
--dtype float16
--use_inflight_batching
--use_gpt_attention_plugin float16
--remove_input_padding
--paged_kv_cache
--use_gemm_plugin float16
--parallel_build
--output_dir engines/fp16/2-gpu

[byshiue]

tritonserver 23.10 docker image is release, you can take a try on endpoint.

[zengqingfu1442]

@yunfeng-scale Try this. python build.py --model_dir /model_dir --world_size 2 --tp_size 2 --dtype float16 --use_inflight_batching --use_gpt_attention_plugin float16 --remove_input_padding --paged_kv_cache --use_gemm_plugin float16 --parallel_build --output_dir engines/fp16/2-gpu

So removing --enable_context_fmha and adding --paged_kv_cache when building engine is the solution to this error?