xverse-65b error

[lwbmowgli]

I successfully built xverse-65b using the llama example, and successfully deployed it using triton, but an error occurred during inference. What is the reason? How should I modify it?
[TensorRT-LLM][ERROR] Encountered an error in forward function: [TensorRT-LLM][ERROR] Assertion failed: Tensor 'past_key_value_0' has invalid shape (1, 2, 8, 1536, 128) (/app/tensorrt_llm/ cpp/tensorrt_llm/runtime/tllmRuntime.cpp:150)

[lwbmowgli]

这是我的build命令
python build_xverse.py --model_dir xverse-65b
--use_gpt_attention_plugin float16
--use_weight_only
--weight_only_precision int4
--max_batch_size 1
--output_dir XVERSE-65B
--world_size 8
--tp_size 8

[StudyingShao]

@lwbmowgli
what is build_xverse.py here? Are you using main branch now?
Please give more details here.

[lwbmowgli]

因为这个和llama具有相同的结构，所以我直接使用了https://github.com/NVIDIA/TensorRT-LLM/blob/v0.7.1/examples/llama/build.py 。所以build_xverse.py这个脚本就是链接中的脚本，7b和13b的版本都能够正常起服务，但是65b的版本就会报告上面的错误。我使用的镜像是23.12，我是否应该切换到24.01测试一下？

[StudyingShao]

I see. will have a try with the latest main branch and get back to you later.

[StudyingShao]

BTW, I saw that --use_weight_only and --weight_only_precision int4 are added here. Do you want to use int4 weight-only per-channel quantization here? If so, a more fine-grained quantization method is recommended. Please try awq with the latest main branch. It has a better accuracy and perf boost with bs = 1 you used here. @lwbmowgli

[lwbmowgli]

大佬，现在有什么新的消息吗？ @StudyingShao