Model output keeps generating

[eladamittai]

Hello, I built the triton 23.12 container with the trtllm 0.7.1 backend using the 3rd build option in the triton trtllm guide, and deployed 2 models, Mistral 7b instruct and phind codellama v2 34b, and when I asked them simple questions that didn't need long answers like "what is 1+1" with 50 max tokens, the output was the answer, but it kept generating . It repeated the question with , 2+2 is 4, 3+3 is 6 etc, till it reached the max tokens. I tried sending it temperature 0 and stop_words, but it kept happening. I also tried using prompt engineering to tell hime to stop after giving the answer, but it didn't work. Happened both in generate and generate_stream. Sent the request through the tensorrt_llm_bls. I loaded both models on vllm, and it worked just fine. I was wondering if this is a feature or a bug of there is a way to stop that in the config.pbtxt or something. Thanks in advance! 🙏

[byshiue]

It should be caused by model capability in most time. To have a fair comparsion with reference (might be vllm here), you should make sure you pass full same runtime argumnents.

stop_words should be also a way to prevent such issue, you could print the stop_ids at https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/all_models/inflight_batcher_llm/preprocessing/1/model.py#L190C13-L190C24 and the outptu_ids (tokens_batch) at https://github.com/triton-inference-server/tensorrtllm_backend/blob/main/all_models/inflight_batcher_llm/postprocessing/1/model.py#L186C40-L186C40 to make sure you setup correct stop words.

[eladamittai]

I tried using the stop words. I noticed that every time it keeps generating, it separates the answer from the rest of the output with ##. I tried sending it, but didn't work. The request: curl -X POST localhost:8000/v2/tensorrt_llm_bls/generate -d '{"text_input": "what is 1+1?", "max_tokens": 100, "bad_words": "", "stop_words": "##", "temperature": 0}'

As for the comparison, I compiled the model on vllm with the same parameters I did for the trtllm

[byshiue]

As I mentioned, please try check the stop ids and output ids.

[eladamittai]

Well, my computer broke, so I can't really test it right now. I'll try it and reopen the issue if needed. Thanks a lot!!!