VLLM output is not complete

[RickyGunawan09]

hai guys,
thank you for making this super library.
i have a question about the output of vllm

i'm using GPU RTX A6000 50GB cuda 12 with model Vicuna13B-v1.5-4k from lmsys
vllm is serve with gpu_memory_utilization 0.8
the parameter that i change for request is:

1. max_token 4096
2. temperature 0

i'm make custom prompt with context from text/document.

why sometimes the output is not complete ?

[yaofeng]

+1

The answer to my query is not complete.
I have tried many queries, and I have the same problems.

I use chatglm3-6b-chat model, and here is my code:

import os
from vllm import LLM, SamplingParams

os.environ["VLLM_USE_MODELSCOPE"] = "True"

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="ZhipuAI/chatglm3-6b", trust_remote_code=True)

query = "Who are you?"
tokenizer = llm.get_tokenizer()
prompt_token_ids=tokenizer.build_chat_input(query).input_ids.tolist()

print(tokenizer.decode(prompt_token_ids[0]))

outputs = llm.generate([query], prompt_token_ids=prompt_token_ids, use_tqdm=False)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

and here is the output:

[gMASK]sop<|user|> 
 Who are you?<|assistant|>

Prompt: 'Who are you?', Generated text: ' \n I am an AI assistant named ChatGLM3-6B,'

[yaofeng]

I found the max_tokens param.

sampling_params = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=1024)