Same generate params but got totally different result

[luohao123]

Hi, I have a params with do_sample False, the result on huggingface are pretty good.

But same set in vllm, the model likes a stupid bird.

sampling_params = SamplingParams(
        temperature=generating_args.temperature,
        top_p=generating_args.top_p,
        max_tokens=800,
        frequency_penalty=generating_args.repetition_penalty,
        use_beam_search=False,
        ignore_eos=False,
    )

Please let me know how to resolve the bias?

same args:

do_sample: Optional[bool] = field(default=True)
    temperature: Optional[float] = field(
        default=0.84,
        metadata={"help": "The value used to modulate the next token probabilities."},
    )
    top_p: Optional[float] = field(
        default=0.6,
        metadata={
            "help": "The smallest set of most probable tokens with probabilities that add up to top_p or higher are kept."
        },
    )
    top_k: Optional[int] = field(default=40)
    num_beams: Optional[int] = field(default=1)
    max_new_tokens: Optional[int] = field(default=800)
    repetition_penalty: Optional[float] = field(default=1.09)

Both set do_sample False. but the result are out of my scope, they are totally different.

[Zhuqln]

can you show the result about the difference?

[BaiMoHan]

This problem may be related to this #712 #450

[luohao123]

@Zhuqln hard to tell, but same question, vllm can not answer correctly.
this caused a massive performance reduce compare with hf transformers....

[hmellor]

Closing this issue as stale as there has been no discussion in the past 3 months.

If you are still experiencing the issue you describe, feel free to re-open this issue.