Inference results from vLLM is inconsistant with HF

[silverriver]

Someone has raised this issue in the vLLM project

In the case of greedy decoding, although LLaMA 7B, 13B, 30B can get meaningful output, the output results are different from HF transformers.

vllm-project/vllm#450 (comment)

[AlpinDale]

Not much of an issue anymore. The minute difference are due to precision differences in floating point operations. Downcasting operations from FP32 to BF16/FP16 introduces a slight amount of numerical instability, which causes the engine to produce slightly different responses in otherwise deterministic settings.

This has not been an issue so far for downstream users, but I'll investigate a fix as soon as I'm able.

[silverriver]

Yeap, this is not a big issue in more use cases.

[boluoyu]

Yeap, this is not a big issue in more use cases.

This is a big problem, and when I use Qwen1.5 , a lot of the inference results from vLLM are wrong