[Feature]: Ensure output consistency when using LoRA with Eagle3 Speculative Decoding #26679
Description
🚀 The feature, motivation and pitch
Currently, when I enable both LoRA and Eagle3 in vLLM, the model runs without crashing, but the generated output is inconsistent with the output from using only the LoRA-adapted model.
My setup is as follows:
A LoRA adapter fine-tuned on the base model.
An Eagle3 drafter model that was trained on the Base Model + LoRA combination.
When I run inference with --enable-lora and --speculative_config for Eagle3, the generated text differs from the output when running with only --enable-lora. Since Eagle3 is an acceleration technique, it should not alter the final output of the target model. This discrepancy suggests that the LoRA adapter is not being correctly applied during the verification stage of the speculative decoding process.
A related PR, #21068 , addresses a RuntimeError crash when using LoRA with speculative decoding by fixing shape mismatches. However, it does not seem to fully resolve this correctness/consistency issue. The problem I'm observing is not a crash, but incorrect output.
Alternatives
Merge LoRA weights: Merge the LoRA weights into the base model to create a new, standalone model, and then use Eagle3 with it. This is highly inflexible and defeats the purpose of vLLM's dynamic LoRA adapter loading, which allows serving multiple adapters concurrently.
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.