Repository navigation
LoRA adapters trained on a 4-bit base degrade when served on a BF16 base — 100-sample comparison #7231
Replies: 1 comment
|
Before blaming NF4, I'd look at the vLLM side, since your comparison changes the engine and the base precision at the same time. There's an open vLLM issue for this exact model family (vllm-project/vllm#49354): on Qwen3.5, Two cheap runs would settle it. vLLM base vs vLLM + LoRA on your 100 prompts: if the outputs are identical, the adapter isn't being applied. Then Transformers + BF16 base + LoRA, which isolates precision from the engine. Also make sure both paths use If precision does turn out to matter, I'd skip AWQ/GPTQ. Load the base in NF4 the way you trained it, call I haven't reproduced any of this, so treat the vLLM bug as the 1st suspect, not a diagnosis. A docs note seems worth it either way. |
Uh oh!
There was an error while loading. Please reload this page.
I ran into something while moving a QLoRA-trained adapter from training to serving, and I could not find it documented anywhere. Posting the numbers in case it is useful to others, and to ask whether this is expected.
Setup
Base:
Qwen/Qwen3.5-4BTraining: QLoRA via
SFTTrainer— NF4 4-bit + double quantization + paged AdamW, LoRAr=16,alpha=32,target_modules="all-linear"(32.5M trainable params, 0.77% of the model)Task: natural-language ops requests -> structured JSON tool calls (fixed schema: tool name, arguments, risk level, confirmation flag)
Data: 2,100 synthetic samples, stratified 1,600 / 200 / 300
Decoding: greedy, identical prompt across all runs
What I compared
The same adapter, served two ways, on the same 100 requests:
| vLLM + BF16 base + LoRA | Transformers + NF4 base + LoRA -- | -- | -- TTFT (mean) | 94 ms | 363 ms Output throughput | 71.6 tok/s | 7.2 tok/s GPU memory | 18.6 GB | 3.8 GB Strict JSON validity | 100% | 100% Strict exact match | 67.0% | 100%The part I found interesting
JSON validity is 100% on both sides, but exact match differs by 33 points. The failures are not malformed output, truncation, or a broken serving path — the model emits well-formed JSON with the correct top-level keys, and the field values drift. Enum-like fields are where it shows up.
My working hypothesis: the adapter was trained on top of an NF4-quantized base, so its weights implicitly absorb some of the quantization error. Serving it on an unquantized BF16 base removes what the adapter was compensating for.
Why I think this is worth writing down
The intuitive migration path is "train with QLoRA to save memory, then serve on BF16 for throughput." That path silently cost me a third of my accuracy, and nothing in the pipeline complains — the adapter loads fine, generations look fine, and only a field-level exact-match check catches it.
What I have not tested
Being upfront about the limits of this:
Single task, single base model, synthetic template-generated data
I have not tried
merge_and_unload()into the quantized base and exporting to AWQ/GPTQ, which I believe is the correct path if you want the throughputI have not checked whether the gap narrows with a lower LoRA rank or with attention-only
target_modulesn=100 for the serving comparison (the offline eval was n=300)
Questions
Is this the expected behavior for adapters trained under QLoRA, or does the size of the gap suggest I did something wrong?
Is merging into the quantized base before export the recommended path here?
Would a note about this in the QLoRA docs be welcome? Happy to open a PR if so.
All reactions