fix(finetune): NF4 QLoRA per-epoch val_loss must reflect trained adapters (FALSIFY-CUDA-EVAL-ADAPTER-SYNC-001)

[noahgift]

The bug

On the CUDA path, train_step writes LoRA adapter deltas into the GPU-resident cuda_blocks, but InstructPipeline::evaluate() computes val_loss on the CPU via model.forward_with_lora(x, &self.lora_layers). The CPU lora_layers are only refreshed by sync_lora_to_cpu(), which was invoked exclusively inside save_checkpoint — never before evaluate() in the epoch loop (instruct_trainer.rs:243).

Consequence: per-epoch val_loss is byte-identical across every epoch and every run, so:

• best_val_loss collapses to the epoch-0 constant
• best_epoch freezes at 0 → the best/ checkpoint is stale-by-N-epochs
• early stopping fires on a phantom plateau (the metric never moved)

Trained weights themselves were fine (final save syncs) — only the validation signal and the decisions it drives were dead.

The fix

evaluate() now calls sync_lora_to_cpu() before the CPU forward (takes &mut self), so val_loss reflects the current GPU-trained adapters. A #[cfg(not(feature = "cuda"))] no-op twin keeps the call unconditional — on CPU/WGPU paths lora_layers are updated in place and already current. This makes evaluate self-consistent for every caller, not just the trainer.

Falsifier — FALSIFY-CUDA-EVAL-ADAPTER-SYNC-001

Contract: contracts/finetune-eval-adapter-sync-v1.yaml (pv validate/pv lint clean). The probe injects a GPU-only adapter change via download→(set B nonzero)→upload — deliberately independent of the optimizer/clip path so it runs on a fresh, uncorrupted CUDA context — then asserts the next evaluate() differs.

Verified live on RTX 4090 (sm_89):

	val_before	val_after	|Δ|	result
RED (sync removed)	14.25047874	14.25047874	0.0	FAILED ✓
GREEN (sync present)	14.25047874	14.22203255	0.02844620	ok ✓

Testing

• CPU suite: 7612 pass (3 pre-existing prune::snapshot_tests insta-snapshot failures — unrelated module, present on origin/main)
• evaluate &mut self change: no other callers of InstructPipeline::evaluate exist workspace-wide; touched test bindings updated to mut
• Falsifier is GPU-gated (--ignored, needs APR_PARITY_MODEL); not yet CI-enforced pending a CUDA runner

Scope bound

sync_lora_to_cpu reconciles Q and V adapters (the default QLoRA target set). Configs that train K/O/gate/up/down would still evaluate those partially stale — documented in the contract as a separate extension.

Surfaced (tracked separately, not in this PR)

While building the falsifier, two orthogonal GPU-training defects re-surfaced on sm_89 (both → CUDA_ERROR_ILLEGAL_ADDRESS): (A) the seq<32 batched-softmax partial-warp shfl.sync UB (a #2252 follow-up), and (B) the fused LoRA grad-clip PTX kernels (gradient_clip_gpu_scale/clip_scale_reduce/squared_sum_reduce) failing to JIT. Next thread: verify via the real apr finetune -m qlora CLI whether these degrade real training on current main.

🤖 Generated with Claude Code