Fine tuning

Fine-tuning: GRPO History

The history of DeepDelve's own local fine-tuning effort, from the first feasibility smoke test through the first trained round's disqualification. Continues into the wiki's Changelog for the later combined multi-objective rounds and their own disqualification.

GRPO training confirmed working on this hardware (2026-07-14)

Before scoping any real training, a bare feasibility smoke test on the RX 9060 XT (RDNA4, ROCm 7.0.2+): a fresh venv on the root ext4 disk (the project's own NTFS mount has zero symlink support, python3 -m venv fails there even with --copies), torch with ROCm support, trl/peft/ accelerate. One real blocker hit and fixed: GRPOTrainer's auto device mapping tried to shard the model across both the discrete GPU and the Ryzen iGPU, crashing with an invalid device function error; fixed by pinning HIP_VISIBLE_DEVICES=0. Real GRPO training then ran end to end, 2 steps on a toy model with a mechanical exact-match reward, reward mean going 0.5 to 1.0, confirming real policy gradient improvement on this exact card with no Docker involved. Real disk cost noted for next time: the venv alone was 13GB, entirely on root, cleaned up immediately after the test.

The first real training round: thin_coverage only (2026-07-18)

Scoped after qwen3:4b and qwen3:8b both failed the bake off the same way: real research happened, but the model didn't act on the completion check's corrective nudge, just repeating a canned "research is complete" response until the retry budget exhausted. A clear, narrow, reward- able target. finetune/reward.py built three reward dimensions calibrated against real examples from that exact bake off session, not synthetic cases. finetune/extract_dataset.py pulled real (context, response) pairs from the project's own run history, cross-referencing run state against persisted session logs; a real bug in the first version (matching only by query text, silently pairing a run with the wrong session since this project's own benchmark queries get re-run verbatim across many sessions) got caught and fixed by also requiring session-start proximity.

The extracted real data was thin, only 5 examples, so the dataset got expanded two ways. A live pilot batch on two brand new topics outside the project's own standard benchmarks confirmed the failure is topic general, not query-specific, though yield was real but low, only one of two new topics actually tripped it. The bigger lever: GRPO doesn't need labeled response pairs the way SFT does, since the model generates its own completions at training time. A new synthetic prompt generator called the real, production check_thin_coverage function directly against varied but realistic scenarios, real production code producing the nudge text, only the underlying scenario was synthetic. A real modeling bug got caught while building it: a 50/50 covered split doesn't trip the check by design, and 10 of the first 20 scenarios were accidentally written that shape, producing nothing. Fixed. Total dataset grew from 5 examples to 348 lines across real and synthetic data.

A literature check for a training-methodology caution before training: "Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration" (arXiv:2604.02869), trained on the same Qwen3 family, found naively dense per-turn rewards catastrophically degraded performance by up to 14 points versus a sparse outcome reward, a structural risk of denser rewards, not a tuning mistake. Directly relevant caution for any future reward expansion beyond a single structural signal.

Training itself ran against Qwen/Qwen3-4B, LoRA rank 16, ~73 minutes, VRAM stable at 15.79 of 17.1GB. Two real reward bugs caught and fixed before training by reading actual base model completions rather than trusting the reward curve: a missing explicit thinking-disabled flag, and a false positive where the reward scored "narrates intent to re-delegate without ever calling the tool" as a pass. Evaluated on 8 held out prompts, 5 real plus 3 genuinely unseen topics: base model 6 of 8, fine tuned model 8 of 8, including sensible, non-degenerate tool calls on all 3 unseen topics, real generalization, verified by reading the actual generated instructions, not just the score.

Live benchmark: the targeted fix worked, a different failure dominated (2026-07-18)

The LoRA got merged, converted to GGUF, and deployed, then run against the exact benchmark query that had disqualified mistral-nemo and both plain Qwen3 sizes. Confirmed zero thin_coverage stalls anywhere in the run, the exact narration-without-action failure that disqualified the base model never recurred. But a second, untouched failure mode dominated: cross-referencing every cited URL against what was actually fetched found 0 of 8 citations in the final report were ever really fetched, 100% fabricated, and the 4 real, correctly grounded findings that did exist in findings.md were silently dropped from the report entirely in favor of the fabricated content. Disqualified overall, but a genuine, narrow, confirmed success on the one behavior it targeted.

Root caused the same day, before scoping the next fine-tuning candidate, per the project's own "structural fix before fine-tuning" rule. Two task branches never called the fetch tool at all, snippet only, yet their synthesis text confidently cited specific URLs from the search results as verified. The upstream defense for exactly this did fire correctly, appending a verification warning, but a separate finding-summary truncation step sliced the warning off before it ever reached FindingsWriter, since both flagged findings measured exactly the truncation limit with zero trace of the warning. A real defense layer that gets silently truncated away is worse than none, it looks like protection while doing nothing. Fixed by reserving the warning's own length outside the truncation budget so it can never be sliced off regardless of body length. Re-tested against the identical benchmark: real, measurable improvement, 3 of 9 cited URLs now traced to real fetches versus 0 of 8 before, but the other 6 still fabricated, all from tasks where the model had genuinely no real alternative source and cited a flagged URL anyway rather than acknowledging the gap. A more precise, now evidence-backed case for the next fine-tuning round: teach the model specifically not to cite a URL its own evidence already named as unfetched.

Public dataset research and a training recipe (2026-07-18)

Several genuine, downloadable multi-turn tool-calling corpora were found and evaluated: Toucan-1.5M (1.5M real trajectories from 495 live MCP servers, the closest structural match to this project's own MCP-based tooling), APIGen-MT-5k (human-verified multi-turn, closest shape to this project's own delegation pattern), xlam-function-calling-60k, and a Hammer irrelevance-detection subset directly useful for teaching a clean, no-narration stop. No standalone public dataset exists for the re-planning/self-correction scenario specifically, confirming this project's own small set of real extracted examples is more valuable than anything public for that one dimension, even though too few to carry general reliability alone. Recommended recipe, matching the pattern used by several of those same datasets' own papers: an SFT/LoRA warm start on a subsample of the public corpora for general schema-compliant tool calling, then a small final GRPO pass on this project's own real logs for the specific behavior gap that actually matters. The warm start step was deprioritized as extra scope and never run; training went directly on the real plus synthetic dataset instead, per the section above.

Status

This first round (thin_coverage only) is superseded by the later multi-objective combined rounds, see the Changelog's "2026-07-28: fine tuning resumed" and "clean re test" entries for the 7-dimension round's own held out results and its eventual live disqualification on citation fabrication and writer convergence, dimensions the combined reward never targeted. The standing methodology rule that came out of this whole arc: every new fine-tuning objective folds into one combined multi-objective GRPO retrain off the same raw base checkpoint, never a separate, isolated single-purpose LoRA, since separately trained adapters cannot be merged or stacked cleanly. As of 2026-08-19, no correctness gap blocks resuming fine-tuning per this rule; a new round is blocked on a different, harder problem, no viable small base model has yet passed a real live benchmark to fine-tune in the first place.