gemcode — teaching a small model to code, step by step

This is a learning project that takes an open-weights model (Gemma 4 12B) and makes it better at writing Python, using the same two-phase recipe that frontier labs use — supervised fine-tuning (SFT) followed by reinforcement learning (GRPO) — and then measures exactly what each phase contributed. Everything is sized to run on a single consumer GPU (RTX 5080, 16GB VRAM).

Results

Pass@1 by training stage, regenerated by python plot_results.py from real benchmark output — never hand-entered numbers. Bars fill in as each stage completes (base → SFT → GRPO).

pass@1 by stage (GRPO column lands once that stage runs):

Benchmark	Base	SFT	Δ (SFT − Base)
HumanEval	81.7%	90.9%	+9.2
HumanEval+	78.7%	87.2%	+8.5
MBPP	82.3%	81.2%	−1.1
MBPP+	70.6%	68.0%	−2.6

SFT bought a big jump on HumanEval (+9 points, and it holds up under the stricter HumanEval+ tests) but slightly regressed MBPP. That's a real and instructive result, not noise: the training traces are free-form agentic coding transcripts, much closer in shape to HumanEval's "complete this function" tasks than to MBPP's short, terse spec-to-snippet prompts — so the style transferred where it matched and cost a little where it didn't. It's exactly the kind of gap GRPO, which trains directly against MBPP-style problems with real pass/fail rewards, is set up to close.

The gap between each benchmark and its + variant (e.g. MBPP 82.3 → 70.6) is the honesty tax: those extra tests catch solutions that passed the sparse original tests by luck.

The big picture

 base model ──► 1_sft.py ──► 2_grpo.py ──► final model
 (Gemma 4 12B)   imitate a      practice with
                 stronger       pass/fail
                 model          feedback
        └──────────── 3_eval.py scores all three ────────────┘

An analogy: SFT is studying worked examples from an expert; GRPO is doing practice problems and getting told which of your attempts were right. Both matter, and they teach different things — which is exactly what the eval is set up to reveal.

Phase 1 — SFT: learning by imitation (1_sft.py)

A base instruction-tuned model already writes decent code. SFT nudges it toward the style and habits of a much stronger model by training it to reproduce that model's outputs, token by token.

• Data: DavidrPatton/Fable-5-GLM-5.2-Traces — ~10.5k transcripts of large frontier models (Fable 5, GLM-5.2) solving real coding tasks, including their step-by-step reasoning.
• Loss masking: we only compute loss on the assistant's tokens. The model shouldn't waste capacity learning to predict the user's questions.
• QLoRA: a 12B model in normal (16-bit) precision needs ~24GB just to store, more to train — far over our 16GB budget. Two tricks fix this:
  • Quantization (the Q): store the frozen weights in 4-bit precision, shrinking the model to ~8GB.
  • LoRA: don't update the 12-billion-parameter model at all. Instead bolt small trainable "adapter" matrices onto each layer (a few tens of millions of parameters) and train only those. The saved checkpoint is just the adapter — a few hundred MB, not a full model copy.
• Unsloth provides fast fused kernels so this trains ~2x quicker than stock HuggingFace.

Phase 2 — GRPO: learning from pass/fail (2_grpo.py)

SFT can only make the model imitate. Reinforcement learning lets it improve beyond its examples, because the feedback is about whether the code actually works, not whether it matches a reference answer.

GRPO (Group Relative Policy Optimization — the method behind DeepSeek-R1) works like this for each training problem:

1. Sample a group of completions (we use 4) from the current model.
2. Score each one: extract the code, run it against the problem's unit tests in a sandbox. All tests pass → reward 1.0; code at least runs without crashing → 0.2; broken → 0.0.
3. Compare completions within the group: nudge the model toward whatever the above-average ones did, and away from the below-average ones.

The beauty of coding as an RL task is the reward is verifiable — a unit test is ground truth. No learned "reward model" that can be gamed, no human labelling.

• Training problems: MBPP (crowd-sourced Python problems with unit tests), train + validation splits only — see the contamination note below.
• Sandbox: sandbox/execute.py runs each candidate in a subprocess with a timeout. It is isolation-lite, fine for MBPP-scale snippets; don't point it at untrusted code from the internet.

Phase 3 — Eval: what did each stage buy? (3_eval.py)

We score three checkpoints — base, after-SFT, after-GRPO — on benchmarks the model never trained on:

• HumanEval (164 problems) and MBPP-test (~378 problems): complete a Python function so its unit tests pass.
• The “+” variants (via EvalPlus) re-score the same solutions against ~80x more tests, catching answers that only passed the originals by luck. Plus scores are always lower and more honest.
• Metric — pass@1: the fraction of problems solved on the first (and only) attempt, with greedy decoding. No retries, no cherry-picking.
• BigCodeBench (optional --bigcodebench flag): harder, library-heavy tasks; its harness wants to execute inside its own docker image.

The contamination rule — the one eval principle worth tattooing on: never train on your test set. GRPO trains on MBPP train+validation only; HumanEval and MBPP-test are never seen in training. Otherwise the before/after chart would measure memorization, not skill.

Running it

Hardware assumed: one NVIDIA GPU with 16GB VRAM, CUDA driver installed (works fine under WSL2).

# one-time setup
sudo apt install build-essential      # Triton JIT-compiles GPU kernels and needs a C compiler
uv venv .venv && uv pip install -p .venv/bin/python -r requirements.txt
source .venv/bin/activate

# the pipeline, in order
python 3_eval.py --model unsloth/gemma-4-12b-it --stage base   # baseline first!
python 1_sft.py                                                # a few hours
python 3_eval.py --model checkpoints/sft/final  --stage sft
python 2_grpo.py                                               # several hours
python 3_eval.py --model checkpoints/grpo/final --stage grpo

# regenerate the chart after any eval
python plot_results.py

Each eval pass is ~540 greedy generations on a 4-bit 12B — expect 1–2 hours.

Files

File	Role
1_sft.py	QLoRA fine-tune on the traces dataset
2_grpo.py	GRPO with sandboxed unit-test rewards
3_eval.py	generates solutions, scores them with EvalPlus / BigCodeBench
plot_results.py	turns results/ into assets/eval_results.png
sandbox/execute.py	subprocess-with-timeout code runner (the reward signal)
checkpoints/, results/	created at runtime, gitignored

Gotchas we hit (so you don't have to)

• The traces dataset can't be loaded whole. The repo mixes three JSONL files with incompatible schemas and load_dataset() dies with a CastError. Load unsloth_messages.jsonl specifically; message content is a list of {type, text} parts that must be flattened to strings.
• Gemma 4 12B needs transformers ≥ 5.14 — older versions don't know the gemma4_unified architecture.
• The Gemma 4 “tokenizer” is a multimodal processor. Call it with text=... as a keyword; a positional string is treated as an image path.
• Gemma 4 thinks before answering. Its chat template opens a thought channel, so completions lead with reasoning. Always extract the last

• TRL renames config fields between releases — e.g. SFTConfig's max_seq_length became max_length in 0.24.
• bigcodebench is installed separately — it pins an old vllm that breaks a normal pip install -r requirements.txt, which is why it isn't in there.
• No merging on 16GB. Eval generates with the 4-bit model + adapter directly; a merged bf16 12B (~24GB) wouldn't fit on the GPU.
• Chart palette: the three stage colors are the first three slots of a CVD-validated categorical palette (blue #2a78d6, orange #eb6834, aqua #1baf7a) — safe for colorblind readers as a set.