Does bootstrapped self-improvement need a ground-truth evaluator?

An empirical study on code generation. We test a simple hypothesis:

A self-improvement ("bootstrap") loop only converges when its evaluator is grounded in reality (real execution / unit tests) — not a self-referential proxy (the model judging itself, or a tunable surface-feature ranker).

Across five experiments on MBPP with Qwen2.5-Coder (1.5B / 3B), the result is an honest map rather than a clean win:

Claim	Status
A self-referential proxy evaluator carries no usable signal	Solid. A solver-as-judge ranks "which candidate is fully correct" at AUC ≈ 0.50 (n≈26.8k pairs); a tunable surface-feature ranker transfers 0.000 to held-out (McNemar p=1.0).
Reality-grounded selection beats proxy selection	Directional, not significant. On a model big enough to use the signal (3B), reality > proxy by +0.032; but the effect is small and swings 0.012↔0.032 across runs.
The mechanism ("carrier") must itself work	Hypothesis. On the 1.5B, every carrier we tried failed: LoRA fine-tuning hurts (catastrophic forgetting), a plastic fast-weight is inert. Only retrieval-augmentation on the 3B helped at all.

No over-claim ("the law is proven"), no under-claim ("nothing worked"). We learned cheaply where the signal is solid, where it is only directional, and where the bottleneck actually sits.

Why this matters

"Self-improving" / "bootstrapping" LLM loops are popular (STaR, self-rewarding, self-critique). This study isolates the variable that those loops depend on but rarely ablate: what grades the candidates. If the grader is the model itself (a proxy), there is nothing to climb toward — the bottleneck is signal-availability, not optimizer-tunability. Reality (unit tests) carries the signal; the model's own judgment does not.

Method

• Benchmark: MBPP. Each task's tests are split into public (shown to the solver / used as the reality-filter) and hidden (held out, used only to score competence). Tasks are split train/test.
• Solver: Qwen/Qwen2.5-Coder-1.5B-Instruct (and -3B-Instruct for the size sweep), bf16.
• Sandbox: each candidate is executed in a subprocess with a timeout against the test asserts.
• One variable per experiment: the evaluator (reality = unit tests vs proxy = solver-as-judge / surface-feature ranker). Everything else (retrieval, K, config, seeds) is held identical across arms.

The five experiments

Script	Experiment	Result
bootstrap_run.py	Exp-1 — tune a surface-feature ranker (len/AST/branches…) to pick the best candidate; does the train gain transfer?	Train 0.641→0.657, held-out +0.000 (McNemar p=1.0). Null: surface features can't capture "which public-passer is fully correct".
premise_check_B.py	Premise check — does a richer proxy (solver-as-judge, "rate 0-100 fully correct") separate correct from incorrect?	AUC 0.498 pooled / 0.504 held-out (n≈26.8k pairs). ≈ random. The proxy has no signal.
exp2_retrieval_bootstrap.py	Exp-2 — build a retrieval bank of solved (problem, solution) pairs; fill it by reality (unit-tests) vs proxy (judge); retrieve as few-shot.	1.5B: reality−proxy +0.017 (n.s.), banks net-hurt. 3B: reality−proxy +0.032, banks help (the model is big enough to use them).
t1_prep.py + t1_run.py	T1 — LoRA fine-tune on solutions selected by reality vs proxy (3 arms × 3 seeds).	base 0.569, reality-FT 0.534, proxy-FT 0.528. Both FT arms below base — the FT carrier net-hurts the 1.5B (forgetting).
p1_run.py	P1 — a non-gradient plastic fast-weight (Hebbian outer-product readout) updated by reality vs proxy.	reality = proxy = 0.5655 exactly. Inert (the readout bias is too weak to steer generation).

generate_cache.py and build_bench.py build the candidate cache and the bench-holder; embed_problems.py computes problem embeddings (via a local Ollama embedding model) for Exp-2 retrieval.

Reproduce

pip install -r requirements.txt           # torch, transformers, peft, numpy
# 1. get MBPP -> mbpp.jsonl  (https://github.com/google-research/google-research/tree/master/mbpp)
python build_bench.py                      # -> bench_holder.json (public/hidden + train/test split)
python generate_cache.py                   # -> cand_cache.jsonl  (N candidates/task, sandboxed eval)  [GPU]
python bootstrap_run.py                    # Exp-1
python premise_check_B.py                  # Premise check
OLLAMA_BASE_URL=<your-Ollama-endpoint> python embed_problems.py  # -> problem_embs.json
python exp2_retrieval_bootstrap.py --model Qwen/Qwen2.5-Coder-3B-Instruct   # Exp-2
python t1_prep.py && python t1_run.py --arm base && python t1_run.py --arm reality --seed 0  # T1
python p1_run.py --arm reality --seed 0    # P1

The committed *_result.json / *_results.jsonl files are the actual outputs reported above. The large regenerable inputs (MBPP, candidate cache, embeddings) are git-ignored — regenerate via the pipeline.

Caveats

• Small models (1.5B / 3B) and one benchmark (MBPP). The reality > proxy effect is real but small and not individually significant at this scale; confirming it would need a larger model or more tasks.
• The proxy-has-no-signal result is the strongest finding and is statistically solid (large n).
• "The carrier must work" is offered as a falsifiable hypothesis (measure carrier helps = reality ≥ no-evaluator, and reality beats proxy = reality > proxy, as two independent metrics — on a working carrier, reality = proxy would refute it), not a proven claim.

License

Apache License 2.0.