LLM-IMPROVER

Can a harness improve any off-the-shelf LLM on a benchmark by verified means — and how much of the gain is the model versus the scaffolding around it?

This is a small, single-GPU, pre-registered study that treats a benchmark as a fitness function and asks a disciplined question: bolt a stock LLM in as a swappable socket, measure its bare score on a held-out split, then try to raise that score by any means a frozen verifier will certify — and report the delta (system − bare), not the absolute number. Every lever is measured one at a time. Negative results are reported as prominently as positive ones, because the point is to find out what actually moves the needle, not to win a leaderboard.

Benchmark: ARC-AGI (few-shot grid-transformation induction). Hardware: a single RTX 5090 (32 GB) + 128 GB RAM. Models: Qwen2.5-3B, Qwen3.6-27B, gemma-4-31B, Llama-3.3-70B — served locally via HuggingFace and llama.cpp (including a 70B run via GPU+RAM offload).

---

The one-screen result

Held-out delta on the frozen ARC training split (exact-match, verifier never sees the test answer):

lever	what it is	result
model capacity	swap the base LLM (3B → 31B → 27B)	bare 3.2% → 9.5% → 34.9% — the dominant lever, ~10×
operator induction	induce a program from the demos, verify on all, apply	+2.4% held-out delta (first real, un-gameable win)
iterative refine	propose → verify → refine on the actual diff	works — 2 programs solved via round-2 refinement on a capable model
best-of-N search	N candidates/turn, verifier keeps any that pass	+1 verified per doubling (0→2→3 as N=1→8→16); null on a below-floor model
object-ontology structure	reframe grids as objects/roles/relations	null at every scale
prompt-diversity board	one model, multiple solver "lenses"	null vs concentrated best-of-N
multi-model message board	3 different models collaborate via a shared verifier-grounded board	null — collaboration doesn't compound; value ≈ best single proposer

The honest through-line: on this task, capacity and verified search are the real levers; structure, prompt-diversity, and multi-model collaboration each came back null. The scaffolding that works is the kind that lets the verifier keep more good samples — not the kind that tries to make the model "reason better" by reframing or discussion.

Full write-ups, with the reasoning behind each null, are in RESULTS_001a.md … RESULTS_001d.md.

---

Why it's built this way (the invariants)

The study is designed to be hard to fool, because the failure mode of "self-improving AI" demos is Goodharting the metric. Five rules:

1. Held-out is sacred. A lever only touches the test input if its program reproduces every training demo. No peeking, ever.
2. The delta is the product. system − bare on held-out. A better absolute score from a bigger model is not a lever result.
3. Model-swap or it's a trick. The base LLM is a socket; any lever must survive swapping it. (It does — the induction delta holds 3B→27B.)
4. One lever at a time. Each slice changes exactly one thing.
5. Baseline before levers. Measure bare first, always.

Disposition vocabulary used throughout (borrowed from the parent research program): SHIPPED (landed + verified consumer), PARTIAL (built, no delta yet), NULL (real, informative negative), RETIRED (removed with a reason). See METHOD.md.

---

What's in the box

file	role
arc.py	ARC loader, frozen dev/held-out split, exact-match scoring
llm_socket.py	swappable base LLM — HF backend and an OpenAI-compatible llama.cpp backend (LLMSocket / LlamaServerSocket); best-of-N sampling on both
induction_lever.py	operator induction over a small grid DSL + arc-dsl single-ops
arc_dsl_bridge.py	sandboxed executor for LLM-written arc-dsl programs (no builtins, exec timeout, grid-size cap)
arc_dsl_lever.py	LLM writes/refines arc-dsl programs, verifier-gated
sampled_lever.py	best-of-N + refine, near-misses ranked by a typed residual
structured_lever.py	object-ontology proposer (the structure arm)
typed_residual.py	classifies a failed program's diff into a named failure mode ("orientation wrong: expected rot90")
message_board.py	multi-model async board: models loaded one-at-a-time collaborate through a persistent, verifier-grounded board (70B via RAM offload)
board_view.py	renders a message board as a readable conversation
run_baseline.py / run_lever.py / run_pass2.py	measurement harnesses

The arc-dsl vocabulary is vendored from michaelhodel/arc-dsl (see arc_dsl/ATTRIBUTION.md); the frozen verifier and everything else is original.

---

Reproduce

# baseline: bare model on the frozen held-out split
python run_baseline.py --backend hf --model Qwen/Qwen2.5-3B-Instruct

# the induction lever's held-out delta
python run_lever.py --backend hf --model Qwen/Qwen2.5-3B-Instruct --source training

# best-of-N search, measured against greedy on the same model
python run_pass2.py --backend hf --model Qwen/Qwen2.5-3B-Instruct --lever sampled --samples 8

# a local llama.cpp model instead (OpenAI-compatible endpoint on :8080)
python run_pass2.py --backend llamacpp --model my-model --lever sampled --samples 16

# the multi-model message board on the hard residual
python message_board.py --source training --n 15 --rounds 2
python board_view.py --solved            # read what the board cracked

Each run writes a self-describing JSON to results/ and prints the bare / pass-1 / pass-2 deltas with attribution.

---

Scope & limits (read this before citing a number)

• Small scale, one benchmark. ≤70B models, hundreds of held-out tasks on ARC. This is a controlled study of levers, not a SOTA attempt.
• The deltas are small (1–3 held-out tasks) and ARC's exact-match scoring makes them noisy; llama.cpp continuous-batching isn't bit-exact, so the bare baseline wobbles ±1 task between runs. Treat the rankings as the finding, not the third significant figure.
• The nulls are scoped, not universal. "Structure didn't help" means on ARC induction, at this round budget, with these models — not "object representations are useless."
• The verifier is only as strong as the oracle. This works because ARC demos are an executable oracle. "Did the model do only what it claimed?" in general is undecidable; the honest deliverable is high-recall detection of claim/scope mismatch, not proof.

---

Part of a larger research program

This is one slice of an ongoing single-person research program on modular / self-expanding AI, run with the same discipline: pre-registered predictions, deterministic multi-seed reruns, and disposition-tagged notes (validated / null / partial / closed). The unifying thesis is that generalization comes from the basis / inductive bias, not from more compute over tokens (No Free Lunch) — so capability should grow by routing over inductive biases and composing frozen modules through canonical interfaces, not by monolithic gradient descent. LLM-IMPROVER is the "outside-the-model" corner of that: the LLM is one swappable, frozen organ; the levers that work are the ones that let a verifier keep more good outputs, not the ones that try to make the model reason differently.

That framing is why the nulls here are unsurprising. A companion result (attention routes linearly; the hard-to-compose capacity is nonlinear synthesis) predicts that reframing/discussion levers — which only re-route in-context content — can't manufacture a solution the model couldn't already produce; only a stronger proposer or more verified search can. Every result on this page is consistent with that.

Individual numbers in this repo reproduce on the hardware listed. Claims about the broader program are made only where a reproducible result backs them.

---

Built and documented in the open. If a result here doesn't reproduce, that's a bug worth reporting — the whole point is that the numbers are real.