fixpoint

Empirically characterizing the convergence of iterative AI code repair — the generate → verify → repair loop.

Practitioners routinely wire an LLM into a loop: generate a solution, run the tests, feed failures back, repair, repeat until green. Folk wisdom says "it usually converges." fixpoint treats that loop as a discrete dynamical system over candidate programs and measures what actually happens: does it reach a passing fixed point, stall, or oscillate?

This repository is deliberately honest about what it proves. It ships:

• a small, dependency-light harness (fixpoint/),
• a data-only benchmark of 43 original Python tasks,
• a fully reproducible, deterministic study (the CI headline), and
• an optional real-LLM study via a local Ollama model (non-deterministic, not used for CI assertions).

We say "empirically characterized," not "formally proven." A lightweight Markov-chain model of the loop is sketched in docs/METHODOLOGY.md with its assumptions stated plainly — it is a model, not a theorem about transformer inference.

---

What it measures

Each repair run is classified by the harness (never by the model) into one of four outcomes, from the observed trajectory of failing-state signatures:

outcome	meaning
CONVERGED	reached all-tests-pass at some iteration ≤ K (a passing fixed point)
STALLED	emitted a byte-identical failing candidate twice (non-passing fixed point)
OSCILLATING	a failing signature recurred after the code changed (a limit cycle)
EXHAUSTED	hit the iteration budget K, still failing, no repeat signature

A signature is blake2b(compiles?, sorted failing test ids, top error line) — the observable shape of the failing state. See docs/ARCHITECTURE.md and docs/METHODOLOGY.md.

---

Headline result (deterministic, reproduced in CI)

The headline uses the ScriptedModel: a controlled, no-network model of a repairer that replays a fixed per-task trajectory. Trajectories are assigned by a fixed hash of the task id into converge / oscillate / stall / exhaust regimes, so the mix is deterministic and not cherry-picked. This makes the study byte-for-byte reproducible on any machine — it is what CI runs.

Running each task with the failing-test output fed back into the repair step, versus without (the model is told only how many tests failed):

condition	converged	convergence rate
error text fed back	22 / 43	51.2 %
no error text	1 / 43	2.3 %
absolute lift		+48.8 pp

Convergence by difficulty (with feedback): easy 46.2 %, medium 77.8 %, hard 16.7 %. Full tables and plots regenerate into results/RESULTS_scripted.md.

Read this before quoting the +48.8 pp number

For the ScriptedModel, the no-feedback trajectories are authored to be worse — that is an explicit modelling choice, declared in docs/METHODOLOGY.md and scripts/build_scripts.py. So the +48.8 pp figure demonstrates two things honestly:

1. the harness correctly classifies all four convergence regimes, and
2. it measures the feedback ablation end-to-end and reports it faithfully.

It is not an empirical claim about how much error messages help real LLMs. The magnitude of the feedback effect is a property of the controlled model, by construction. The real effect can only be estimated with a real model — see the next section.

---

Optional real-model study (Ollama, non-deterministic, not in CI)

fixpoint can drive a real local LLM (default qwen2.5-coder:7b-instruct) at temperature=0, fixed seed, via the Ollama HTTP API. Because real inference is not perfectly reproducible across builds/hardware, these numbers are reported as measured, with the model id recorded in every run, and are never used in CI assertions.

In the single committed run (results/RESULTS_ollama.md), the model one-shot-solved most tasks — it converged on the very first generation (iteration 1) for 40 / 43 tasks, under both conditions:

condition	converged	rate	iters-to-converge
error text fed back	40 / 43	93.0 %	all at iteration 1
no error text	40 / 43	93.0 %	all at iteration 1
absolute lift		+0.0 pp	—

Read this honestly: because a 7B coder model gets these small tasks right on the first try, the repair loop rarely engaged at all, so this run tells us almost nothing about repair convergence and shows no feedback effect. The remaining 3 tasks were all hard (group_anagrams, spiral, topo_sort) and stalled/oscillated. This is a useful counterpoint to the scripted headline, not a confirmation of it: to actually stress the repair loop with a real model you need harder tasks (or a weaker model) where the first generation fails. A partial re-run on this machine reproduced the same one-shot behavior for the first 36 tasks before an unrelated socket timeout (local inference-server contention) aborted it — consistent with the committed file, but again, non-deterministic and excluded from CI.

To reproduce (requires a local Ollama with the model pulled):

ollama pull qwen2.5-coder:7b-instruct
python -m fixpoint.cli run --model ollama:qwen2.5-coder:7b-instruct --both-conditions
python scripts/make_report.py results/runs_ollama_qwen2.5-coder_7b-instruct.jsonl --label ollama

---

Install & reproduce

# install (editable, with dev extras)
make install            # or: pip install -e ".[dev]"

# unit tests (22 tests)
make test               # or: pytest -q

# benchmark soundness gate + deterministic study
make study              # validates benchmark, then runs the scripted study

# regenerate RESULTS_scripted.md (+ PNG plots if matplotlib present)
make report

Or with Docker (hermetic, no Ollama needed):

docker build -t fixpoint .
docker run --rm fixpoint          # reproduces the deterministic scripted study

CLI directly:

python -m fixpoint.cli list                                  # list the 43 tasks
python -m fixpoint.cli run --model scripted --both-conditions

---

Benchmark soundness

scripts/validate_benchmark.py (run in CI) asserts, for every task, that a reference solution passes all hidden tests and the initial (buggy/stub) code fails at least one — so no task is trivially green. Current status: all 43 reference solutions pass, all 43 initial solutions fail.

---

Project layout

fixpoint/            harness: task, verify, model, loop, study, cli
benchmark/tasks/     43 tasks as JSON (data, not code)
benchmark/scripts.json  deterministic ScriptedModel trajectories
scripts/             benchmark builders, reference solutions, validator, reporter
results/             raw runs (JSONL), summaries (JSON), RESULTS_*.md, plots
docs/                ARCHITECTURE.md, METHODOLOGY.md
tests/               unit tests for the harness

Limitations (see METHODOLOGY for the full list)

• 43 tasks → wide confidence intervals; rates are indicative, not precise.
• Single model family for the real-model run; results may not generalize.
• Convergence means "passes our hidden tests," not "provably correct."
• Oscillation is detected at (failing-tests, top-error) granularity — a modelling choice that matches the practical notion of "stuck."

License

COCL (Cognis Open Collaboration License). The benchmark tasks and their hidden test suites are original works authored for this project; no external dataset is redistributed. See LICENSE and NOTICE.