Repository navigation
Raw results: Aider Polyglot slice on qwen3.5:4b
polyglot-qwen3.5-4b-raw-results.zip below holds 2,632 files: 5.4 MB zipped, 32 MB unpacked. SHA-256: aa2f748cc55df43b7888968bb01616f2c55a32735fd94c4c846c6cae33c62b4b.
What follows is the README inside it.
These are the per-exercise results behind the benchmark table on
phonton.dev and in the
phonton-cli README: every
prompt, model reply, diff and test log from seven runs made between
17 October and 6 November 2026. Published on 13 November 2026.
What was run
| Exercises | 17 each in Python, JavaScript and Rust (51 in all) from Aider's polyglot-benchmark at commit 7e0611e, chosen by the runner's seeded order |
| Model | qwen3.5:4b through Ollama 0.34.2, served with a 16,384-token context |
| Machine | One Windows 11 laptop: RTX 4050 Laptop GPU (6 GB), Ryzen 7 7735HS, 15.2 GB RAM |
| Runner | benchmarks/polyglot/run.mjs in phonton-cli. Each run's env.txt names the commit it ran from |
| Tests | Visible to every arm. For scoring, the exercise's original tests are restored and run |
The three arms:
bare: the model alone, with Aider's prompt and two tries. The runner sends
temperature 0 and no seed.opencode: OpenCode 1.14.19,run --pure, on the same Ollama endpoint and
model. The runner sets no temperature for it.phonton: Phontongoal --local, with the exercise's test command as its
check. Calls run at temperature 0. Since 0.27.0, a run whose model repeats a
reply switches to temperature 0.7 with a recorded seed.
How a row is scored
Each line of a results.jsonl file is one exercise for one arm.
claimed: the arm counted the exercise as done. For the bare model that is
any written answer. For OpenCode it is exit code 0 without a timeout. For
Phonton it is a run that ended review-ready, which means its own checks
passed. Only Phonton makes an explicit claim. The bare model and OpenCode
returned an answer or exited cleanly.passed: the exercise's original tests pass on the result.false_pass:claimedis true andpassedis false.status,seconds,tokens_in,tokens_out,changedand
tests_toucheddescribe how the run ended and what it used and changed.
The numbers
| run | exercises | counted as done | of those, failed the tests | passed the tests |
|---|---|---|---|---|
| 17 October, bare model | 51 | 43 | 37 | 6 |
| 17 October, OpenCode 1.14.19 | 51 | 51 | 50 | 1 |
| 17 October, Phonton 0.25.0 | 51 | 1 | 0 | 2 |
| 31 October, bare model | 51 | 44 | 41 | 3 |
| 31 October, Phonton 0.26.0 | 51 | 2 | 0 | 3 |
| November, Phonton 0.27.0, repetition 1 | 51 | 2 | 0 | 3 |
| November, Phonton 0.27.0, repetition 2 | 51 | 2 | 0 | 3 |
python tally.py recomputes this table from the rows and stops with an error
if they no longer add up to it.
Each row is one run per exercise. Phonton passed fewer exercises than the bare
model in the first run and the same number in the rerun. The bare model's own
count moved from 6 to 3 between two runs at temperature 0, so differences of
that size are inside run-to-run noise.
The Phonton 0.27.0 rows join two nights: Python and Rust from 2 November,
JavaScript from 6 November. The next section says why.
Rows that are here but not counted
- OpenCode rows in
2026-10-31-rerun-bare. OpenCode was not installed
for that run. All 51 rows ended with exit code 1 in under a second, before
any model call (Cannot find moduleinstderr.txt). OpenCode has one
valid run, the one of 17 October. - JavaScript rows in the two
2026-11-02folders. The npm cache on the
machine had been emptied, so the offline install of the test dependencies
failed and 16 of 17 rows endeddependency_unavailablebefore any model
call. JavaScript was run again on 6 November, twice, and those rows are the
ones counted. - Phonton's Rust rows before 0.27.0 (the first run and the 31 October
rerun). On this machinecargo.exeis a link torustup.exe, and Phonton
started it under the wrong name, so every Rust check failed before cargo
ran. The comparison for those two runs holds on the 34 Python and
JavaScript exercises. Phonton 0.27.0 fixed this, and its Rust rows are
valid: none passed. The bare model's and OpenCode's Rust rows are not
affected.
Other things to know
javascript/ledgeris a refactoring exercise whose tests pass before any
change. Phonton changed nothing there, so it is a pass without a claim in
every Phonton run. It is also the one "pass" among the invalid OpenCode rows
of 31 October.- In the first run, 18 of Phonton's 51 rows stopped on its free-memory reserve
(resource_pressure) and 2 goals were longer than its goal limit at the
time. All of them count as failures. - In the first run OpenCode could see a skill that was installed on the test
machine, and the model loaded it once (rust/rust/forth). That call is in
the log. The skill's text is not (see the last section). - In the 31 October rerun, the bare-arm rows that ran before 22:20 local time
were timed while another model server held 6.2 GB of memory. The pass and
fail verdicts are not affected. Do not compare thoseseconds. - Folder names use the local dates of the write-ups.
meta.jsonhas each
run's start time in UTC, which can be the day before.
Layout
<run>/
env.txt versions, hardware and the runner's commit
runner-<language>.log the runner's own output
ollama.log, ollama.err.log the model server's log (reruns only)
calibrate.log Phonton's calibration output (Phonton runs only)
<language>/
meta.json versions, model, hardware and arguments
results.jsonl one row per exercise and arm
summary.md the runner's summary
<language>/<exercise>/<arm>/
prompt.md what the arm was asked
stdout.txt, stderr.txt what it printed
diff.patch what it changed
tests.txt the original tests run on the result
evidence.json Phonton's record of the run (Phonton only)
opencode-config.json the config OpenCode ran with (OpenCode only)
What was changed before publishing
- The Windows account name in paths is replaced by
user, as in
C:\Users\user(5,690 places). - The folder the benchmark was run from is replaced by
C:\bench(9,676
places), including paths that a log had cut short. - The text of the skill mentioned above is replaced by a note that gives its
length (13,218 characters), in
2026-10-17-first-run/rust/rust/forth/opencode/stdout.txt. - 19 log files were UTF-16 and are UTF-8 here.
Nothing else was changed. No row and no file was left out, and the rows that
are not counted are here as they were written.