Skip to content

Raw results: Aider Polyglot slice on qwen3.5:4b

Choose a tag to compare

@mattbaconz mattbaconz released this 13 Nov 13:33
· 7 commits to main since this release
0310584

polyglot-qwen3.5-4b-raw-results.zip below holds 2,632 files: 5.4 MB zipped, 32 MB unpacked. SHA-256: aa2f748cc55df43b7888968bb01616f2c55a32735fd94c4c846c6cae33c62b4b.
What follows is the README inside it.

These are the per-exercise results behind the benchmark table on
phonton.dev and in the
phonton-cli README: every
prompt, model reply, diff and test log from seven runs made between
17 October and 6 November 2026. Published on 13 November 2026.

What was run

Exercises 17 each in Python, JavaScript and Rust (51 in all) from Aider's polyglot-benchmark at commit 7e0611e, chosen by the runner's seeded order
Model qwen3.5:4b through Ollama 0.34.2, served with a 16,384-token context
Machine One Windows 11 laptop: RTX 4050 Laptop GPU (6 GB), Ryzen 7 7735HS, 15.2 GB RAM
Runner benchmarks/polyglot/run.mjs in phonton-cli. Each run's env.txt names the commit it ran from
Tests Visible to every arm. For scoring, the exercise's original tests are restored and run

The three arms:

  • bare: the model alone, with Aider's prompt and two tries. The runner sends
    temperature 0 and no seed.
  • opencode: OpenCode 1.14.19, run --pure, on the same Ollama endpoint and
    model. The runner sets no temperature for it.
  • phonton: Phonton goal --local, with the exercise's test command as its
    check. Calls run at temperature 0. Since 0.27.0, a run whose model repeats a
    reply switches to temperature 0.7 with a recorded seed.

How a row is scored

Each line of a results.jsonl file is one exercise for one arm.

  • claimed: the arm counted the exercise as done. For the bare model that is
    any written answer. For OpenCode it is exit code 0 without a timeout. For
    Phonton it is a run that ended review-ready, which means its own checks
    passed. Only Phonton makes an explicit claim. The bare model and OpenCode
    returned an answer or exited cleanly.
  • passed: the exercise's original tests pass on the result.
  • false_pass: claimed is true and passed is false.
  • status, seconds, tokens_in, tokens_out, changed and
    tests_touched describe how the run ended and what it used and changed.

The numbers

run exercises counted as done of those, failed the tests passed the tests
17 October, bare model 51 43 37 6
17 October, OpenCode 1.14.19 51 51 50 1
17 October, Phonton 0.25.0 51 1 0 2
31 October, bare model 51 44 41 3
31 October, Phonton 0.26.0 51 2 0 3
November, Phonton 0.27.0, repetition 1 51 2 0 3
November, Phonton 0.27.0, repetition 2 51 2 0 3

python tally.py recomputes this table from the rows and stops with an error
if they no longer add up to it.

Each row is one run per exercise. Phonton passed fewer exercises than the bare
model in the first run and the same number in the rerun. The bare model's own
count moved from 6 to 3 between two runs at temperature 0, so differences of
that size are inside run-to-run noise.

The Phonton 0.27.0 rows join two nights: Python and Rust from 2 November,
JavaScript from 6 November. The next section says why.

Rows that are here but not counted

  1. OpenCode rows in 2026-10-31-rerun-bare. OpenCode was not installed
    for that run. All 51 rows ended with exit code 1 in under a second, before
    any model call (Cannot find module in stderr.txt). OpenCode has one
    valid run, the one of 17 October.
  2. JavaScript rows in the two 2026-11-02 folders. The npm cache on the
    machine had been emptied, so the offline install of the test dependencies
    failed and 16 of 17 rows ended dependency_unavailable before any model
    call. JavaScript was run again on 6 November, twice, and those rows are the
    ones counted.
  3. Phonton's Rust rows before 0.27.0 (the first run and the 31 October
    rerun). On this machine cargo.exe is a link to rustup.exe, and Phonton
    started it under the wrong name, so every Rust check failed before cargo
    ran. The comparison for those two runs holds on the 34 Python and
    JavaScript exercises. Phonton 0.27.0 fixed this, and its Rust rows are
    valid: none passed. The bare model's and OpenCode's Rust rows are not
    affected.

Other things to know

  • javascript/ledger is a refactoring exercise whose tests pass before any
    change. Phonton changed nothing there, so it is a pass without a claim in
    every Phonton run. It is also the one "pass" among the invalid OpenCode rows
    of 31 October.
  • In the first run, 18 of Phonton's 51 rows stopped on its free-memory reserve
    (resource_pressure) and 2 goals were longer than its goal limit at the
    time. All of them count as failures.
  • In the first run OpenCode could see a skill that was installed on the test
    machine, and the model loaded it once (rust/rust/forth). That call is in
    the log. The skill's text is not (see the last section).
  • In the 31 October rerun, the bare-arm rows that ran before 22:20 local time
    were timed while another model server held 6.2 GB of memory. The pass and
    fail verdicts are not affected. Do not compare those seconds.
  • Folder names use the local dates of the write-ups. meta.json has each
    run's start time in UTC, which can be the day before.

Layout

<run>/
  env.txt                     versions, hardware and the runner's commit
  runner-<language>.log       the runner's own output
  ollama.log, ollama.err.log  the model server's log (reruns only)
  calibrate.log               Phonton's calibration output (Phonton runs only)
  <language>/
    meta.json                 versions, model, hardware and arguments
    results.jsonl             one row per exercise and arm
    summary.md                the runner's summary
    <language>/<exercise>/<arm>/
      prompt.md               what the arm was asked
      stdout.txt, stderr.txt  what it printed
      diff.patch              what it changed
      tests.txt               the original tests run on the result
      evidence.json           Phonton's record of the run (Phonton only)
      opencode-config.json    the config OpenCode ran with (OpenCode only)

What was changed before publishing

  • The Windows account name in paths is replaced by user, as in
    C:\Users\user (5,690 places).
  • The folder the benchmark was run from is replaced by C:\bench (9,676
    places), including paths that a log had cut short.
  • The text of the skill mentioned above is replaced by a note that gives its
    length (13,218 characters), in
    2026-10-17-first-run/rust/rust/forth/opencode/stdout.txt.
  • 19 log files were UTF-16 and are UTF-8 here.

Nothing else was changed. No row and no file was left out, and the rows that
are not counted are here as they were written.