Skip to content

Phonton CLI v0.22.2

Choose a tag to compare

@github-actions github-actions released this 09 Oct 09:37
· 12 commits to main since this release
4dc5f6b

phonton-cli v0.22.2 — Half the cost, fewer false passes

Same results at half the cost

Phonton used to plan a test-writing subtask for every new function, whether
or not the goal asked for tests. Those generated test files were most of the
tokens of a one-method goal, and the project's own tests already run on
every edit. Test subtasks are now planned only when a goal asks for tests.

Cloud smoke suite, DeepSeek flash, 12 tasks x 3 runs, hidden tests:

Build Accepted Tokens Cost Wall time
0.22.1 36/36 302,442 $0.186 852 s
0.22.2 36/36 193,129 $0.096 381 s

Write "with tests" in a goal to get them.

First run

  • phonton goal asks once in a terminal before running the project's own
    build and tests. Without a terminal it stops before any model call unless
    --allow-host-checks is given; it used to spend tokens and then fail.
  • phonton models and phonton models catalog print what is installed, what
    fits this machine, and the next step (--json for the full output).
  • phonton review shows "review-ready" or "failed: reason" and paths
    relative to the repository.

Fewer false passes on small models

A local model is only useful if "review-ready" means something. This release
runs the local-model smoke suite in CI and fixes what it found.

Local model results

benchmarks/smoke, qwen2.5-coder:3b in Ollama on a GitHub Actions
ubuntu-latest runner (4 vCPU, 16 GB, no GPU), 3 runs per task, scored by
hidden tests. A "false pass" is a run Phonton marked review-ready that the
hidden tests then failed.

Fixture 0.22.1 accepted 0.22.1 false passes 0.22.2 accepted 0.22.2 false passes
todo-api (4 small files) 9/12 3 12/12 0
ledger (339-line module) 1/12 5 2/12 2
shop (28 KB module) 1/12 0 1/12 0

The other runs ended without a candidate: the harness refused edits that
failed checks or made no change instead of handing them over. The remaining
false passes are real semantic misses (a $ currency format, a regex that
still rejects parentheses) that only tests covering the new behavior catch.
A 3B model is good for small, focused edits; larger ones need a larger model.

Fixes

  • Values scope the edit. A goal that changes a value ("from 120 to 80")
    is scoped to the files containing it, plus files the goal names. The 3B
    model used to edit store.js, which matched through a local variable named
    todo, and the existing tests passed. todo-title-limit: 0/3 to 3/3.
  • Named identifiers must appear. When the goal calls or quotes an
    identifier that does not exist yet (balanceOn(account, date)), a
    candidate that does not add it is rejected and the model is told why. It
    used to edit parseDate instead and pass the existing tests.

Diagnostics

PHONTON_LOG=debug (or any tracing filter, e.g. phonton_worker=debug)
appends to phonton.log in the Phonton home: each worker attempt, why
verification rejected it, and hunks Phonton recounted or moved. On the
cloud ledger and shop tasks (DeepSeek flash, 24 runs, all accepted) it
showed 13 hunks recounted and 14 moved across 28 worker diffs; each would
have failed verification before 0.22.1.

Reproduce

Run the workflow .github/workflows/local-smoke.yml (inputs: a released
version or source, the model, runs per task), or locally:

node benchmarks/smoke/run.mjs --arm local --runs 3 --runtime unverified

Install

npm install -g phonton-cli
phonton doctor