Skip to content

0.7.0

Choose a tag to compare

@pham-canh pham-canh released this 04 Nov 09:51
· 61 commits to main since this release

[0.7.0] — 2026-10-25

oc-harness run now stops for the reasons it already claimed to stop for,
the read-only lanes no longer have a write-capable shell, and the token budget
counts the context a request reads instead of only its uncached part.

Fixed

  • A denied tool call was fed back as another turn, and the retry spent the
    window.
    experimental.continue_loop_on_deny was true, so opencode
    handed the guard's refusal back to the model and continued the loop. The
    measured session made 138 requests and 7,559,016 context tokens in 29
    minutes on six shared token window denials, each retry re-sending the
    whole prompt. The generated config now sets it to false, and the guard
    blocks a single request that re-sends more than OC_HARNESS_TURN_STOP
    (default 300k) with "do not retry".

  • The in-event shared-window stop did not refuse anything. When the
    step-finish sum crossed OC_HARNESS_BUDGET_WINDOW_STOP, the guard set
    windowSpent but never set windowWhy, so sharedWindowSpent() returned
    an empty reason and every tool call was allowed. The reason is set where
    the sum crosses, and the stop is written to the ledger with
    recordStop() so a new session cannot outlive the provider's window.

  • A corrupt ledger locked the machine for six hours, or vanished. It is
    now moved to budget.json.corrupt and the window starts empty; the broken
    bytes are kept as evidence. recordSpend() and recordStop() repair
    before writing instead of reading the corruption as an empty window.

  • The bundled ECC subagents were loaded on every request. Their
    descriptions came from prompts/agents/*.txt, were not lanes, and so were
    not covered by the stale-lane rule. They are removed from the generated
    agent map when that is the prompt, and named in the config header; a
    hand-written agent that reuses that prompt path is removed like the
    bundle's own, while any other prompt is left alone.

  • The token budget undercounted a cached session five to forty times, so it
    never fired.
    A step-finish part reports input, cache.read,
    cache.write, output and reasoning. The guard summed input alone -
    the part of the prompt that was not cached - while the cached prefix, which
    is what a long conversation actually re-reads, was ignored. One measured
    stream reported 844 uncached tokens against 36,480 cached ones on the same
    step, then 391 against 9,088. The 2M warn and 5M stop were therefore five
    to forty times the real spend, and the provider's window ran out before
    either fired. requestTokens() in src/budget.js now counts input + cache.read + cache.write for both the session total and the shared-window ledger; the
    toasts and refusals say "context tokens" because that is what they count.

  • The A/B's --budget had the same undercount. The suite cap summed
    inputTokens, so 120 runs reported about 1.3M of spend while their streams
    carried 7.1M. parseStream now also returns contextTokens, the cap and the
    report use it, and --rescore re-reads the old streams to put the corrected
    number on them. inputTokens is still recorded so a table can be compared
    with the ones written before this.

  • The test suite wrote to the real window ledger. guard-hook.test.js
    exercised the guard with the default OC_HARNESS_BUDGET_FILE, so every test
    run added its 1,500 and 600 tokens to
    ~/.local/state/opencode-harness-kit/budget.json - the live ledger showed
    twelve test entries and nothing else. The tests now point the ledger at
    their own temporary directory.

  • --rescore graded a timeout as a wrong answer. A run that hit the
    ceiling has a stream that stops mid-sentence, which reads exactly like an
    answer that cited nothing; only the sidecar knows which it was. Four runs
    that timed out at 12 steps were re-scored as four wrong answers, and the
    table then showed 18/23 where the honest count is 18/19.

  • A resume crashed the report on a sidecar from an older version. The
    re-score reads escaped, tools and text from every result; a sidecar
    written before those fields existed threw a TypeError after every run had
    already been paid for. Missing fields are read as empty, and the corrected
    token count is written back so a second resume does not depend on the stream
    still being there.

  • The A/B's preflight said "ok" to a run that had failed. It checked only
    the timeout and the quota, so a run that reported an error event and exited
    in three seconds passed. The first suite with that preflight spent eighteen
    runs learning that every kit-off arm had failed with
    ProviderModelNotFoundError, scored as "no output, not scored". It now
    reads the stream, and any error event stops the suite with exit 2 and the
    provider's own message. An exit 0 with zero steps and no text is the same
    shape - a spent route - and is refused the same way.

  • The kit-off arm could not resolve the model it was told to use. opencode
    keeps its model catalogue in ~/.cache/opencode/models.json, and the arm's
    temporary HOME had no cache, so a bare model: string was a model the
    installation had never heard of. The cache is opencode's, not the kit's, so
    it is symlinked into the temporary home the same way ~/.local/share/opencode
    already was.

  • An error in a measured run was counted as a wrong answer. A run that
    produced no tool call and no text, but did report an error, now stops the
    suite at once with that error instead of waiting for --fail-fast and then
    scoring the refusal.

  • The lane counter's "no telemetry" test failed on a machine that had one
    two directories up.
    The lookup walks up to the project root by design, so
    /tmp/.space/telemetry.jsonl - written by any opencode run started in
    /tmp - made it find a file and fail for a reason that was not the code's.
    The test now runs twelve levels down, past the walk-up's ceiling, so nothing
    above its own tree is looked at; the walk-up itself is unchanged.

  • The A/B was governed by the kit's own budget. The kit-on arm runs on the
    real config, so the guard's session and window thresholds applied to it: 120
    runs spend more than the window's stop, and from the run that crossed it the
    guard refused tool calls - every later run would be scored as a wrong answer
    for a reason the suite is not measuring. The suite now sets all four
    thresholds to 0 for its runs and points OC_HARNESS_BUDGET_FILE at the run
    directory, so a benchmark neither trips the guard nor charges the account's
    real window. The suite's own cap is --budget.

  • An empty plan reported success. A plan with no tasks printed
    "every task is done (0/0)" and exited 0. Nothing was done, so it now stops
    with exit 2 and says the plan has no tasks.

  • A blocked task did not stop the tasks after it. The loop marked the
    blocker and carried on, then reported it at the end. It now runs in file
    order and stops at the first blocked task, so pending work before the
    blocker still lands and work after it no longer builds on a failure.

  • The task file was written back from the copy read at the start. A
    human edit or an agent deletion made mid-run was overwritten, and a deleted
    task file was recreated. Write-back re-reads the file, refuses to write a
    stale copy, and goes through rename so a reader never sees half a plan.

  • A task was matched by title and position only. With two same-titled
    tasks, removing or renaming the first made the loop mark the second one
    done. Matching now uses title + change + files, and an ambiguous match
    stops the run instead of guessing.

  • verify: true and verify: echo ok were treated as checks. Both exit
    0 whatever the agent did. A verify line made only of no-ops now blocks the
    task before the agent runs; a no-op before a real command
    (true && npm test) is still accepted.

  • doc and verify opened their bash maps with "*": "allow". The
    read-only lanes could write through sed -i, awk or python3 -c
    regardless of their edit: "deny". doc now defaults to deny and verify
    defaults to ask, each with the commands its job actually needs.

  • A run that hit the lane's step cap reported only "verify failed". At the
    cap OpenCode exits 0 with a text-only answer, so the loop saw a failed
    check and blamed the change, while the log held the real reason. run now
    counts step_finish events in the transcript and, when the count reaches
    the role's configured cap, says so in the stop message and in the task's
    recorded output, with the setting to raise.

  • An agent that answered nothing was reported as a failed change. A spent
    model route shows up in one of two shapes. Some providers exit 0 with an
    empty stdout and the reason on stderr: the loop then ran the verify against
    an untouched tree and blocked the task for work the model never attempted.
    Others hang until the ceiling kills them, which at the default budget is two
    hours spent on one task with nothing to show for it.
    run now stops with exit 3 when stdout is empty without running the verify,
    and a timeout names the route and the limit instead of saying the agent
    "failed". The task is marked blocked either way, with the stderr and the
    advice in its output.

  • The A/B ground through a quota error as if it were an answer. The free
    tier can print {"type":"error"} with "Free usage limit reached" on stdout
    and exit 0; the suite read that as an empty stream and needed --fail-fast
    runs before stopping. It now stops on the first one with exit 2 and the
    provider's own words, because the refusal measures the provider and not the
    kit.

  • The kit-off arm made paid title calls. makeNokitHome pinned model
    but not small_model, so the title agent used opencode's default paid
    model. A real run made 1,073 title calls to gpt-5.4-nano; every one was
    refused for insufficient funds. Both are pinned now.

  • A killed A/B left the agent running. spawnSync blocks the event loop,
    so Ctrl-C killed the parent and the child was reparented to systemd - one
    survived 2h54m against the real auth store. The capture uses spawn, a
    process group, a timeout timer that kills the group, and SIGINT/SIGTERM
    handlers that do the same.

  • run started without opencode, and outside a git work tree. The
    first was reported as a failed agent that blocked the first task; the
    second silently switched off the files: scope check. Both are now
    refused before the lock is taken, the second unless --allow-extra-files
    is passed.

  • The scope check counted the loop's own bookkeeping as an out-of-scope
    edit.
    The plugin writes .space/telemetry.jsonl on every agent call, so
    a task that touched only the files it named was still blocked for changing
    .space/. Loop state is excluded now; a real edit outside files: is
    still caught.

  • The A/B's wall-time row read the stream, not the ceiling. A run killed
    at 180s has a stream that stops at 58s, so the report printed it as a 58s
    run and the wall column favored whichever arm was killed sooner. Each
    sidecar now records the harness's own timer, --rescore prefers it, and a
    resumed sidecar without one falls back to the stream's last event.

Added

  • A per-session token budget, counted by the guard. The free tier is one
    shared window and opencode shows no number for it, so a long session spends
    it silently - one made 599 requests and 12,025,980 input tokens, and the
    first sign was the provider refusing the next request. The guard sums the
    input tokens each step-finish part reports, warns once at
    OC_HARNESS_BUDGET_WARN (default 2M) and refuses tool calls at
    OC_HARNESS_BUDGET_STOP (default 5M) with the number and the fix. 0
    disables either. The count is per session, so a subagent is not charged to
    the conversation that started it.
  • oc-harness use --no-skills. opencode writes every discovered skill's
    name and description into the system message, so a large skill collection
    is a fixed cost on every request. The flag turns the skill tool off for the
    lanes, which carry their instructions in prompts/, and the config header
    says so. Off by default.
  • A shared-window ledger, so a new session does not reset the budget. The
    per-session count answered the wrong question: the free tier is one window
    per account, and two sessions were measured spending one window 47 and 33
    minutes apart. The guard now writes each session's spend to
    OC_HARNESS_BUDGET_FILE (default $XDG_STATE_HOME/opencode-harness-kit/budget.json)
    and sums the entries inside OC_HARNESS_BUDGET_WINDOW_MINUTES (default 60),
    warning at OC_HARNESS_BUDGET_WINDOW_WARN (4M) and refusing tool calls at
    OC_HARNESS_BUDGET_WINDOW_STOP (8M) in every session, including one opened
    after the window ran out. The file is written atomically; an unreadable file
    is moved to budget.json.corrupt and the window starts empty (if the move
    cannot be written, the call fails closed), and entries older than six hours
    are pruned.
  • eval/ab.mjs --out <dir>: one stream and one result per run, and a
    rerun with the same --out skips the runs that already have a result. A
    suite stopped by the quota or by a reboot resumes instead of starting over.
    The directory also gets run-manifest.json with the suite hash, target
    fingerprint and model; a resume whose manifest differs stops with exit 2
    instead of mixing two experiments, and a directory from before the manifest
    existed is adopted with a warning.
  • eval/ab.mjs --budget <tokens>: cap the suite's input tokens. Checked
    before each run, so the run that crosses the cap is still measured; the next
    one stops the suite with exit 2.
  • A preflight run in eval/ab.mjs (on by default, --no-preflight to
    skip): one question to the model before the suite, exit 2 if it does not
    answer, times out, or its stream carries an error event. Answering is the
    test: exit 0 with zero steps and no text stops the suite too.
  • oc-harness run --review: after verify passes, ask the reviewer lane for
    a schema'd verdict, oc-harness/review@1. A fail, prose, a malformed
    object or a crashed reviewer stops the run with exit 5 and marks the task
    blocked. The verdict is kept at .space/review/<review>-<slug>.json, keyed
    by review rather than task round so a fix keeps both the failing and the
    passing verdict; the contract is in docs/REVIEW-VERDICT.md.
  • .space/run.lock: one writer per task file. A lock whose pid is gone is
    cleared; a live one is not. --force-unlock breaks one explicitly.
  • run --allow-extra-files: documented escape hatch for the scope check.
  • eval/compare.mjs: grade any agent CLI against the same suite with the same
    citation resolver and mutation comparison. eval/ab.mjs can only compare
    the kit with opencode, because it reads opencode's event stream; this runs a
    plain command in a fresh copy of the subject, so an Aider or SWE-agent
    number is comparable with the kit's own. A command that is not installed is
    reported as unusable, not as a 0% pass rate.
  • eval/ab.mjs --fail-fast <n> (default 2): stop when n runs in a row produce
    no tool call and no text. A spent free route hangs and times out rather than
    failing, which looked like a benchmark that "stopped after a few minutes"
    when it was really waiting out one dead run at a time. A timeout whose last
    60 seconds (or a twentieth of the ceiling, whichever is smaller) are quiet
    counts toward the streak too, because a spent route hangs before the
    ceiling. The raw streams are kept for --rescore.

Changed

  • The build lane's step cap is 120, not 40. A session that reaches the
    cap is forced into a text-only response, which in an interactive session
    looks like the agent stopping mid-task after a few minutes and in
    oc-harness run looks like a task that never finished. 40 steps is a
    reasonable ceiling for one planned task and not for a session that is
    building and testing a change. The other lanes keep their caps.
  • The README and docs/EXIT-CODES.md describe run as build + verify, not
    plan → build → review → finish. It never planned, committed or pushed.
  • Exit code 5 is now used, and documented.