Repository navigation
0.7.0
[0.7.0] — 2026-10-25
oc-harness run now stops for the reasons it already claimed to stop for,
the read-only lanes no longer have a write-capable shell, and the token budget
counts the context a request reads instead of only its uncached part.
Fixed
-
A denied tool call was fed back as another turn, and the retry spent the
window.experimental.continue_loop_on_denywastrue, so opencode
handed the guard's refusal back to the model and continued the loop. The
measured session made 138 requests and 7,559,016 context tokens in 29
minutes on sixshared token windowdenials, each retry re-sending the
whole prompt. The generated config now sets it tofalse, and the guard
blocks a single request that re-sends more thanOC_HARNESS_TURN_STOP
(default 300k) with "do not retry". -
The in-event shared-window stop did not refuse anything. When the
step-finishsum crossedOC_HARNESS_BUDGET_WINDOW_STOP, the guard set
windowSpentbut never setwindowWhy, sosharedWindowSpent()returned
an empty reason and every tool call was allowed. The reason is set where
the sum crosses, and the stop is written to the ledger with
recordStop()so a new session cannot outlive the provider's window. -
A corrupt ledger locked the machine for six hours, or vanished. It is
now moved tobudget.json.corruptand the window starts empty; the broken
bytes are kept as evidence.recordSpend()andrecordStop()repair
before writing instead of reading the corruption as an empty window. -
The bundled ECC subagents were loaded on every request. Their
descriptions came fromprompts/agents/*.txt, were not lanes, and so were
not covered by the stale-lane rule. They are removed from the generated
agent map when that is the prompt, and named in the config header; a
hand-written agent that reuses that prompt path is removed like the
bundle's own, while any other prompt is left alone. -
The token budget undercounted a cached session five to forty times, so it
never fired. Astep-finishpart reportsinput,cache.read,
cache.write,outputandreasoning. The guard summedinputalone -
the part of the prompt that was not cached - while the cached prefix, which
is what a long conversation actually re-reads, was ignored. One measured
stream reported 844 uncached tokens against 36,480 cached ones on the same
step, then 391 against 9,088. The 2M warn and 5M stop were therefore five
to forty times the real spend, and the provider's window ran out before
either fired.requestTokens()insrc/budget.jsnow countsinput + cache.read + cache.writefor both the session total and the shared-window ledger; the
toasts and refusals say "context tokens" because that is what they count. -
The A/B's
--budgethad the same undercount. The suite cap summed
inputTokens, so 120 runs reported about 1.3M of spend while their streams
carried 7.1M.parseStreamnow also returnscontextTokens, the cap and the
report use it, and--rescorere-reads the old streams to put the corrected
number on them.inputTokensis still recorded so a table can be compared
with the ones written before this. -
The test suite wrote to the real window ledger.
guard-hook.test.js
exercised the guard with the defaultOC_HARNESS_BUDGET_FILE, so every test
run added its 1,500 and 600 tokens to
~/.local/state/opencode-harness-kit/budget.json- the live ledger showed
twelve test entries and nothing else. The tests now point the ledger at
their own temporary directory. -
--rescoregraded a timeout as a wrong answer. A run that hit the
ceiling has a stream that stops mid-sentence, which reads exactly like an
answer that cited nothing; only the sidecar knows which it was. Four runs
that timed out at 12 steps were re-scored as four wrong answers, and the
table then showed 18/23 where the honest count is 18/19. -
A resume crashed the report on a sidecar from an older version. The
re-score readsescaped,toolsandtextfrom every result; a sidecar
written before those fields existed threw aTypeErrorafter every run had
already been paid for. Missing fields are read as empty, and the corrected
token count is written back so a second resume does not depend on the stream
still being there. -
The A/B's preflight said "ok" to a run that had failed. It checked only
the timeout and the quota, so a run that reported an error event and exited
in three seconds passed. The first suite with that preflight spent eighteen
runs learning that every kit-off arm had failed with
ProviderModelNotFoundError, scored as "no output, not scored". It now
reads the stream, and any error event stops the suite with exit 2 and the
provider's own message. An exit 0 with zero steps and no text is the same
shape - a spent route - and is refused the same way. -
The kit-off arm could not resolve the model it was told to use. opencode
keeps its model catalogue in~/.cache/opencode/models.json, and the arm's
temporaryHOMEhad no cache, so a baremodel:string was a model the
installation had never heard of. The cache is opencode's, not the kit's, so
it is symlinked into the temporary home the same way~/.local/share/opencode
already was. -
An error in a measured run was counted as a wrong answer. A run that
produced no tool call and no text, but did report an error, now stops the
suite at once with that error instead of waiting for--fail-fastand then
scoring the refusal. -
The lane counter's "no telemetry" test failed on a machine that had one
two directories up. The lookup walks up to the project root by design, so
/tmp/.space/telemetry.jsonl- written by anyopencode runstarted in
/tmp- made it find a file and fail for a reason that was not the code's.
The test now runs twelve levels down, past the walk-up's ceiling, so nothing
above its own tree is looked at; the walk-up itself is unchanged. -
The A/B was governed by the kit's own budget. The kit-on arm runs on the
real config, so the guard's session and window thresholds applied to it: 120
runs spend more than the window's stop, and from the run that crossed it the
guard refused tool calls - every later run would be scored as a wrong answer
for a reason the suite is not measuring. The suite now sets all four
thresholds to0for its runs and pointsOC_HARNESS_BUDGET_FILEat the run
directory, so a benchmark neither trips the guard nor charges the account's
real window. The suite's own cap is--budget. -
An empty plan reported success. A plan with no tasks printed
"every task is done (0/0)" and exited 0. Nothing was done, so it now stops
with exit 2 and says the plan has no tasks. -
A blocked task did not stop the tasks after it. The loop marked the
blocker and carried on, then reported it at the end. It now runs in file
order and stops at the first blocked task, so pending work before the
blocker still lands and work after it no longer builds on a failure. -
The task file was written back from the copy read at the start. A
human edit or an agent deletion made mid-run was overwritten, and a deleted
task file was recreated. Write-back re-reads the file, refuses to write a
stale copy, and goes throughrenameso a reader never sees half a plan. -
A task was matched by title and position only. With two same-titled
tasks, removing or renaming the first made the loop mark the second one
done. Matching now uses title +change+files, and an ambiguous match
stops the run instead of guessing. -
verify: trueandverify: echo okwere treated as checks. Both exit
0 whatever the agent did. A verify line made only of no-ops now blocks the
task before the agent runs; a no-op before a real command
(true && npm test) is still accepted. -
docandverifyopened their bash maps with"*": "allow". The
read-only lanes could write throughsed -i,awkorpython3 -c
regardless of theiredit: "deny".docnow defaults to deny andverify
defaults to ask, each with the commands its job actually needs. -
A run that hit the lane's step cap reported only "verify failed". At the
cap OpenCode exits 0 with a text-only answer, so the loop saw a failed
check and blamed the change, while the log held the real reason.runnow
countsstep_finishevents in the transcript and, when the count reaches
the role's configured cap, says so in the stop message and in the task's
recorded output, with the setting to raise. -
An agent that answered nothing was reported as a failed change. A spent
model route shows up in one of two shapes. Some providers exit 0 with an
empty stdout and the reason on stderr: the loop then ran the verify against
an untouched tree and blocked the task for work the model never attempted.
Others hang until the ceiling kills them, which at the default budget is two
hours spent on one task with nothing to show for it.
runnow stops with exit 3 when stdout is empty without running the verify,
and a timeout names the route and the limit instead of saying the agent
"failed". The task is marked blocked either way, with the stderr and the
advice in its output. -
The A/B ground through a quota error as if it were an answer. The free
tier can print{"type":"error"}with "Free usage limit reached" on stdout
and exit 0; the suite read that as an empty stream and needed--fail-fast
runs before stopping. It now stops on the first one with exit 2 and the
provider's own words, because the refusal measures the provider and not the
kit. -
The kit-off arm made paid title calls.
makeNokitHomepinnedmodel
but notsmall_model, so the title agent used opencode's default paid
model. A real run made 1,073 title calls togpt-5.4-nano; every one was
refused for insufficient funds. Both are pinned now. -
A killed A/B left the agent running.
spawnSyncblocks the event loop,
so Ctrl-C killed the parent and the child was reparented to systemd - one
survived 2h54m against the real auth store. The capture usesspawn, a
process group, a timeout timer that kills the group, andSIGINT/SIGTERM
handlers that do the same. -
runstarted withoutopencode, and outside a git work tree. The
first was reported as a failed agent that blocked the first task; the
second silently switched off thefiles:scope check. Both are now
refused before the lock is taken, the second unless--allow-extra-files
is passed. -
The scope check counted the loop's own bookkeeping as an out-of-scope
edit. The plugin writes.space/telemetry.jsonlon every agent call, so
a task that touched only the files it named was still blocked for changing
.space/. Loop state is excluded now; a real edit outsidefiles:is
still caught. -
The A/B's wall-time row read the stream, not the ceiling. A run killed
at 180s has a stream that stops at 58s, so the report printed it as a 58s
run and the wall column favored whichever arm was killed sooner. Each
sidecar now records the harness's own timer,--rescoreprefers it, and a
resumed sidecar without one falls back to the stream's last event.
Added
- A per-session token budget, counted by the guard. The free tier is one
shared window and opencode shows no number for it, so a long session spends
it silently - one made 599 requests and 12,025,980 input tokens, and the
first sign was the provider refusing the next request. The guard sums the
input tokens eachstep-finishpart reports, warns once at
OC_HARNESS_BUDGET_WARN(default 2M) and refuses tool calls at
OC_HARNESS_BUDGET_STOP(default 5M) with the number and the fix.0
disables either. The count is per session, so a subagent is not charged to
the conversation that started it. oc-harness use --no-skills. opencode writes every discovered skill's
name and description into the system message, so a large skill collection
is a fixed cost on every request. The flag turns the skill tool off for the
lanes, which carry their instructions inprompts/, and the config header
says so. Off by default.- A shared-window ledger, so a new session does not reset the budget. The
per-session count answered the wrong question: the free tier is one window
per account, and two sessions were measured spending one window 47 and 33
minutes apart. The guard now writes each session's spend to
OC_HARNESS_BUDGET_FILE(default$XDG_STATE_HOME/opencode-harness-kit/budget.json)
and sums the entries insideOC_HARNESS_BUDGET_WINDOW_MINUTES(default 60),
warning atOC_HARNESS_BUDGET_WINDOW_WARN(4M) and refusing tool calls at
OC_HARNESS_BUDGET_WINDOW_STOP(8M) in every session, including one opened
after the window ran out. The file is written atomically; an unreadable file
is moved tobudget.json.corruptand the window starts empty (if the move
cannot be written, the call fails closed), and entries older than six hours
are pruned. eval/ab.mjs --out <dir>: one stream and one result per run, and a
rerun with the same--outskips the runs that already have a result. A
suite stopped by the quota or by a reboot resumes instead of starting over.
The directory also getsrun-manifest.jsonwith the suite hash, target
fingerprint and model; a resume whose manifest differs stops with exit 2
instead of mixing two experiments, and a directory from before the manifest
existed is adopted with a warning.eval/ab.mjs --budget <tokens>: cap the suite's input tokens. Checked
before each run, so the run that crosses the cap is still measured; the next
one stops the suite with exit 2.- A preflight run in
eval/ab.mjs(on by default,--no-preflightto
skip): one question to the model before the suite, exit 2 if it does not
answer, times out, or its stream carries an error event. Answering is the
test: exit 0 with zero steps and no text stops the suite too. oc-harness run --review: after verify passes, ask the reviewer lane for
a schema'd verdict,oc-harness/review@1. A fail, prose, a malformed
object or a crashed reviewer stops the run with exit 5 and marks the task
blocked. The verdict is kept at.space/review/<review>-<slug>.json, keyed
by review rather than task round so a fix keeps both the failing and the
passing verdict; the contract is indocs/REVIEW-VERDICT.md..space/run.lock: one writer per task file. A lock whose pid is gone is
cleared; a live one is not.--force-unlockbreaks one explicitly.run --allow-extra-files: documented escape hatch for the scope check.eval/compare.mjs: grade any agent CLI against the same suite with the same
citation resolver and mutation comparison.eval/ab.mjscan only compare
the kit with opencode, because it reads opencode's event stream; this runs a
plain command in a fresh copy of the subject, so an Aider or SWE-agent
number is comparable with the kit's own. A command that is not installed is
reported as unusable, not as a 0% pass rate.eval/ab.mjs --fail-fast <n>(default 2): stop whennruns in a row produce
no tool call and no text. A spent free route hangs and times out rather than
failing, which looked like a benchmark that "stopped after a few minutes"
when it was really waiting out one dead run at a time. A timeout whose last
60 seconds (or a twentieth of the ceiling, whichever is smaller) are quiet
counts toward the streak too, because a spent route hangs before the
ceiling. The raw streams are kept for--rescore.
Changed
- The
buildlane's step cap is 120, not 40. A session that reaches the
cap is forced into a text-only response, which in an interactive session
looks like the agent stopping mid-task after a few minutes and in
oc-harness runlooks like a task that never finished. 40 steps is a
reasonable ceiling for one planned task and not for a session that is
building and testing a change. The other lanes keep their caps. - The README and
docs/EXIT-CODES.mddescriberunas build + verify, not
plan → build → review → finish. It never planned, committed or pushed. - Exit code 5 is now used, and documented.