Repository navigation
v1.7.0 — degradation resilience
Degradation-resilience pass over the three-channel fallback path, driven by three instrumented
probes (kept in the gitignored tasks/probes/, re-runnable, no real requests). The probes showed
that channel switching was immediate but failure detection was effectively unbounded, and that
one of the two channel branches did not degrade at all.
Upgrading from 1.6.x — read this. Two defaults change, and both can alter what a run does.
timeout_secondsnow means "per fallback link", not "per HTTP attempt". A seat that used
to succeed only by burning two or three retries past its timeout can now be cut at the budget
boundary and handed to the next fallback instead. Action: if a seat legitimately needs
retry headroom, raise that seat'stimeout_seconds— a seat's worst case is now
expanded links × timeout_seconds, so the value is readable as a bound. Count expanded
tries: acliseat with no explicitcli_kindbecomes two links (auggie, then codex) when
both binaries are present. Everycliseat in the shipped config setscli_kind, so there the
two counts agree. Nothing to change if
your seats normally answer on the first call. The first time a link is actually cut, the run
prints a one-time[budget]note on stderr explaining this and naming the seat, so you do not
have to have read this entry.- A
cliseat withmodel:but nocli_kind:and noauggie_model:is now rejected at
startup instead of silently running auggie's default model. Action: the error names both
fixes — addauggie_model: <auggie-side id>, or setcli_kind: codex/auggieexplicitly.
Everycliseat in the shippedconfig.example.yamlalready setscli_kind, so a stock
config is unaffected.Revert path: no runtime opt-out flag ships for either change — both are corrections to what
the existing knob and the existing config already claimed to mean, and a flag would preserve the
silent-failure mode they remove. To go back, pin the previous release:
/plugin install moa@moa-skillafter/plugin marketplace add sdsrss/moa-skill#v1.6.2, or for a
direct-copy installgit checkout v1.6.2 -- skills/moa.
Changed
timeout_secondsnow bounds a whole fallback link, not a single HTTP attempt. It was applied
per attempt and handed unchanged to every link, socall_model's two internal retries multiplied
it: a probe with threeapilinks attimeout_seconds: 240issued 9 HTTP attempts and could
run 2169 s (~36 min) for one seat, and the quorum grace window did not help because it only
opens after quorum is reached — when two of three seats fail, it never opens at all, so the
bound was missing exactly when degradation mattered. Each link now gets its own wall-clock budget
covering its retries, backoff and JSON repair round; a link that exhausts it fails with
err_class: budgetand yields to the next one. A seat's worst case is thereforeexpanded links × timeout_seconds— expanded, because acliseat with no explicitcli_kindbecomes two tries
(auggie, then codex) when both binaries are present; the shipped config setscli_kindon every
cliseat, so there the config count and the expanded count agree. The budget is allocated
per link rather than shared across the seat, so
every fallback the user configured is still tried at least once. What is tightened is wall clock,
not retry policy: fast failures (a 429 returned immediately) consume almost no budget and still
get the full retry count, and the common "one call, succeeds" path is unchanged. No new config
option — this makes the existing knob mean what it reads like. (ISSUE-007)- A
cliseat that would run an unknown model is now rejected at config validation instead of
warned about.channel: cliwithout an explicitcli_kindresolves to auggie when the binary
is present, and that path only honoursauggie_model— so a member carryingmodel:but no
auggie_model:silently ran auggie's default model and recordedmodel_used: null. The cost is
not merely "config ignored":references/synthesis.mdrequires the arbiter to disclose each
seat's model family so readers can discount an "everyone agrees" result, and a seat of unknown
family makes that hard rule unexecutable — cross-family de-correlation is the committee's whole
premise, so silently running an unknown model is a correctness failure, not an acceptable
degradation. The error names both fixes (addauggie_model, or setcli_kindexplicitly). Every
cliseat in the shippedconfig.example.yamlsetscli_kind, so this gate cannot fire on a
stock config; a test now asserts that. (ISSUE-008)
Fixed
-
generate/refineno longer hang after they are done. Stragglers abandoned by the grace
window keep running in background threads, andconcurrent.futures' atexit handler joins them
at interpreter shutdown — a probe measured the dispatcher returning at 0.55 s while the process
only exited at 6.1 s, which at the defaulttimeout_seconds: 240means up to 4 minutes of
silence afterdoneis printed. That lands precisely whereSKILL.mdstep 3 tells the arbiter to
backgroundgenerateand dispatch CH1 seats in parallel, and it stallsgenerate && stats
chains.main()now exits the process directly once a straggler has been abandoned. Artifacts are
safe: everywrite_memberhappens on the main thread'son_donecallback, so abandoned threads
never write files. Before exiting it also deletes the CLI channels' temp directories, which the
abandoned thread is parked inside and whichos._exitwould otherwise leave behind — the auggie
channel writes the whole briefing toprompt.txtin there, so skipping the cleanup would strand
review material in the system temp dir. Only the clean path does this — a non-zerosys.exit
still takes the regular (slower) shutdown, where the user is reading an error anyway. (ISSUE-009) -
apiseats now degrade on unparseable output, likecliseats already did. The two channel
branches of_dispatch_channelshandled the same failure asymmetrically: when a member's output
could not be parsed as JSON even after the repair round, theclibranch raised and the seat fell
through to the next link in itsfallbackchain, while theapibranch returned aparsed=None
result immediately — consuming the seat and voiding every remaining fallback link, so anapi
seat with a configured degradation chain behaved as if it had none. The failure also recorded
err_class: null, making this class invisible to the error tally instats.json, and dropped the
usageof the two calls (generate + repair) that had already been billed, so cost reporting
under-counted exactly when things were going wrong. Theapibranch now records aparse-class
failure carryingusageandraw, then continues down the chain._failtakes optional
usage/rawarguments; all other call sites keep their previous shape. (ISSUE-006)Member artifacts for failed seats now always carry a
usagekey (nullwhen nothing was billed).
This holds even when the repair round itself raises — the generate round's already-billedusage
and itsrawride out on the exception rather than dying with the stack frame, which matters
because the per-link budget below makes "generate ate the budget, repair cannot run" a routine
path rather than a rare one. -
A failed seat's artifact now names the link that actually ran.
_failbuiltmodel_usedand
protocolfrom the member's primary channel while therawandusageit carried came from
whichever fallback link produced them, so one link's bytes shipped under another link's identity —
androster[].model_knownwould report the config'smodelas known for a seat that never ran
it, defeating the family-composition checkreferences/synthesis.mdasks the arbiter to perform.
Failure records inside the fallback loop now carry the real link'smodel_used/protocol/
channel_used; the last of those also restores a field v1.6.2 populated and the first cut of this
release had regressed tonull.
Added
members_skippedseparates abandonment from failure. A seat dropped when the grace window
expired is a deliberate "we are not waiting for it", not a fault, but it counts in
members_failedall the same.members_failedkeeps its old meaning (every non-successful seat)
so existing readings do not shift;members_skippedis a subset of it, and real failures are
members_failed - members_skippedwithout readingerr_classseat by seat. (ISSUE-011)roster[].model_knownflags seats running a channel default model —cli_kind: codexwith
model: null, the shipped fallback idiom, is one — whose family cannot be determined.
synthesis.mdnow tells the arbiter to exclude those seats from family counts and name them in
the report rather than guessing from the config. (ISSUE-008 companion)dry-runstates that its call count is a lower bound, naming what it excludes: JSON repair
rounds, transient retries (with theirmax_tokensdoubling), discussion rounds, and subscription
seats that turn billable on degradation.
Deliberately not shipped
- Aggregate reporting of what failed seats burned. This release grew
token_usage.wasted_tokens
/wasted_membersand then withdrew them before release, because pre-ship review showed the pair
wrong in both directions at once. Under-count: the largest single token sink in the file is
call_model's truncation retry, which doublesmax_tokenseach attempt — measured at
3000 → 6000 → 12000, i.e. 21000 tokens genuinely billed across three HTTP 200s — and it discards
each attempt'susageinside its own retry loop, so the figure never reaches the aggregator and
reports0. Over-count: when a provider omits the usage block,_merge_usage({})yields an
all-zero but truthy dict, so a seat that was never billed is counted as a wasted member.
A cost field that exists to make hidden spend visible, and is wrong in both directions, is worse
than no field — this CHANGELOG would have been telling you to read it. The root cause is
structural (usage rides local variables along the normal return path, so every exception path is
a drop site) and patching the sites one at a time does not converge; the fix is a per-seat
accumulator that records spend independently of how the stack unwinds, and it gets its own
release. Per-seatusageinmember_*.jsonis still preserved on a best-effort basis and makes
no completeness claim.