v0.26.0

🚀 aiHelpDesk v0.26.0: Every Hop Earns Its Own Trust (No More Free Passes Mid-Chain)

Two major production readiness items have landed in this release and in particular the long-overdue internal cleanup of the faulttest CLI and its underlying library that had been duplicated for a long time and a single field addition meant editing both by hand.

Feature-wise, the previous v0.25.0 release made a single playbook's trust self-correcting. A cert, once earned, could go stale and get automatically revoked. It also flagged a known gap on the way out the door: only a fault's declared entry point could ever earn that trust in the first place. A playbook that only ever reached the mid-chain (e.g. a SysAdmin Agent handed a Docker-inspect job partway through a diagnosis or a remediation playbook one hop downstream) was structurally uncertifiable no matter how many times it ran correctly. v0.26.0 closes that gap.

The theme for v0.25.0 was "the gate that closes itself." With v0.26.0 we are following with the natural sequel: "the gate that knows exactly who it's gating.", because a chain no longer inherits trust from its entry point alone. Instead, every hop earns (and can lose) its own.

Here are a few release highlights:

✅ Refactor: cmd/faulttest and faultlib, deduplicated.

We consider this to be an important production readiness milestone. The story here is that we knew from the get-go that one of the aiHelpDesk's "four whales", the fault injection testing, is not going to be used just internally, but we'll expose it as a critical customer facing system too. That indeed happened when aiHelpDesk went Beta in April.

That wasn't the question. The question was how tightly coupled should be the internal and the external fault injection systems. The internal is reacher, more exhaustive and because we control the database and the OS, it's easier to exercise many of the failure scenarios. For the customer facing fault injection testing, there's much less control, especially with the BYOD (let alone the OS) and so the number of failure modes is limited. On the flip side, the external one needs a CLI, so it has its own set of features not needed for the internal CI.

And so the question was if the two should be separate enough to allow for the flexibility of growth independently or not. We initially settled on the module architecture and shared libs, but over the last ten releases it became obvious that this is not sufficient and the two started to diverge. The feature (flexibility) gradually turned into a tech debt. With v0.26.0 we are consolidating the two! Types, config and scoring logic unified between the two packages, net -274 lines, including a real live {{server_id}} template bug fixed along the way.

This is a big milestone for us and a prerequisite work for per-fault injection timeouts below.

✅ Refactor: per-fault injection timeouts.

This is another prod readiness item (which got unblocked by the dedup described above). Fault injection previously shared one unbounded outer context with the rest of a run. This sometimes led to a hung injection that could silently stall the whole batch with no independent bound. New inject_timeout catalog field (90s default, overridable — 30m for k8s-node-memory-pressure) gives injection its own bounded context via faultlib.RunFaultCycle/TeardownFault. As part of this refactoring, we also fixed a tricky teardown-skip-on-injection-failure bug found while implementing this.

✅ Every hop in an escalation chain earns its own stability cert.

This was an important feature item where faulttest run --repeat N --approval-mode=force now posts a FaultStabilityCert for every distinct playbook series a chain actually passes through, not just the fault's declared entry point. A synthetic fault ID (using this format: <entry-point-id>::hop:<series-id>) reuses the existing cert-posting pipeline completely unchanged with zero backend or schema changes. New vault hop-certs <series-id> CLI command makes previously-invisible chain-only playbooks discoverable.

This feature has been live-verified against a real 3-hop DB→SysAdmin→K8s chain on a deployed Helm/K8s cluster, so that both previously uncertifiable playbooks in that chain earned their first-ever cert!

doc link

✅ Corroborated Decline

This is a new feature that augments the more obvious write/destructive-absence check and it's designed for edge cases where no action is legitimately taken by a (most often a mid-hop, intermediate) playbook. This is an important use-case for the complex multi-hop, cross domain triage where a single agent can't reach a definitive conclusion.

doc link

🐛 Bug fixes

All found via live testing against a real deployed chain, not unit tests alone.

Shipping the hop-cert feature meant actually running it repeatedly against a real Helm/K8s deployment, not just testing it in isolation. That process is what surfaced the following nine bugs live, plus one caught in design review before it ever shipped:

• A naive design would have made every hop cert permanently un-earnable. attribution_consistent defaults to false in the DB schema when omitted from a POST payload. Posting hop certs without it set would have looked like the feature worked (certs exist, visible via the new CLI command) while the trust gate stayed blocked forever. This would be a much worse failure mode than the original gap, since it's far harder to notice. Caught via a design-review stress pass before implementation, not live.
• Multi-hop tool_calls only ever reflected the primary hop, causing Tools: 0%. The final response was built by merging extra on top of the first hop's own raw capture. For chains where the fault's real expected tools are only called by a later hop, this silently understated tool evidence to zero. Fixed by aggregating and deduping tool_calls across every hop in the chain. (578128e)
• Write-mismatch DIRTY with no way to recognize a corroborated, genuine decline. An agent that investigated, found nothing to write and correctly escalated instead was flagged identically to a silent tool-call failure. New declinedActionSignal requires two independent self-reported protocol lines to agree (ACTION_TAKEN: none + a well-formed handoff) before downgrading. This is a deliberately not a blanket downgrade, since a model could in principle fake either signal alone. (578128e)
• 7 read-only tools across the SysAdmin and Database Agents never called RecordToolCall. Only the write tools did. Genuinely-executed diagnostic tool calls were invisible to delegation verification, producing false mismatches. (a5d5a99, 602f2e6)
• A shared SysAdmin prompt hardcoded "use check_host and restart_container" for every playbook, even the two of three that explicitly forbid calling restart_container in their own guidance. That led to misclassifying pure-diagnosis hops as write-class delegations and telling the model it could use a tool its own playbook prohibited. Made the boilerplate playbook-specific. (c182ef8)
• Cross-hop verification-flag leaking. Force-mode auto-chaining can put multiple hops under one shared trace_id. A whole-trace lookup couldn't tell them apart, so a later hop's genuine mismatch leaked backward onto an earlier, actually-clean hop's reported status. Replaced with a time-windowed read of each hop's own delegation_verification events, bounded by the next hop's start time (not its own CompletedAt, an unsafe/racy boundary found via a real test failure on the first attempt). (995107c, ff44cc8)
• No corroboration path for a terminal, policy-denied write hop. declinedActionSignal needs a handoff line to a next hop. A terminal hop with nothing downstream to hand off to can never produce one, even when the audit trail already independently proves the write was attempted and genuinely blocked. New hasActionClassDenial checks for a real policy_decision deny event instead because a single, code-derived signal is sufficient here, since (unlike self-reported text) the model can't fabricate it. A follow-up coverage pass caught a real stale-MismatchReason bug this fix exposed (not cleared when an unrelated narration mismatch re-fired Mismatch=true) and a real gap in policy_decision query-shape test coverage. (2512a12, bd2963a)
• A downstream hop's TRANSITION_TO/ESCALATE_TO could leak onto the primary run's own persisted record. The auto-chain loop's success path was already fixed to keep escalated_to/transitioned_to scoped to the primary hop's own signal, but the two sibling branches (the informed/force-gate interrupt and the recommended=monitor early-exit) still overwrote it unconditionally whenever they fired several hops into a chain.
  Live-caught: a force-gated SysAdmin hop's own signal corrupted the triage run's record, making it misclassify that hop as the remediation chapter instead of an escalation hop. Fixed by guarding all three sites and live-reverified against the real chain afterward. (6c29b6c, 6bdf194)
• --infra-config had no environment-variable fallback, unlike every sibling flag. Omitting it silently resolved {{server_id}} to an empty string, producing a confusing prompt and a 0% score with no indication why. Added the missing FAULTTEST_INFRA_CONFIG fallback. (7dc7f6a)
• host-container-stopped's catalog timeout was too tight for force-mode's synchronous multi-hop auto-chaining, so that 3 sequential LLM calls in one HTTP request routinely exceeded the original 60s budget. Bumped to 180s with a comment explaining why this fault class needs the larger window.

📚 Documentation

New:

• docs/samples/SAMPLE017.md,
• vault hop-certs coverage in docs/CONSISTENCY.md's "Certification scope" section (rewritten from "known gap" to how the fix actually works).
• docs/MUTATION_TOOLS.md §5.10, "Corroborated Decline," covering both declinedActionSignal and hasActionClassDenial.

Updated

For the time-windowed leak-prevention fix and the downstream-signal-leak fix:

• docs/INCIDENTS.md
• docs/API.md
• docs/AUDIT.md
• docs/JOURNEYS.md
• docs/AIGOVERNANCE.md
• docs/VAULT.md

For the ACTION_TAKEN: parsing change and the now-instrumented read-only tools:

• docs/SYSADMIN_AGENT.md

[boris@ ~/helpdesk]$ git log release/v0.25.0..release/v0.26.0 --oneline
6bdf194 (HEAD -> release/v0.26.0, origin/release/v0.26.0) fix: add test coverage + doc for the downstream hop's transition_to/escalation_to leaking.
6c29b6c fix: stop a downstream hop's transition_to/escalate_to from leaking onto the primary run's persisted record.
bd2963a chore: add test coverage for policy-denial corroboration.
2512a12 feat: add new policy-denial corroboration for write-absence mismatch.
eee2210 doc: update the docs for the time-based window leak-prevention feature.
ff44cc8 chore: add test covearge for time-based window change.
995107c feat: cross-hop verification-flag leak prevention in the incident narrative.
c182ef8 fix: sysadmin agent's prompt/classification fix for pure diagnostic PBs.
0bf0074 doc: date the doc for all 3 enhancements.
625103b chore: add additional test coverage for two more testing gaps.
602f2e6 chore: add test coverage for the new ToolAuditor.RecordToolCall.
a5d5a99 feat: add tool-execution audit instrumentation in sysadmin and db agents.
578128e feat: Tools: 0% and Dirty->Clean for multi-hop triage.
b4a4323 doc: minor cosmetic fixes in sample 017 doc.
0d87f56 doc: fix a link in sample 017.
7dc7f6a fix: fault injection test failure on TestFaultInjection/db-wal-disk-full.
e475bbe doc: update the docs, include `vault hop-certs`.
740d0f7 chore: add test coverage, unit + integration for interm hop PBs stability certs.
ed9868c feat: every hop in a diag chain earns its own stability cert.
2288748 feat: vault.go's journeySummary struct unnecessarily mirrored audit.JourneySummary.
f7a0d91 chore: add an injector test teardown coverage.
cd256ce feat: add the new per-fault context.WithTimeout feature.
1e07656 chore: last part of the faulttest dedup refactor.
621e887 chore: continue with the faulttest refactor: config dedup.
208100b chore: continue dedup faulttest: evaluator.go dedup.
83616c7 chore: continue with the faulttest dedup: HarnessConfig embedding refactor.
45dd480 feat: prod readiness: start dedup of a long standing faulttest/faultlib dedup.
e4ddbbe (origin/main, origin/HEAD, main) doc: minor numbering in sample 017 doc.
721d635 doc: add two more sections in sample 017 doc.
80ab32a doc: add sample 017 doc page.
525db21 (tag: v0.25.0) Merge pull request #25 from borisdali/release/v0.25.0

Full Changelog: v0.25.0...v0.26.0