v0.20.0
🚀 aiHelpDesk v0.20.0 release: The Trusted, Fully Accountable System
This release made every published claim about aiHelpDesk Operational SRE/DBA Flywheel verifiable! Model certs are now per-model. Judge verdicts are now auditable. aiHelpDesk customers have now a formal, documented Bill of Rights. And when the data behind a stability metric is thin, the system says so.
✅ Production Readiness
Major agentutil refactoring. This is a central component used by all agents with now server wiring moved to agentutil/serve and retry logic to agentutil/retryutil. Library logic is now independently testable.
✅ vault cert-compare: model upgrade gating
Compare triage diagnosis stability certs across two model names side-by-side. Shows which faults regressed, which improved, and which haven't been certified yet for the new model. Regressions sort first.
faulttest vault cert-compare gemini-3.5-flash claude-opus-4-5
Model names are opaque strings, which means that vault cert-compare works for any vendor switch (Gemini → Claude, Opus → Sonnet, etc.). diag_rate reflects triage scoring only (keyword + tool + category across --repeat runs)
[boris@ /tmp/helpdesk/helpdesk-v0.20.0-darwin-arm64]$ ./faulttest vault cert-compare \
> gemini-3.5-flash claude-sonnet-4-6 \
> --gateway $GW --api-key $KEY
Gateway: http://localhost:8080 · version: v0.20.0-0469ff0 · host: <your-host>
Stability comparison: gemini-3.5-flash → claude-sonnet-4-6
STABLE/UNSTABLE = triage diagnosis cert only (keyword + tool + category scoring across --repeat runs; remediation not included)
FAULT gemini-3.5-flash sonnet-4-6 CHANGE
────────────────────────────────────────────────────────────────────────────────
Lock Contention / Deadlock STABLE UNSTABLE ⚠ REGRESSION
db-lock-contention diag_rate: 100% → 40% (Δ-60%)
Max Connections Exhaustion UNSTABLE STABLE ✓ IMPROVEMENT
Vacuum Bloat STABLE (no data) ? NOT RUN YET
3 fault(s) total · 1 regression(s) ⚠ · 1 improvement(s) ✓ · 1 not run under both models ?
⚠ 1 regression(s) detected — promote claude-sonnet-4-6 only after investigating the faults above.
✅ vault judge-accuracy: judge gets accountability
Shows whether judge verdicts recorded at vault diff --judge time proved correct once run data accumulated. Per-series breakdown: what the judge predicted, what the data showed, whether they agreed. The judge_verdict column is also now
visible in vault versions.
faulttest vault judge-accuracy [--gateway URL] [--api-key KEY]
[boris@ /tmp/helpdesk/helpdesk-v0.20.0-darwin-arm64]$ ./faulttest vault judge-accuracy --gateway $HELPDESK-GATEWAY-URL --api-key $HELPDESK-API-KEY
Gateway: http://localhost:8080 · version: dev · host: <your-host>
SERIES VERSION JUDGE VERDICT RUNS SUCCESS% JUDGE MODEL
───────────────────────────────────────────────────────────────────────────────────────────────────────────
pbs_vacuum_triage 1.5 APPROVE 1 100% claude-haiku-4-5-20251001
JUDGE VERDICT is the prediction recorded by `vault diff`.
SUCCESS% is the actual outcome after runs on this version.
[boris@ /tmp/helpdesk/helpdesk-v0.20.0-darwin-arm64]$ ./faulttest vault versions pbs_vacuum_triage --gateway $HELPDESK-GATEWAY-URL --api-key $HELPDESK-API-KEY
Gateway: http://localhost:8080 · version: dev · host: <your-host>
Version stats for pbs_vacuum_triage
VERSION RUNS SUCCESS% AVG STEPS AVG TIME AVG DIAG AVG REMED APPROACH OK JUDGE VERDICT
────────────────────────────────────────────────────────────────────────────────────────────────────
1.3 12 83% – 17s 85% – – –
id=pb_41a441cd
1.4 9 88% – 1m26s 81% 100% 100% –
id=pb_71ead4d4 from=faulttest-2519dede-db-vacuum-needed-r2
1.5 * 1 100% – 6m32s 100% 100% 100% APPROVE
id=pb_68e0c666 from=faulttest-e2157c1b-db-vacuum-needed judge=claude-haiku-4-5-20251001
* = currently active SUCCESS% = resolved + transitioned
id/from lines show playbook_id and the run that generated that version
✅ vault import: playbook import from YAML
Import a local playbook YAML file from the CLI. Lands as a draft (visible in vault drafts), activatable with --activate. The approval_mode field round-trips correctly.
faulttest vault import playbook.yaml [--activate] [--force]
[boris@ /tmp/helpdesk/helpdesk-v0.20.0-darwin-arm64]$ ./faulttest vault import /tmp/my-playbook.yaml --gateway $HELPDESK-GATEWAY-URL --api-key $HELPDESK-API-KEY
Gateway: http://localhost:8080 · version: dev · host: <your-host>
Parsed: Connection Pool Exhaustion — Manual Import v ()
Saved: pb_545daf1d v series=pbs_50cb5d07
To diff before activating:
faulttest vault diff pb_545daf1d --gateway http://localhost:8080
To activate:
faulttest vault activate pb_545daf1d --gateway http://localhost:8080
[boris@ /tmp/helpdesk/helpdesk-v0.20.0-darwin-arm64]$ ./faulttest vault drafts pbs_vacuum_triage --gateway $HELPDESK-GATEWAY-URL --api-key $HELPDESK-API-KEY
Gateway: http://localhost:8080 · version: dev · host: <your-host>
Pending drafts — 8 awaiting review
DRAFT ID SERIES VERSION NAME CREATED
─────────────────────────────────────────────────────────────────────────────────────────────────────────────
pb_545daf1d pbs_50cb5d07 Connection Pool Exhaustion — Manual Imp… 2026-07-07 ↑
pb_68e0c666 pbs_vacuum_triage Vacuum & Bloat Triage 2026-07-02
pb_34f99e9c pbs_vacuum_triage Vacuum & Bloat Triage 2026-07-02
pb_e047968e pbs_vacuum_triage Vacuum & Bloat Triage 2026-07-02
pb_faefa188 pbs_vacuum_triage Vacuum & Bloat Triage 2026-07-02
pb_a7380179 pbs_generated_3eb6c3e5 PostgreSQL Vacuum Bloat Detection and Ana… 2026-07-02 !
pb_ea59a5aa pbs_generated_11d9b85b PostgreSQL Table Bloat and Vacuum Recovery 2026-07-02 !
pb_6bf82716 pbs_generated_91243b76 PostgreSQL Table Bloat and Vacuum Deficit… 2026-07-02 !
! = orphan draft (series not pinned); run with --purge-orphans to delete all 3
↑ = imported via vault import
To activate a draft:
faulttest vault activate <DRAFT_ID> --gateway http://localhost:8080 --api-key <key>
✅ vault list: full diag/remed breakdown by version
[boris@ /tmp/helpdesk/helpdesk-v0.20.0-darwin-arm64]$ ./faulttest vault list --gateway $HELPDESK-GATEWAY-URL --api-key $HELPDESK-API-KEY
Gateway: http://localhost:8080 · version: dev · host: <your-host>
FAULT PLATFORM DIAG PLAYBOOK REMED PLAYBOOK LAST TEST STABLE INCIDENTS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
db-max-connections any pbs_connection_triage pbs_connection_remediate 2026-07-07 PASS UNSTABLE(3) 13 runs 62% resolved 100% accurate avg: 2.1 steps, 22s recovery (generated)
diag 1.4 * 1r 100% avg: 58s recovery 100% approach OK
diag 1.3 19r 74% avg: 37s recovery 100% approach OK
remed 1.4 * 4r 75% avg: 13s recovery
remed 1.3 9r 100% avg: 3.0 steps, 26s recovery
db-long-running-query any pbs_slow_query_triage pbs_slow_query_remediate 2026-06-22 PASS STABLE(5) 14d 2 runs 100% resolved 100% accurate avg: 4.0 steps, 10s recovery (system)
diag → vault versions pbs_slow_query_triage
remed → vault versions pbs_slow_query_remediate
db-lock-contention any pbs_lock_contention_triage pbs_slow_query_remediate 2026-06-23 PASS STABLE(5) 14d 2 runs 100% resolved – avg: 4.0 steps, 10s recovery (system)
diag → vault versions pbs_lock_contention_triage
remed → vault versions pbs_slow_query_remediate
db-table-bloat any pbs_vacuum_triage pbs_vacuum_remediate 2026-06-22 PASS STABLE(5) 14d 11 runs 91% resolved 100% accurate avg: 7.2 steps, 9m48s recovery (system)
diag 1.5 * 1r 100% avg: 6m32s recovery 100% approach OK
diag 1.4 9r 89% avg: 1m26s recovery 100% approach OK
diag → vault versions pbs_vacuum_triage
remed → vault versions pbs_vacuum_remediate
db-high-cache-miss any pbs_cache_miss_triage pbs_cache_miss_remediate 2026-07-04 PASS STABLE(5) 14d 1 runs 100% resolved – avg: 4.0 steps, 11s recovery (system)
diag → vault versions pbs_cache_miss_triage
remed → vault versions pbs_cache_miss_remediate
db-connection-refused any pbs_db_restart_triage pbs_db_restart_triage (never) — 1 runs 0% resolved – avg: 6s recovery (system)
→ vault versions pbs_db_restart_triage
db-auth-failure any pbs_auth_failure_triage (none) (never) — -
db-not-exist any - (none) NO PLAYBOOK — -
db-replication-lag any pbs_replication_lag pbs_replication_remediate READY — 0 runs (system)
db-idle-in-transaction any pbs_connection_triage pbs_connection_remediate 2026-06-22 PASS STABLE(5) 14d 13 runs 62% resolved 100% accurate avg: 2.1 steps, 22s recovery (generated)
diag 1.4 * 1r 100% avg: 58s recovery 100% approach OK
diag 1.3 19r 74% avg: 37s recovery 100% approach OK
remed 1.4 * 4r 75% avg: 13s recovery
remed 1.3 9r 100% avg: 3.0 steps, 26s recovery
db-tx-lock-chain-blocker any pbs_lock_chain_triage pbs_lock_chain_remediate 2026-07-01 FAIL STABLE(5) 14d 3 runs 33% resolved 100% accurate avg: 19.3 steps, 1m13s recovery (system)
diag → vault versions pbs_lock_chain_triage
remed → vault versions pbs_lock_chain_remediate
db-terminate-direct-command any - (none) NO PLAYBOOK STABLE(5) 14d -
k8s-crashloop k8s pbs_k8s_pod_crash_triage pbs_k8s_pod_crash_remediate READY — 0 runs (system)
diag → vault versions pbs_k8s_pod_crash_triage
k8s-pending k8s pbs_k8s_pending_triage (none) (never) — -
k8s-image-pull k8s pbs_k8s_image_pull_triage (none) (never) — -
k8s-no-endpoints k8s pbs_k8s_no_endpoints_triage (none) (never) — -
k8s-pvc-pending k8s pbs_k8s_pvc_triage (none) (never) — -
k8s-oomkilled k8s pbs_k8s_pod_crash_triage pbs_k8s_pod_crash_remediate READY — 0 runs (system)
diag → vault versions pbs_k8s_pod_crash_triage
k8s-scale-to-zero k8s pbs_k8s_scale_to_zero_triage pbs_k8s_scale_to_zero_remediate READY — 0 runs (system)
db-vacuum-needed any pbs_vacuum_triage pbs_vacuum_remediate 2026-07-07 PASS STABLE(5) 14d 11 runs 91% resolved 100% accurate avg: 7.2 steps, 9m48s recovery (system)
diag 1.5 * 1r 100% avg: 6m32s recovery 100% approach OK
diag 1.4 9r 89% avg: 1m26s recovery 100% approach OK
diag → vault versions pbs_vacuum_triage
remed → vault versions pbs_vacuum_remediate
db-disk-pressure any pbs_disk_pressure_triage (none) 2026-06-22 PASS STABLE(5) 14d -
→ vault versions pbs_disk_pressure_triage
host-container-stopped docker/vm pbs_db_restart_triage (none) (never) — -
→ vault versions pbs_db_restart_triage
host-pg-crash docker/vm pbs_db_restart_triage (none) (never) — -
→ vault versions pbs_db_restart_triage
db-pg-hba-corrupt any pbs_pg_hba_triage pbs_db_config_recovery 2026-06-22 FAIL UNSTABLE(1) 14d 0 runs (system)
db-process-kill any pbs_db_restart_triage pbs_db_restart_triage 2026-06-22 FAIL UNSTABLE(1) 14d 1 runs 0% resolved – avg: 6s recovery (system)
→ vault versions pbs_db_restart_triage
db-config-bad-param any - (none) NO PLAYBOOK UNSTABLE(1) 14d -
compound-db-pod-crash multi - (none) NO PLAYBOOK — -
compound-db-no-endpoints multi - (none) NO PLAYBOOK — -
db-wal-disk-full docker/vm - (none) NO PLAYBOOK UNSTABLE(1) 14d -
db-wal-disk-full-k8s k8s pbs_k8s_pod_crash_triage (none) 2026-06-22 PASS UNSTABLE(5) 14d -
→ vault versions pbs_k8s_pod_crash_triage
db-wal-stale-slot any pbs_wal_stale_slot pbs_stale_slot_remediate 2026-06-29 PASS STABLE(5) 14d 14 runs 79% resolved 100% accurate avg: 3.4 steps, 24s recovery (system)
diag 1.3 * 8r 100% avg: 2m12s recovery 100% approach OK
diag 1.2 9r 0% avg: 3m55s recovery
remed → vault versions pbs_stale_slot_remediate
db-checkpoint-warning any pbs_checkpoint_bgwriter_triage pbs_bgwriter_remediate 2026-06-22 PASS STABLE(5) 14d 1 runs 100% resolved – avg: 6.0 steps, 16s recovery (system)
diag → vault versions pbs_checkpoint_bgwriter_triage
remed → vault versions pbs_bgwriter_remediate
✅ vault journey --detail: execution trace
Drill into a Journey to see agent reasoning interleaved with tool calls, the original query, delegation chain, incident run ID link and the linked remediation journey when present.
faulttest vault journey <trace_id> --detail
faulttest vault journey --incident --limit 20 # defaults to 7-day window
--incident now defaults to 7 days (was 24h): incident-linked journeys accumulate over days and were silently invisible under the old default.
✅ Calibration data quality banner
vault calibration warns when the human-feedback dataset behind a stability metric is thin. A cert backed by 2 human verdicts is not comparable to one backed by 50. The banner surfaces this before you act on the number.
[boris@ /tmp/helpdesk/helpdesk-v0.20.0-darwin-arm64]$ ./faulttest vault calibration db-long-running-query --gateway $HELPDESK-GATEWAY-URL --api-key $HELPDESK-API-KEY
Gateway: http://localhost:8080 · version: dev · host: <your host>
┌─────────────────────────────────────────────────────────────────────┐
│ ⚠ Data quality: 0 of 1 run(s) have human operator feedback.
│ This table measures self-consistency (LLM judge vs. itself),
│ not calibration against human judgment. To build a meaningful
│ calibration dataset, run faulttest interactively (without
│ --approval-mode force) or submit feedback via:
│ faulttest vault feedback <run-id> --gateway $GW --api-key $KEY
└─────────────────────────────────────────────────────────────────────┘
Diagnosis calibration — pbs_slow_query_triage (1 runs with agent confidence + operator feedback)
CONFIDENCE RUNS CORRECT ACCURACY CALIBRATION
─────────────────────────────────────────────────────────────────
90-100% 1 1 100% INSUFFICIENT_DATA ⚠ 1/1 keyword (no judge)
70-89% 0 0 – INSUFFICIENT_DATA
<70% 0 0 – INSUFFICIENT_DATA
Note: all 1 run(s) above use auto_judge feedback (LLM judge score ≥ 0.8, --approval-mode force).
Calibration measures self-consistency, not human judgment. Run interactively to collect real operator verdicts.
✅ Trigger context on playbook runs
PlaybookRun now stores the original alert text or context string that initiated the run. Visible in vault incidents and the incident narrative, which now closes the "what fired this?" gap in the audit trail.
[boris@ /tmp/helpdesk/helpdesk-v0.20.0-darwin-arm64]$ curl -s "$HELPDESK-GATEWAY-URL/api/v1/fleet/playbook-runs/$RUN_ID" -H "Authorization: Bearer $HELPDESK-API-KEY" | jq '{run_id, trigger_context}'
{
"run_id": "plr_828472f2",
"trigger_context": "PagerDuty #1234: connection count at 93% on prod-pg-01"
}
✅ Model-scoped stability certs
fault_stability_cert is now keyed by (fault_id, diagnosis_model). Previously all models shared a single row per fault, so switching models silently overwrote the prior cert. Each model now has its own cert row; cert history is preserved across model switches. Migration runs automatically on first startup.
✅ Internal reasoning directive
All agents now emit a reasoning block before any action. This populates the --detail execution trace and makes the Journey audit trail interpretable without reading raw tool output.
Two new accountability documents
- CUSTOMER_RIGHTS.md:
10 explicit commitments covering what aiHelpDesk will and won't do, what data you own, what you can audit and where the gaps are. Written to be honest, not aspirational. - SECOND_OPINION.md:
Guidance for customers who want to verify or challenge an AI recommendation before acting. Covers when to get a second opinion, how to frame the request and what to do when the AI is wrong.
Improvements
vault versions: fixed a logic issue where a newly imported playbook would disappear from the pending list until run stats were gathered.- Positional args in
faulttest: flags and positional arguments can now be given in any order.faulttest run db-max-connections --repeat 5andfaulttest run --repeat 5 db-max-connectionsare now equivalent. - Playbook count in docs corrected:
VAULT.mdstated "7 expert-authored system Playbooks." The correct count is 34: 24 PostgreSQL, 8 Kubernetes, 2 Host/Docker.
Bug fixes
- 1/
vault journey --incident --detail: execution trace was missing (returned header only) - 2/
vault journey --incident: wasn't returning incident-linked journeys (server-side filter regression) - 3/
vault journey --incident: origin field (agent / direct_tool / agent_approve) was missing from the detail header - 4/
vault journey --incident: historical journeys older than 24h not shown — default window widened to 7d - 5/
vault import: imported playbook wasn't appearing in vault drafts - 6/
faulttest db-wal-disk-full: intermittent failure whenlogging_collector=onsilences container stdout —get_host_logsnow returns an explicit hint directing the agent toread_pg_log_filerather than concluding "no log data" - 7/ Stability cert silently posted under empty model name when
HELPDESK_MODEL_NAMEis unset — now logs a warning and skips the POST - 8/ Calibration banner: border rendering was unclosed in some terminal widths
Compatibility
- SQLite databases from v0.19.0 upgrade automatically on first startup (
fault_stability_certcomposite PK migration, trigger_context column on playbook_runs). - No Helm chart changes required for existing K8s deployments.
faulttestbinary is fully backward-compatible with v0.19.0 gateway deployments.
List of changes
[boris@~/helpdesk]$ git log release/v0.19.0..release/v0.20.0 --oneline
b906d5e (HEAD -> release/v0.20.0, origin/release/v0.20.0) fix: problem with `vault journey --incident` not showing historical/older records.
0469ff0 fix: add a simple verification on the diagnostic model.
5006ebf fix: cosmetic issues with the compare-certs.
7ccb78d fix: TestFaultInjection/db-wal-disk-full fails again in a faulttest run.
12c9116 chore: fix vault's wording on the number of PBs we ship out of the box. Presently it's 34 (not 7).
634cd4f chore: add test coverage for vault cert-compare.
daba3ac chore: add new directive to all agents to request internal reasoning before any decisions.
ad6b744 fix: cosmetic display issue: add origin (agent/direct_tool/agent_approve) to vault journeys --incident --detail.
b9ec25e fix: vault journey --incident --detail didn't return execution trace.
95b5a67 fix: vault journey --incident doesn't return incident-linked journeys.
822e060 fix: faulttest-gateway failure on TestFaultInjection/db-wal-disk-full.
c274a62 chore: additional test coverage for vault journey --detail.
5070ef5 chore: add unit test coverage for vault journey --detail.
061c8b7 feat: add vault journeys --detail.
4f42451 chore: increase unit test coverage for agentutil after split.
48fe4d1 chore: refactor agentutil: move server wiring to agentutil/serve.
d282317 fix: vault versions had a logical issue with new PB disappearing from pending until stats gathered.
d856c20 fix: CI failure of not adding the new source column to the test.
59f92c3 fix: issue with the imported PB (vault import) not showing up as a draft (vault drafts).
8eeb117 fix: border for the data quality banner warning is not closed + document it.
a8a4b99 fix: e2e test failure in TestPlaybooks_JudgeVerdictRoundtrip.
46a677f chore: add docs for the new v0.20 features: vault import, judge accuracy, etc.
5b1b28d chore: add e2e tests for the trigger context and judge verdict.
9202546 chore: add 10 new unit tests to extend test coverage for the new v0.20 features.
b94e3a6 feat: judge-accuracy to gauge the judge accountability.
60e9b93 feat: positional args in faulttest + approval mode drop + calib banner + trigger context.
2aedeb4 chore: add the customer bill of rights + second opinion docs.
1a1f886 (origin/main, origin/HEAD, main) chore: update the bench sample 9 doc: `vault diff` to show the version diffs.
96c52f5 chore: minor cosmetic doc fidex in bench sample 8 and consistency.
9629ec2 chore: minor doc fixes in jugement layer + bench sample 9.
50fecd5 chore: add the judgment layer doc + the refs from vault and principles.
d12e19b chore: fix the connection-triage PB + docker compose timeouts + add the bench sample 9 doc.
7ad5856 fix: bug in `vault incidents` where the fault column may be incorrect.
3b5b868 fix: minor cosmetic fix in `vault list` incorrectly rendering resolution rate for triage PBs.
6278bb2 (tag: v0.19.0) fix: pin adk-go clone to v0.6.0 tag in CI and release workflows.
67dc84a Merge pull request #19 from borisdali/release/v0.19.0
Full Changelog: v0.19.0...v0.20.0