🚀 aiHelpDesk v0.27.0: Tested Where It Counts
Release v0.26.0 made sure every hop in a chain earned its own trust. v0.27.0 asks a blunter question about the trust mechanism itself: does the evidence backing a diagnosis actually prove what it claims and does that proof reach every deployment platform aiHelpDesk ships on?
A few release highlights:
✅ Upstream deps release bump
As part of the ongoing production readiness effort (towards the upcoming GA release), in v0.27 we updated the upstream dependencies for both A2A (bump from v0.3.3 to v0.3.15) and ADK-Go (bump from v0.6.0 to v1.6.0).
✅ Objective evidence framework
Feature-wise, the headline of this release is the objective evidence framework. We started with a simple idea, hardcoded it initially into a single tool (K8s Agent's get_pods), let the trusted users / early adopters provide feedback, extended it to a few more tools (e.g. get_events), then to another agent (from K8s Agent to DB Agent) and in this release we generalize it to a full declarative, cross-domain framework.
The framework rests on internal/evidence that provides a strongly typed-probe, which is further combined with the customer-facing YAML-rule mechanism layered on top (ToolSchema[T], LoadRules, Evaluate), replacing what initially used to be a few hand-written, K8s-specific functions.
A tool registers a small, fixed set of named probes. An operator-editable YAML file says which probes to threshold and what signal results from it. The rules reference probes by name, so a typo is a loud load-time error, not a silently-dead check.
Additionally, the force-gate has been enhanced to checks correctness, not just the presence. The initial implementation gated identically on any objective evidence firing, regardless of whether the model's own diagnosis got it right.
This was a good start when the evidence only ever appeared alone. With the real, more complex, multi-hop, cross-domain triages, the moment a hop needed to both cite evidence and chain onward, the initial implementation became insufficient. A second, symmetric probe layer over the model's own parsed output means a textbook-correct diagnosis that correctly cites the fired signal no longer trips the same gate as a wrong one. That's why we now additionally introduced the confirmed/unconfirmed status to surface explicitly instead of only being inferable by absence.
The DB Agent gets its first two real objective evidence consumers in this release: get_active_connections (idle_in_transaction_seconds) and get_replication_status. The latter is a synthesized per-call ReplicationSummary whose disconnected probe requires a primary role + zero connected replicas + an inactive lagging slot together. Not just "zero replicas" alone, which would false-positive on a primary that never had one attached.
As of this release the objective evidence now reaches all 3 deployment platforms, so that K8s, Host and Docker Compose all gained the necessary env/ConfigMap/volume wiring.
✅ 3rd LLM anti-fabrication layer
All of this now allowed us to unveil the third layer of the LLM fabrication detection and protection mechanism: objective-evidence content verification.
Layers 1 and 2 both verify that a claimed action really happened (e.g. a mutation took effect). Neither checks whether the content of what a model reports about a R/O tool's result is accurate. This is different because a model can genuinely call get_replication_status, then still misdescribe what it returned. That's the gap that Layer 3 is closing.
✅ Enhanced scoring
As of this release the failure injection testing catalog scoring has been enhanced and it now depends on confirmed evidence too. We refer to it as the evidence-content gate. This was prompted by a direct external feedback about what a pass should mean when the evidence query itself comes back empty.
✅ New Replication faults
Meet three new faults (in addition to the previously shipped db-replication-lag): db-replica-disconnected, db-replica-container-stopped and db-replica-stalled. The first of the trio (db-replica-disconnected) now escalates to two new SysAdmin playbooks that inspect the replica's own container/logs. This is a genuine diagnostic value a DB Agent could hardly ever see.
The last one of the trio (db-replica-stalled) covers a replica that's connected and current, but has stopped applying WAL, not to be confused with both lag and disconnection.
🐛 Bug fixes
- Hypotheses re-ranking bug in competing-hypothesis display ordering.
psqlx(expanded psql output) parsing edge case.db-replica-disconnected's injection mechanism made durable (its original approach silently never took effect because the setting it relied on requires a full server restart, not a reload) plus afresh-connection-per-pollretry loop for a genuine, non-obvious Postgres visibility quirk.auditd's Docker Compose entrypoint silently ignored any passed override command.
📚 Documentation
- New docs/OBJECTIVE_EVIDENCE.md
- Updated docs/FAULTTEST.md
- Updated docs/HA_DR.md
List of changes
[boris@ ~/helpdesk]$ git log release/v0.25.0..release/v0.27.0 --oneline
591c9f7 (HEAD -> release/v0.27.0, origin/release/v0.27.0) fix: minor fixes for objective evidence in docker compose yaml + the entry point to generate hashes.
3a0d5c0 fix: clamping issue in docker containers preventing auto-chaining to sysadmin agent.
6941446 doc: update faulttest and objective evidence docs with the addition of the evidence-based scoring extension.
03216d0 chore: add test coverage to the evidence-based scoring extension.
94aefcd feat: add conditional evidence-gate extension into testing/faultlib.
13f3f63 feat: enhance scoring to include objective evidence signal (if defined).
f9bf0aa feat: wire objective evidence framework to all 3 depl platforms.
a96d637 fix: hypotheses re-rank bug.
a8c74b0 doc: update the ha/dr doc with minor changes for `db-replica-stalled` (it's actually bridgeable) and so is the inject side of `db-replica-container-stopped`.
c902b20 doc: update the docs for the new db-replica-stalled fault.
40c1d8b fix: psqlx parsing issue.
d448172 fix: bump up the timing thresholds.
b48faae feat: add new db-replica-stalled fault + pivot (Ending C) replication-lag-triage.yaml PB to diagnose it.
e5ac38e chore: add integration test + update the docs.
e8a9631 chore: add unit test coverage for the obj evidence additions to the vault.
ec78924 feat: add visibility to the new objective evidence framework via the vault commands.
2fc8f7a doc: add the new top-level objective evidence page + links.
ea74bca chore: add diagnostic as the default purpose for running faulttests.
0521261 feat: allow faulttest to derive the remediation PB's type and set the purpose accordingly (elevating from diagnostic if needed).
d4cbd7b fix: add missing wiring for the objective evidence confirmed/unconfirmed indicator in faulttest.
cd2af2b feat: add explicit objective evidence confirmed indicator for ease of troubleshooting.
4be0795 feat: enhance the existing objective evidence framework.
67286d1 feat: add two new sysadmin's PBs: replica connectivity triage + remediation (replica-restart).
0113db1 fix: minor testing issues with item 5 for db-replica-disconnected fault.
101374d chore: add test coverage for the db objective evidence.
13403a6 feat: introduce the objective evidence (from k8s) to the db agent.
05cbeb1 feat: new evidence package to support objective evidence declaratively.
f2d7b24 chore: A2A and ADK-Go version bumps to 0.3.15 and 1.6.0 respectively.
ddc994b doc: another doc typo, now in samples readme.
48d0e8d doc: fix minor issues in the sample 018 doc.
3d2c218 doc: add sample 018 doc.
6359322 feat: add docker cli to control fault injection testing of `type: docker`.
5f3afa9 fix: TestGetHistory_RespectsLimit unit test sporadic failures.
ae5b457 doc: minor correction in ha/dr doc.
438d08b chore: add test coverage + docs for the new db-replica-disconnected fault.
5761ec9 feat: added new fault `db-replica-disconnected` to verify the "absent row != zero lag" edge case.
c63b4a9 (tag: v0.26.0, origin/main, origin/HEAD, main) Merge pull request #26 from borisdali/release/v0.26.0
a7b2723 (origin/release/v0.26.0, release/v0.26.0) fix: linter issue with targetDriftDetail no longer used.
6bdf194 fix: add test coverage + doc for the downstream hop's transition_to/escalation_to leaking.
6c29b6c fix: stop a downstream hop's transition_to/escalate_to from leaking onto the primary run's persisted record.
bd2963a chore: add test coverage for policy-denial corroboration.
2512a12 feat: add new policy-denial corroboration for write-absence mismatch.
eee2210 doc: update the docs for the time-based window leak-prevention feature.
ff44cc8 chore: add test covearge for time-based window change.
995107c feat: cross-hop verification-flag leak prevention in the incident narrative.
c182ef8 fix: sysadmin agent's prompt/classification fix for pure diagnostic PBs.
0bf0074 doc: date the doc for all 3 enhancements.
625103b chore: add additional test coverage for two more testing gaps.
602f2e6 chore: add test coverage for the new ToolAuditor.RecordToolCall.
a5d5a99 feat: add tool-execution audit instrumentation in sysadmin and db agents.
578128e feat: Tools: 0% and Dirty->Clean for multi-hop triage.
b4a4323 doc: minor cosmetic fixes in sample 017 doc.
0d87f56 doc: fix a link in sample 017.
7dc7f6a fix: fault injection test failure on TestFaultInjection/db-wal-disk-full.
e475bbe doc: update the docs, include `vault hop-certs`.
740d0f7 chore: add test coverage, unit + integration for interm hop PBs stability certs.
ed9868c feat: every hop in a diag chain earns its own stability cert.
2288748 feat: vault.go's journeySummary struct unnecessarily mirrored audit.JourneySummary.
f7a0d91 chore: add an injector test teardown coverage.
cd256ce feat: add the new per-fault context.WithTimeout feature.
1e07656 chore: last part of the faulttest dedup refactor.
621e887 chore: continue with the faulttest refactor: config dedup.
208100b chore: continue dedup faulttest: evaluator.go dedup.
83616c7 chore: continue with the faulttest dedup: HarnessConfig embedding refactor.
45dd480 feat: prod readiness: start dedup of a long standing faulttest/faultlib dedup.
e4ddbbe doc: minor numbering in sample 017 doc.
721d635 doc: add two more sections in sample 017 doc.
80ab32a doc: add sample 017 doc page.
525db21 (tag: v0.25.0) Merge pull request #25 from borisdali/release/v0.25.0
Full Changelog: v0.26.0...v0.27.0