v0.25.0

🚀 aiHelpDesk v0.25.0: The Cert That Doesn't Go Stale

Release v0.24.0 closed the verification gaps, so that every claim about what happened on a run became checkable.
This v0.25.0 release closes a second, quieter gap: a cert that was true when it was earned doesn't necessarily stay true.
A playbook gets edited. A model gets swapped. A once-earned STABLE+CLEAN verdict keeps displaying as current with nothing to say otherwise. Until now.

✅ Cert history: not just today's verdict, but a history and the trend.

vault accuracy <fault-id> now appends a Cert history (last N) section once a fault+model pair has been re-certified more than once. Each row represents a past recertification, most recent first. Adjacent rows are diffed automatically and a ↳ changed since ... line reports exactly what changed (IsStable/IsClean/AttributionConsistent flips, warning_distribution deltas naming which specific signal appeared or disappeared, playbook_version/taxonomy_version changes). This is the "why" not just the "what". With history. Backed by a new append-only fault_stability_cert_history table.

link to doc

✅ Attribution consistency joins the trust gate.

trustNotYetEarnedForceGate, the adaptive gate checker that decides whether a real (not a failure injection test) incident is allowed to auto-chain into remediation unattended, now requires AttributionConsistent, not just IsStable + IsClean. This is a small, but an important adjustment that we found live: a fault scored 100% pass rate while the model's own root-cause attribution split 3-ways across runs, passing every existing gate while silently disagreeing with itself on why. Not anymore.

link to doc

✅ Predictable-vs-inconsistent warning annotation.

vault accuracy's warning-type breakdown now distinguishes a signal that fires on every run of a fault. We refer to these warning types as predictable (i.e. structurally baked in, not fixable by prompting) vs. varies (i.e. the one that fires on occasion). The latter, varies type is worth investigating. This new warning type (predictable/varies) is derived from the same warning_distribution/n_runs data already on the cert.

link to doc

✅ get_events objective evidence, which is now a second real source.

We started with the objective_evidence (the CLEAN cert's strongest, most deterministic signal) in the v0.24 release with exactly one source: pod restart/OOMKilled state from get_pods. recordEventDistressEvidence mirrors that pattern against get_events' typed Evicted/FailedScheduling/node-pressure reasons. That's our second independent source (which incidentally surfaced the fix for a real multi-signal bug: objectiveEvidenceForceGate previously reported only the first matching signal per hop, silently dropping any others once two real sources existed).

✅ Playbook-version staleness detection, with a fix in hand.

A cert now stores the playbook's version, updated_at and concrete pb_* ID at the moment it was earned.
vault accuracy compares the stored version against the playbook's live current version on every lookup and warns when they diverge. Moreover when both the stored and current pb_* IDs are known, it also prints a ready-to-run faulttest vault diff <id-then> <id-now> ready-to-execute command instead of merely asserting "the playbook changed" with no way to act on it.

✅ Helm-native vault queries and custom-catalog mounting.

A new lightweight, read-only vault-query Job template (vaultQuery.enabled/.subcommand/.target) runs any faulttest vault <subcommand> in-cluster without hand-crafting a Pod manifest or granting RBAC.
faulttest.catalog (set via --set-file) mounts a customer fault catalog as a shared ConfigMap, consumed by the existing faulttest/recertify Jobs and the new vault-query Job alike.
This closes a real gap where a documented "fully supported" extension point only actually worked if the customer hand-wrote K8s manifests themselves.

Regression alerts, the moment they happen.

Every cert Upsert now detects, transactionally, whether this specific recertification is the moment a fault+model stopped meeting the STABLE + CLEAN + Attribution-Consistent bar it previously met.
When one does, faulttest run --repeat N fires the same --notify-url webhook already used for end-of-run reports. Immediately, not after the rest of a possibly-long batch finishes.

recertify reaches custom faults and fires the regression webhook.

The scheduled recertification CronJob gained .Values.recertify.notifyURL and picked up the same shared catalog mount. Previously it could neither recertify a customer's own faults nor ever trigger the regression alert from unattended,
scheduled recerts.

A trust-gate hint for chain-only playbooks.

trustNotYetEarnedForceGate checks the currently-completing hop's own trust before letting it hand off further. This is important because that means a chain-only playbook (never any fault's own designated entry point, only ever reached mid-chain) can permanently fail this gate with no --repeat run able to fix it. The gate now adds a trust_gate_note explaining the likely cause whenever the blocked hop was itself reached via chaining, pointing at docs/CONSISTENCY.md's new "Certification scope" section. The full fix would be to certify every distinct series a chain actually passes through, not just the fault's declared entry point. This is scoped for v0.26+ release.

🐛 Bug fixes

• proxyToAuditd's generic query-string forwarding double-appended on 5 pre-existing routes that had their own manual RawQuery workarounds, breaking vault list's playbook lookups (INCIDENTS: MISSING) the moment the generic fix shipped. Found live. All 5 sites fixed, with a regression test.
• fault_stability_cert_history never got a playbook_id migration. addVersioningColumnsSQLite()/migratePostgres() only ever ALTER TABLE'd the main cert table, while the history table had no forward-migration path. Only CREATE TABLE IF NOT EXISTS, a no-op against a table that already exists. Confirmed live against a real long-running deployment (insert cert history: SQL logic error: table fault_stability_cert_history has no column named playbook_id). Fixed for both SQLite and
  Postgres, with a regression test that reproduces the exact error before the fix and passes after.
• vaultQuery.enabled panicked helm upgrade --reuse-values on a pre-existing release. A brand-new top-level values key is genuinely nil (not an empty map) on a release that predates it, where {{ if .Values.vaultQuery.enabled }} could crash the
  template. Found live on a real --reuse-values upgrade. Fixed with a nil-guard, verified via a direct -f <file with vaultQuery: null> reproduction and a permanent regression test.
• make e2e could silently test the wrong stack. Nothing verified that localhost:8080 actually reached the intended docker-compose stack make e2e had just built. That is, a stale kubectl port-forward or another locally-run gateway process could squat on the port for hours, with docker compose --wait's container-internal healthcheck reporting "Healthy" the whole time regardless. Fixed via a new pre-flight check that compares the running Gateway's reported version against the exact build tag this invocation produced, failing fast with an actionable error instead of quietly testing stale code for 5+ minutes.
• A stale want 204 assertion in TestGatewayFaultStability_AttributionRoundtrip, left over from before v0.25.0 changed the Upsert endpoint's success response to 200 {"regressed":bool}.

📚 Documentation

New:

• docs/CONSISTENCY.md's "Certification scope: entry-point playbooks only (known gap)" subsection.
• docs/samples/SAMPLE016.md, a full walkthrough sample demonstrating the CLEAN cert's five signals including target_drift via a custom fault catalog.

Updated:

• docs/VAULT.md#vault-accuracy (cert-history section, trend-diff lines, playbook-version staleness + vault diff hint examples, disambiguation against the pre-existing, unrelated vault history command),
• docs/CONSISTENCY.md's versioning-fields table and §7.3,
• docs/ATTRIBUTION_CERTS.md's "three suspects" section,
• docs/FAULTTEST.md (new §9.6 "Running a custom catalog from Helm").

List of changes

[boris@ ~/helpdesk]$ git log release/v0.24.0..release/v0.25.0 --oneline
d917a20 (HEAD -> release/v0.25.0, origin/release/v0.25.0) feat: cert scope limited to entry-point PBs only (known gap, not chained).
d637d1b fix: addVersioningColumnsSQLite()/migratePostgres() only handled forward-migtation for fault_stability_cert, not fault_stability_cert_history.
516545d fix: bug in the previous attempt to fix the make file with the sanity check (the image tag/sha was off).
d8950cc fix: e2e error fixes: helm template + make file sanity check for port 8080 + old bug in gateway.
9aed467 chore: add unit test coverage for cert history + vault diff + vault query.
5f97347 feat: vault diff linkage via playbook-ID capture.
57578f1 feat: add cert history reasoning/trending + helm --catalog/--notify-url.
2213442 fix: regression due to the new cert history.
33ff4d3 doc: a few more cosmetic changes for sample 016 doc.
ba6fbc3 doc: minor fixes in sample 016 doc.
0124394 chore: 3 new/fixed tests in e2e + 2 new (verified passing) in integration.
9b6d6b3 chore: add test coverage for playbook version staleness + docs.
d10d079 feat: playbook-version tracking + cert history + regression alert on any stable+clean cert.
1993345 chore: add test coverage for objective evidence multi tool calls.
eb58290 feat: add get_events objective_evidence.
cbe120c feat: add a predictable-vs-inconsistent annotation to --repeated N and `vault accuracy`.
7b8e292 feat: add trust-gate AttributionConsistent to the stable/clean cert.
0c77a22 (origin/main, origin/HEAD, main) doc: fix minor typos and add case 6 to sample 016.
457961d doc: add sample 016 doc.
1e929f3 (tag: v0.24.0) Merge pull request #24 from borisdali/release/v0.24.0

Full Changelog: v0.24.0...v0.25.0