v0.31.0

🚀 aiHelpDesk v0.31.0: Restore / Recovery time!

v0.30 proved the backup was healthy or told you precisely why it wasn't.
v0.31 is the release where that distinction stops being academic: when the data directory itself is gone, aiHelpDesk now diagnoses it correctly, proposes the one correct fix, gates it behind Informed Consent and actually restores from backup. And because "detect" and "destroy-then-restore" are very different leagues of trust, this release also hardens the evidence pipeline underneath every force-gate and generalizes real SSH dispatch beyond the handful of pgBackRest-specific tools that had it, so the whole chain works on a genuine host/VM, not only inside Docker.

PITR, point-in-time targets, estimating how much data a restore actually loses, parallelism, incremental and other similar features are still to come - see Roadmap in docs/BACKUP.md. This release is deliberately narrow: restore to the latest good backup, one standalone instance, nothing cleverer yet.

---

✅ Restore from backup, for real

• New restore_from_backup tool (agents/sysadmin):
  Restore-to-latest only using pgBackRest's own default recovery target, no --type/--target flag.
  Refuses outright if pg_isready reports the instance still accepting connections. Confirmed live, this is the one thing standing between restoring a dead instance and destroying a live one.

• New fault db-pgdata-corrupted:
  A 4-hop chain (pbs_db_data_loss_triage → pbs_sysadmin_host_inspect → pbs_pgbackrest_health_triage → pbs_pgbackrest_restore_remediate) spanning three agents, each front-loading every condition it can determine on its own into its own FINDINGS rather than bouncing back for a decision already made.

• A second, code-level backstop that doesn't depend on the model getting it right:
  The pgbackrest_backup_unhealthy objective-evidence signal gates the destructive step and, unlike every other gate, it cannot be bypassed by approval_mode=force.
  If the model's prose says "healthy", but the tool's own structured result disagrees, the gateway forces a pending_gate regardless of caller intent.

• Classified policy.ActionDestructive, same tier as restart_container/restart_service, flowing through the existing Policy Engine with no special-casing.

• Live-verified e2e:
  Found six real bugs to get there, the last one the actual root cause: get_host_logs was silently discarding stderr on success and PostgreSQL's own fatal startup message goes to stderr.
  No more acceidental "successful" runs by guessing from the weaker circumstantial evidence. Model now comes back with the actual signal or is not allowed to come back at all.

✅ Hardened the evidence pipeline (that the force-gate depends on)

None of the above is trustworthy if the thing deciding "healthy" vs. "not" can be fooled by clever wording. This release closes four (four!) real gaps in that layer:

• splitEvidenceQuoteParts now splits on ;/—, not just ":
  Quoted evidence using either separator was silently dropped from consideration a few times during live testing. Not anymore.

• Multi-hop remediation was misclassified as diagnosis-only (and vice versa) in vault incidents and the Gateway's own narrative builder:
  Both used to classify a hop by which signal reached it (TRANSITION_TO vs ESCALATE_TO).
  In a complex, 4-hop triage, this turned out to be misleading, so it's now classified by the hop's own playbook_type instead.

• get_pgbackrest_status was auditing the raw CLI output blob instead of its structured result:
  Fixed so Layer 4 evidence-provenance checks see what the model actually saw.

• A confirmation probe pgbackrest_backup_unhealthy checked only whether a stanza name appeared anywhere in the model's response. As proven during live testing, this was insufficient when two runs with the identical backup_stale=true condition diverged on the gate purely by coincidental wording.
  Switched to evidence_quote_contains_value, which checks the actual evidence field.

• A stale-leaf-dispatch bug:
  A force-mode auto-chain gate firing 2+ hops deep re-dispatched an already-completed hop instead of the real pending leaf, so that the actual remediation playbook silently never ran.
  Fixed with a prior-run-ID walk to the true leaf.

• vault incidents now shows gate_reason directly, instead of leaving "why did this gate fire" to be inferred.

• The waitForChildRunComplete gate_pending bug: a force-mode auto-chain treated any outcome besides ""/"transitioned" as complete, so a second objective-evidence gate firing deeper in the chain (e.g., the pgBackRest health-triage hop's own gate on backup_stale=true) silently stalled. The remediation playbook was never dispatched and the 2-minute recovery-check loop spent its whole budget polling a database nothing had tried to fix. This is distinct from the stale-leaf- dispatch bug already listed (that one was about which hop gets re-dispatched, as this one is about recognizing a second gate as non-terminal at all).

• The Milliseconds duration bug: ToolExecution.Duration / Outcome.Duration were plain time.Duration tagged duration_ms. Go's default JSON encoding is raw nanoseconds, so a real ~5.77s tool call produced duration_ms=5774637750, overflowing the Postgres int4 column storing it. Fixed with a proper Milliseconds type, which is now covered by both a unit test and a live-Postgres integration test.

✅ Real host/VM (systemd) dispatch: generalized SSH beyond pgBackRest

v0.30 gave the pgBackRest-specific tools real SSH dispatch.
Everything else in SysAdmin's toolkit, namely check_host, get_host_logs, restart_container, restart_service haven't been retrofied yet to support it, no other fault in the catalog exercised this path. Not anymore!

• All four tools now dispatch through the shared runOnHost (real SSH when configured, local exec otherwise):
  Zero behavior change for every existing Docker/K8s test.

• New per-target pg_log_dir config field:
  Debian/Ubuntu and RHEL/Red Hat package PostgreSQL into different default log directories, neither matching the Docker fixture's own path.

• pbs_sysadmin_host_inspect and pbs_pgbackrest_restore_remediate get an explicit runtime=systemd branch alongside the existing Docker/K8s ones.

• New fault db-pgdata-corrupted-vm:
  Reuses the identical diagnosis/remediation chain over real SSH instead of docker exec
  Also a new testing/docker/setup-pgbackrest-vm.sh script to provision/setup a real Ubuntu or RHEL VM for it.

• Current status:
  Live-verified e2e against a real Linux VM with the full 4-hop chain correctly diagnoses a missing pg_control, confirms backup health over real SSH and actually restores and restarts the instance, independently confirmed (not just the agent's own self-report). Getting there surfaced several bugs along the way, all fixed: restart_service / get_host_logs dispatching without sudo (rejected by polkit, not a real service failure), SSH host-key algorithm negotiation picking a different key type than the one pinned in known_hosts and a port-only server-resolution fallback that could silently dispatch to the wrong real server when two targets shared a port, which now requires host+port together and refuses outright on true ambiguity rather than guessing.

✅ New incident TL;DR; feature

As the failure scenarious get more and more complex, the triage and remediation complexity also increases and so a high level summary helps quickly grasp what went on. Enter the new vault incidents <run-id> --summary feature, which offers a one line per hop (playbook, outcome, truncated findings) plus a final score, instead of the full deep-dive view. In our testing this is very useful for exactly the kind of multi-hop chain this release is about.

🐛 Bug fixes

• A text-alignment bug in faulttest's gate preview:
  A multi-line YAML description printed via a single unindented %s, misaligning every line after the first.
  Fixed with proper word-wrapping.

• Dead code found auditing the SSH-dispatch fix above:
  checkDiskImpl/checkMemoryImpl carried a fallback for an error string execInProcess can no longer return, with a comment that had become factually false.
  Removed and closed the resulting test-coverage gap with two new tests proving check_disk/check_memory dispatch correctly over real SSH too. This is a capability that the fix delivered effectively for free.

• execSSH (faulttest's own fault-injection path) had no -p <port> support:
  Always assuming port 22. Closed now since a real VM's sshd may not be reachable on the default port.

• The pbs_pgbackrest_restore_remediate hallucinated-escalation-target bug:
  The playbook's own guidance never named a target for the backup_stale=true precondition case, so the model invented a nonexistent one (pbs_pgbackrest_backup_stale_action) and the hallucination-guard that's supposed to catch this gracefully (isAllowedNextPlaybook) failed open because this playbook declared no allow-list. As expected, the bogus ID sailed through to a hard 404 instead of a clean human-escalation. Fixed on both ends: explicit precondition handling in the playbook and a declared transitions_to allow-list so future hallucinations are caught gracefully.

📚 Documentation

• docs/BACKUP.md:
  New §3 shoulld read as live-verified, with the one real residual gap (WAL-chain completeness) pointed at the Roadmap.

• docs/SYSADMIN_AGENT.md:
  Corrected to no longer claim SSH dispatch is pgBackRest-tool-pair-only.
  Section §5 rewritten to separate binary-selection (containerRuntimeBin) from dispatch-location (runOnHost).

• docs/FAULTTEST.md/testing/README.md:
  Clarified integration-test location and coverage for the new data-loss faults.

• docs/VAULT.md:
  new vault incidents <run-id> --summary subsection.

• docs/MUTATION_TOOLS.md §5.6
  New subsection on the declared-vs-resolved-target distinction (checkTargetScope can't see what resolveHost actually resolved to internally).

List of Changes

[boris@ ~/helpdesk]$ git log release/v0.30.0..release/v0.31.0 --oneline
5c93931 (HEAD -> release/v0.31.0, origin/release/v0.31.0) doc: update the backup doc with the explicit precondition check (stop, cite the WAL-coverage caveat, escalate to a human).
5065f6d feat: enhance `vault incidents <run_id> --summary` to also show run_id (plr_...).
bb4e783 fix: add instructions to escalate to a human (for now) in the pgbackrest restore PB.
e18d78b doc: update the doc for `vault incidents` with the --summary addition.
e5ef031 chore: close the integration test gap for the millisec change.
a22dee5 chore: add test coverage for the new Millsec type.
4601b83 feat: new vault incidents --summary option.
78285d0 fix: this is a major remediation bug for force approval mode (with the presence of objective evidence).
51cd2d6 fix: two bugs found in live testing on ubuntu: missing sudo on systemctl restart and journalctl.
a6fc8c1 doc: clarify the distinction with checkTargetScope (compares conn string against intended target).
3ce292c fix: model mismatching the targets.
dd98ba7 chore: introduce pgbackrest-vm.test as a stable name referenced in the infra.json file.
d6def24 fix: duration_ms data was wrong, with nanosecond value survives intact, just mislabeled.
f69ab56 feat: pbs_sysadmin_docker_inspect PB inconsistencies (not just cosmetic, but confusing the model too).
f587dab fix: tighten the db prompt + seed.go + bump up the data loss triage PB.
f47e4b0 fix: fw issues + listen_addresses + pg_hba.conf.
a628c47 doc: update the backup and mutation tools docs to include the third, by-design PB protection.
f4d1912 doc: add the missing desc of set_archive_command, get_pgbackrest_status and run_pgbackrest_backup to the mutation tools doc.
6bbe581 fix: minor isssue with PG_UNIT in the catalog.
3ba398b fix: collapse HELPDESK_SSH_KEY_HOST and HELPDESK_SSH_KEY_CONTAINER_PATH into one env var for now.
f29c509 fix: sysadmin may run as a container (via docker compose) or via a direct process on a host (via go run) - resolve via env vars duality.
9ea7590 chore: update docker compose to add bind mount for ssh key + add infra entries.
8a1d8de fix: two issues with the pgbackrest setup shell script: ssh.service on ubuntu + sudo match.
67381ab fix: teardown shell script bug (sudo) + infra tag missing.
2c5399b fix: remove dead code in sysadmin agent + add tests + update the docs.
5e082a3 feat: automated host/VM (systemd) target for db-pgdata-corrupted.
07adb86 fix: minor cosmetic alignment issue in remediation plan + a unit test.
a220f21 fix: tighten vault incidents (gate appearance and reason) and swap confirmation probe.
21c9862 fix: new failure mode in multi-chain escalation - wrong dispatch.
067c0f8 fix: still tripping the unverified gate, add a new escape condition in the parser.
4ad5d28 fix: issue of incomplete captured audit (result vs. original of result.Output).
026dd98 fix: objective evidence parser fix + vault incident multi-hop fix.
a9e3714 fix: at least some of postgres fatal errors end up in stderr, not stdout.
a941b89 fix: bump up the request window from 50 to 300 lines in pbs_sysadmin_docker_inspect PB.
53cfdbe fix: still not reaching the remediation (although cross-domain hop works fine) - buffering issue.
858da71 fix: bump up the timeout on db-pgdata-corrupted fault in the catalog from 90s to 300s (because of 4 hops).
c3967e0 fix: sysadmin's docker inspect PB's guidance was weak.
69a6df2 fix: failure injection (fast) error for db-pgdata-corrupted fault.
febfaa1 doc: add clarification with respect to location of the integration tests in testing/readme.
a8bf195 chore: add integration and e2e test coverage for the data_loss faults.
104934f chore: update the actual pgBackRest error message + document the restore scenario.
4e8550e feat: expand coverage + confirm full chain (4 PBs, one cross domain escalation) + 2 bugs.
ecca3c9 chore: similar to the previous commit, but this time aligning database-config-recovery.
94683ab chore: align the naming convention for the backup triage/remed PBs.
72548a1 feat: add new restore from backup logic: new tool, new remed PB, update triage PB.
16a0648 feat: enhance make recertify to support recertify_judge* vars.
12d3431 (origin/main, origin/HEAD, main) doc: further update the contributing doc based on a feedback.
d1dbf23 doc: update the samples readme to add a link to the 044 blog post.
cf72846 doc: add a link to the blog post to samples 022 transcript.
a08f06d doc: update the audit + contrib docs.
1d320a4 (tag: v0.30.0) Merge pull request #31 from borisdali/release/v0.30.0

Full Changelog: v0.30.0...v0.31.0