v0.34.0

🚀 aiHelpDesk v0.34.0: Quantifiable Recovery (aka how much data to lose in PITR and is it still within RPO?)

There are a number of major headlines in this release revolving one theme: recovery now has a number you can hold it to. Release v0.33 taught aiHelpDesk to restore to a moment you choose. v0.34 is where it has to say what that costs and what you allowed it to cost:

👉 Projected data loss is now measured.

👉 There's now a way to formally state your objectives: RPO, RTO and availability SLO.

👉 You declare how much data loss (RPO) and how much recovery time (RTO) each database may have, once for the fleet, per environment or per database.

👉 Before a PITR, aiHelpDesk now measures how much history it would discard and compares it with the RPO and the gate you approve shows the comparison.

👉 If the restore would exceed the objective, a configured policy decides: just say so (warn), require an explicit second acknowledgement (acknowledge) or outright refuse (refuse).

👉 Every recovery decision is now on the record: a new typed audit event, recovery_decision, records the four stages of a recovery: the plan, the gate, the execution and the outcome, with the objectives in force, every option considered, who approved and what they acknowledged and how long the recovery took. It sits in the tamper-evident chain and shows up in the incident narrative, the journey and the CLI.

Still deliberately narrow: the objective is enforced on time targets and on LSN targets only when it can be proven (the archive's own timestamps give a lower bound). RTO is shown against what actually happened, but is not yet predicted. Reading the WAL for the exact loss and the commit count, an RTO estimate from real incidents, a standing RPO-exposure alert, recovery preference and availability budgets are on the Roadmap

---

Here are the highlights:

✅ Objectives: say how much recovery may cost

• New in infra.json: environments, defaults and a per-database objectives block with rpo, rto, availability_slo, recovery_preference and enforcement (warn, acknowledge, refuse).
• Each value resolves database, then environment, then fleet default and carries the level it came from (rpo 1s [database], rto 30m [environment:prod]), so nobody has to guess which line won.
• A mistake in an objective never costs you the inventory: it is logged and the rest of the file loads. A misspelled environment, an unreadable rpo or an invalid enforcement fall toward acknowledge, never silently toward warn.
• Durations are strings (15m, 4h, 1d) and rpo: "0s" is a value (for no loss tolerated). rpo: 900 is reported as a config error with how to fix it.
• availability_slo and recovery_preference are read and validated now.
• Helm: the chart already passes infrastructure through as JSON and new tests prove objectives and environments render into the file the agents read.

doc link

✅ How much would this restore discard?

• plan_point_in_time_recovery now returns a data_loss_estimate: for a time target, the span of history that would be discarded; for a running instance, measured from the instance's own clock, so skew between hosts does not matter.
• It grows while the instance runs, so the restore measures it again just before the stop and decides on that figure. In a live run the plan said 9s and the decision was made on 1m 05s.
• The estimate says what it is: a span of history, not a row count ("Transactions are not counted").
• It is compared with the RPO and the verdict is one of within, exceeds, may_exceed, not_comparable, no_objective or objective_invalid.
• LSN targets have no timestamp. When the archive segment holding the target is old enough, its repository time is a rigorous lower bound (with a 60 second clock-skew margin on a running instance), which is enough to prove a breach. Otherwise the verdict is not_comparable and says why, with the bytes of WAL that would be discarded. It does not claim a number it cannot back.
• pbs_pgbackrest_pitr_triage carries the estimate and the RPO check into the report, and helpdesk-client --pitr-plan prints an RPO CHECK.

doc link

✅ Enforcement, at the gate

• warn: the comparison is shown, nothing more.
• acknowledge: an exceeded objective needs a separate, explicit acknowledgement in the gate approval (acknowledge_objective_breach, --acknowledge-objective-breach in the CLI), on top of acknowledging the data loss and the stop.
• refuse: the restore is refused, even with the acknowledgement.
• may_exceed and objective_invalid ask for the acknowledgement but never refuse, because a configuration problem should not trap an operator in an outage.
• A refusal happens before anything is stopped and the instance keeps serving.
• The model is not allowed to grant the acknowledgement: the restore tool's model-facing arguments have no field for it and it comes only from the human's approval at the Gateway.
• Verified against a real restore: a refusal leaves the instance serving with its data, an acknowledgement lets the same plan through.

✅ Every recovery decision is on the record

• One recovery_decision event per stage:

  • plan: what was proposed, with the objectives in force, a digest of the config they came from and each option's loss and verdict
  • gate: who approved and exactly what they acknowledged (data loss, stopping the instance, exceeding the objective)
  • execution: what the restore decided just before it stopped anything, on the re-measured loss
  • outcome: the recovery duration, shown against the RTO.

• The "recovery duration" is the remediation hop after the approval (restore, restart, verify). It is not the downtime, which began earlier and includes the wait for the approval and the narrative says so.

• The events are covered by the audit hash, so editing one breaks /v1/verify. A coverage review found that the hash had covered only the core fields of any event, which is now fixed and the new payload is now in it.

• Rendered where people look: a RECOVERY DECISION section in vault incidents, a Recovery: line in the journey and the narrative API.

• A recovery is never held for its record: the database may be down and bringing it back comes first. If the write fails, the result carries decision_unrecorded and an error is logged. A spool, a fail-closed rule and a break-glass override are on the Roadmap.

• Connection strings are redacted when the event is built, for every emitter. A review found a password in a gate event's target and a ''-quoted one left partly visible. Both are fixed and tested.

doc link

✅ faulttest proves it, on real infrastructure

• make faulttest-pitr now asserts the trail after the restore: all four stages, the objectives named, an acknowledged breach on record, no password in the clear and a chain that still verifies. It requires FAULTTEST_AUDIT_URL.
• New fault db-pitr-dropped-table-lsn (an LSN target) and make pitr-objectives-infra, which writes an infra copy with a tight objective.
• Live runs: the time target exceeded a 1s RPO, the gate sent the acknowledgement and the restore proceeded on it (loss 1m 05s, recovery duration 1m 29s against a 30m RTO). The proof was that the LSN target came out as not_comparable and proceeded without one, as designed.
• New integration tests: the decision event through a real Postgres-backed auditd (payload, journey enrichment, chain, tamper), real restores under refuse and acknowledge (instance untouched on refusal, data correct after the acknowledged restore) and a Helm render test for objectives.
• After a gate resolves, the harness now verifies that a child remediation run really started whenever it can reach auditd.
• make faulttest-* targets each write their own log, so one run no longer overwrites another's (FAULTTEST_LOG still overrides).

doc link

🐛 Bug fixes

• Strict loading would have dropped every database from infra.json over a typo in one objective: now lenient, logged.
• An undeclared environment silently weakened enforcement to warn: now falls toward acknowledge.
• The objective lookup used a different matcher than the restore's own host resolution (container name, ambiguity, a race): both now share lookupServer.
• The direct tool path for plan_point_in_time_recovery bypassed the data-loss estimate: it now takes the same path as the playbook.
• faulttest-pitr: the estimate was missing when the plan was feasible and the playbook narrative still printed the old data-loss sentence.
• Passwords in a recovery event's target and reason are masked, including ''-quoted and escaped forms.
• The audit hash covered only an event's core fields, typed payloads are now covered where it matters and the limit is documented.

⚠️ Upgrade notes

• Restart auditd, the gateway and the sysadmin agent together: the new event type is written by the agent and the gateway, and its hash input has a new optional field (old events verify unchanged).
• Every service that writes to auditd needs its audit credential when the identity provider is static; a missing key means plan and execution decisions are dropped (and logged as errors).
• make faulttest-pitr requires FAULTTEST_AUDIT_URL.
• New optional gate field: acknowledge_objective_breach.
• make e2e runs docker compose down -v on the deploy stack and deletes its audit history, so do not run it against a stack you want to keep.

📚 Documentation

• docs/OBJECTIVES.md (new): the objective model, resolution, enforcement, what the estimate can and cannot prove, with flow charts and a table of contents.
• docs/PITR.md: flow charts, "LSN targets and the lower bound", "What is recorded".
• docs/AUDIT.md: §3 hash coverage corrected, §4.6 recovery_decision fields.
• docs/JOURNEYS.md, INCIDENTS.md, VAULT.md: where the decision shows up. docs/MUTATION_TOOLS.md and docs/FAULTTEST.md §6.6 updated.
• make docs-links checks every internal link and runs in CI; 37 broken links fixed. A tree-wide gofmt pass, with make fmt-check to keep it clean.

List of Changes:

[boris@ ~/helpdesk]$ git log release/v0.33.0..release/v0.34.0 --oneline
5e508e7 (origin/release/v0.34.0, release/v0.34.0) fix: lintner found four minior issues.
97d4045 chore: add fmt job to .github/workflows/ci.yml that runs make fmt-check.
8bc9fb1 chore: enhance the make file to dedicate a separate log file for every target to avoid collisions.
71b961e doc: update the faulttest doc to describe the child-run check instead of tool evidence.
f624699 chore: add more test coverage: trail assertion + new LSN fault and objectives overlay.
59dea14 chore: add overall test (beyond unit tests) coverage for v0.34 release.
b63ba4a chore: add test coverage for the lower bound, recover decision.
5eb33e3 feat: segment-time lower bound + outcome stage + unrecorded flag.
351bf72 chore: add test coverage for the new recovery_decision, vault incident/journey narrative + bug fix in password exposure.
8b7271d feat: add new `recovery_decision` event - built e2e.
efc6549 doc: add two flow charts for approval + order of ops.
b0f90fb chore: add test coverage for enforcement.
9a983ba feat: an unreadable objective now needs the acknowledgement, like a breach.
f3ea847 feat: objective enforcement built e2e.
ec72216 chore: objective with misspelled env may lead to warn + add tests.
ddc418b feat: add an objectives configuration slice.
dc18f23 feat: technically this is just an objective doc update, but it states direction.
5f23610 feat: triage PB now carries the data-loss estimate and the RPO check.
868353e chore: add test coverage for checking data loss against rpo.
acd5c5c feat: pitr plan (and restore result) now set the `data-loss` estimate against the rpo.
59cf141 feat: redesign the objective loading to log/skip problematic config, not to refuse the whole infra.json.
75bec24 feat: add the base of objective model (rto, rpo, avail slo).
7ccb9b6 fix: fix the narrative based on the last faulttest-pitr run.
0bad002 fix: still issue in faulttest-pitr: the plan was feasible and the old `data_loss sentence` was there, but data_loss_estimate was null.
7719ca0 chore: continue with the faulttest-pitr fixes: PB.
d1a0cbf chore: fix issues in `make faulttest-pitr`.
275ae83 chore: add test coverage for data loss estimate.
8f879b5 feat: add the first cut of the data loss estimate feature.
9478772 feat: fix remaining issues, so that `make docs-links` passes now and runs as part of the CI.
2497239 chore: new `make docs-links` linter + add .env to .gitignore + minor fixes in deploy/docker-compose/README.md.
acd1ccc chore: go fmt cleanup.
6f971ec doc: add the pgbackrest make target to the sample 027 doc.
98ad17a doc: add sample 027 for the pitr post.
b0e4326 doc: add sample 026 doc.
ac71721 fix: increase timeout for faulttest CI tests.
d36fc27 doc: add toc to the deploy/*/README.md docs and fix the refs.
57645fa (tag: v0.33.0) Merge pull request #35 from borisdali/release/v0.33.0

Full Changelog: v0.33.0...v0.34.0