Repository navigation
🚀 aiHelpDesk v0.35.0: Budgeted Recovery
RPO is what you can lose. RTO is how long you can wait. The SLO is how much downtime you can afford in total
Until now those were three numbers in three different places and a restore only knew about the first two. v0.35 adds them up.
There are three headlines in this release:
1. A budget and a ledger: your SLO, in minutes, with receipts
- Declare an
availability_slo(and, if you like, anavailability_window, 30 days by default) and aiHelpDesk turns it into a budget: 99.9% over 30 days is 43m 12s of downtime. It tells you how many worst-case recoveries fit in it and says so out loud when yourrtois longer than the whole budget. - Every recovery incident is charged to a ledger: what each outage cost and what is left to spend. (and it's smart enough to keep apart failure injection tests and the outages that never confirmed a recovery). The budget is the allowance, the ledger is the receipt.
- This is what it looks like after three real restores on a database whose SLO allows 4m 19s:
Availability budget: pgbackrest-k8s
Budget availability 99.99% over 30d allows 4m19s of downtime per window; rto 30m is longer than the whole budget: one recovery that takes the full rto breaks the SLO
Spent 6m 48s in 3 outage incident(s), 158% of the budget
Left -2m 29s ⚠ OVERSPENT: the budget is used up
2. The gate now asks: can you afford this restore?
- Before anyone approves a PITR, the plan sets it against all three numbers: the history it would throw away against the
rpo, how long this server's restores really take against thertoand the downtime it would cost against what is left of the budget ("0s of the 4m19s budget already spent, only 4m19s left and this restore may take up to the rto of 30m: it may use more than is left"). - A new configuration setting,
availability_enforcement(warn,acknowledge,refuse), allows you to decide what happens when the budget cannot absorb it. Withacknowledge, a restore needs a second, "explicit yes" and the restore tool itself refuses without it, before it stops anything. We verified that on Docker, a VM and K8s: the instance kept serving and the same plan went through with the acknowledgement. - The recovery-time figure is no longer a guess: it comes from this server's own past real restores (nothing below three samples, median and longest from three, a p90 from ten).
3. Exact loss and a backup posture that knows your objectives.
- The plan no longer says only how much time a restore discards. It counts the transactions committed after the target, by decoding the WAL with
pg_waldump. When it cannot count, it says why and gives a lower bound. It never reports "nothing lost" for a count it could not make. get_pgbackrest_statusnow reads your declaredrpoandrtoagainst what is really there and a new R/O playbook,pbs_pgbackrest_objective_exposure_triage, files the answer on a schedule you set.
Still deliberately narrow: this applies only to restores for now, not HA/DR yet (that is next and it will be judged against the same budget). The ledger counts incident time from detection to recovery, which is longer than the outage itself and the notes below say by how much.
Here are the details:
✅ Expanded code coverage
This is an important production readiness item. Starting with aiHelpDesk Beta release in April we embarked on quest of relentless improvement of our code coverage and over the course of ~20+ releases we've been gaining a few percentage points, going from from 56% in April to 72% in v0.34. In this release that number didn't improve much (it's now 73%), but we removed the exclusions, particularly the cmd/ subdir exclusion, from the codecov calculations and through refactoring, expanded our coverage to effectively reflect the entire aiHelpDesk codebase.
✅ The budget
- New in
infrastructure.jsonobjectives:availability_slo(a percentage),availability_window(a duration) andavailability_enforcement. They resolve database, then environment, then fleet default, like the other objectives and carry the level they came from. budget = window x (100 - availability_slo) / 100. The check shows the budget, the number of worst-case recoveries it absorbs (budget / rto) and flags anrtothat is longer than the whole budget.- With no
availability_slo, there is no budget and nothing changes. - A contradiction (an
rtolonger than the whole budget) is logged at startup at error level by every service that reads the file (but blocks nothing). - A mistake in an objective still never costs you the inventory: it is logged and the rest of the file loads.
✅ The ledger
- Incidents now know which database server they are about (
target_key). Before this release an incident did not carry one, so nothing could be charged. - Every shipped playbook series (46 of them, pinned by a test) is classified by availability impact:
outage,degradedornone. An incident is classified by the worst playbook among all the runs it spanned. - Only real outage incidents are charged. A
faulttestinjection never counts and neither do degraded or harmless incidents, nor ones whose playbooks declare nothing: they are listed beside the charge and cost nothing. - An outage that ended
needs_human,escalatedorabandonedor that has been open for more than a day, has an unknown real downtime. It is listed as unconfirmed, not charged and the remaining budget is shown both ways. GET /api/v1/availability-ledger?database=<key>andfaulttest vault budget <database>print it. If an incident could not be tied to any database, the ledger says how many, so it is not read as the whole picture.- The duration is detection to recovery: it includes the triage, the model's reasoning and the wait for an approval. It is not the downtime (see Known limitations).
✅ The budget at the gate
helpdesk-client --pitr-planprints a BUDGET CHECK beside the RPO CHECK: the budget, what is spent, what is left and the verdict.availability_enforcementdefaults towarn. Withacknowledge, a restore that exceeds or may exceed the budget needs--acknowledge-objective-breach, the same flag as an RPO breach (one acknowledgement covers every breach and the record says which there were). Withrefuse, a restore of a serving instance that the spent budget cannot absorb is refused. A down instance is asked, not refused, because refusing would leave it down and spend the budget faster.- The restore tool enforces it, not the model and not the prompt. A refusal happens before anything is stopped or changed.
- A gate can be approved once. If the approval lacks a required acknowledgement the restore refuses, the gate is used up and the way on is a new triage run (the gateway answers a second approval with HTTP 409). The refusal says so.
- The budget, the duration it assumed and where that came from (
declared_rto,history_longestorhistory_p90) are recorded in therecovery_decisionevents.
✅ A recovery-time estimate from real restores
- From this server's resolved, real restores in the last 90 days: below three samples the status is
no_historyorinsufficient_historyand the declaredrtostands alone. From three, the median and the longest, planned for as the ** longest**. From ten, a p90, planned for as the p90 (below ten a "p90" would be the longest under another name, so it is not offered). How do you get that many restores - that's v0.36 😉. - It is set against the declared
rtoas a count of past restores that broke it and it replaces thertoas the restore's duration in the budget check. - The size of the cluster each restore wrote back is now recorded on the decision, so restore times can later be compared across a server's growth.
- Live: on a same size smallest database, after three restores the longest was 1m 41s on the Docker fixture, 36s on a systemd VM and 55s on a K8s Pod.
✅ Exact loss: the transactions a restore would discard
plan_point_in_time_recoverydecodes the WAL from where recovery would stop (Postgres own rule) withpg_waldumpand counts the commits after it. Exact for a serving instance read to its own position. "At least N" when it is capped or the instance is down and the reason when it cannot be determined.- It runs on the database host: through
docker exec, over SSH or throughkubectl exec. It needspg_waldumpthere (found by the server's major version). It can be turned off withHELPDESK_PITR_COUNT_TRANSACTIONS. - Verified against real WAL on a Docker container, an Ubuntu VM over SSH and a K8s Pod. The parser was tested against PostgreSQL 16.14, 16.15, 17.7 and 18.6.
- It provides additional information to an approver, sitting next to the RPO check (and it never refuses anything by itself).
✅ Backup posture against your objectives
get_pgbackrest_statusgains anobjective_exposureblock: the age of the newest archived WAL against therpo(an upper bound, since an idle server writes no WAL) and the longest past restore against therto.backup_staleis unchanged.- New playbook
pbs_pgbackrest_objective_exposure_triage: R/O, no transitions, five endings (within, not assessed, the history breaks therto, the archive is older than therpo, the status could not be read). Only the quiet endings are recordedresolved. One run records one incident, so schedule it daily, not every few minutes.
✅ faulttest and tooling
- New faults
db-pitr-dropped-table-vmanddb-pitr-dropped-table-vm-lsnrun the whole point-in-time chain against a systemd VM and the recovery-trail check is wired into thefaulttestCLI. scripts/demo-restore-cycle.shdrives one complete restore cycle the way an operator would (so the incident counts), against a Docker fixture, a VM over SSH or a K8s pod. It is a repository script, not part of the deploy bundle.
🐛 Bug fixes
- A plan made seconds after a WAL switch read the new, still empty WAL segment and lost its transaction count. The empty segment is no longer read (found on a fresh fixture).
- The evidence check only looked at a tool's results, not its errors, so a model quoting a restore refusal word for word was flagged as unverified evidence. It now checks both.
- A refusal told the operator to "approve the gate again", which the gateway rejects. It now says to start the triage again.
- The ledger's note about incidents that could not be tied to a database counted old fault-injection incidents too. It counts real ones only.
- PITR's attribution: the classifier prompt and the playbook's root-cause classes now carry definitions (a VM run was filed under the wrong class). The same for the objective-exposure playbook.
faulttestagainst a real VM: a parsing weakness in the verifier, the insert position of a fault target and a record-start edge case.- A unit test depended on
HELPDESK_INFRA_CONFIGbeing unset in the shell.
⚠️ Known limitations
- Two clocks The recovery-time estimate measures the remediation hop after the approval (restore, restart, verify). The ledger charges the whole incident, from the start of the triage, so it also holds the model's reasoning and any wait at the gate. In our runs the ledger charged from about 1.2 times the estimate (Docker) to about 2.6 times (the VM), on databases small enough that the restore itself took under two minutes. The gate compares one with the other, so it is optimistic by about the difference. The wait for an approval does not shrink with the size of the data. Neither figure is the downtime.
- The estimate is only as good as its history: three restores is a small sample and restores of an old, smaller server make it optimistic.
- The
refuseenforcement mode has not been run live. The tests cover it. - The audit trail is tamper-evident, not tamper-proof against someone who controls the audit database: the chain is not signed or anchored outside it and the event-level principal is not in the hash. The recovery decision itself, including who approved, is.
- With
staticidentity on K8s, the auditor and secbot poll auditd without credentials and get 401: they cannot read the event feed (the agents and the gateway are unaffected). This predates the release and is not fixed yet. - A model still makes small mistakes around the restore (a wrong parameter name, a loose quote). The decisions that matter are in tools that refuse regardless.
📚 Documentation
- docs/OBJECTIVES.md: the availability budget (§5.2), the ledger (§5.3), the budget at the gate (§5.4), the recovery-time estimate (§5.5), the backup-posture check (§5.6) and the "Two clocks" note.
- docs/PLATFORMS.md: every live run above, by platform.
- docs/PITR.md: the budget check and that a gate can be approved once.
- docs/VS_YOUR_SCRIPTS.md (new): seven questions to ask of your own restore automation and where aiHelpDesk is not better. docs/FOR_WHOM.md now says what aiHelpDesk is and is not (it does not manage your databases and instead, it is the incident-time decision and accountability layer on top of the tooling you run).
- New samples: SAMPLE028 (PITR on a VM), SAMPLE029 (the transaction count: canary runs and three platforms), SAMPLE030 (the budget, the ledger and the estimate).
List of Changes
[boris@ ~/helpdesk]$ git log release/v0.34.0..release/v0.35.0 --oneline
71d9a48 (origin/release/v0.35.0, release/v0.35.0) fix: three minor linter findings.
3315e57 doc: finish the last round of k8s testing + add new vs your scripts doc.
5dbc399 fix: ledger's unattributed count now asks auditd for origin=real only.
d347c0b chore: adjust the restore script to accomodate K8s too.
06cb627 chore: include adjustments for the docker findings run + vm run (and update the docs.
a4619dd chore: update the demo restore cycle shell script to be vm/ssh compliant too (not just docker).
32698f9 fix: tier b issues (seems like a race condition).
00a6815 fix: second approval doesn't work (leads to 409), update the wording in a script + docs.
0e031ce doc: update the objectives and the platforms docs based on the last round of testing in docker.
8af71ea feat: add a demo script to exercsize the restore/v0.35 steps + fix an objective exposure PB.
d040c58 chore: update the agenutil unit test to not depend on the shell's env var + update the docs based on the docker live tests.
c882e42 doc: cosmetic change in sample 030.
8106708 doc: update sample 029 and add sample 030.
58213e7 chore: more work on objective exposure PB + sample 029 with the two pg_waldump canary runs.
fe6faf4 fix: continue adjusting the objective exposure PB (against VM).
e2c1af0 fix: further adjusting the objective exposure triage PB.
2d3ac22 chore: live testing of pgbackrest-objective-exposure-triage PB against a vm: not reachable ending.
74193b6 feat: add the new pgbackrest-objective-exposure-triage PB.
26c9857 doc: update objectives and platforms docs based on the vm test results.
323410b feat: acounting for postgres version diff in pg_waldump output.
3aee612 chore: add test coverage for exact loss via pg_waldump.
ba10af0 feat: exact data loss via pg_waldump is built e2e.
9b87af7 feat: continue building exact data loss via pg_waldump.
44f96fc feat: start building out the exact loss based on pg_waldump.
32039b4 chore: add test coverage for objective exposure.
1a4653d feat: restore size + objective exposure.
c190a4c feat: p90 now for 10+ samples and for 3-9 samples it's median/longest.
d0cafde chore: add test coverage for the recovery estimate vs. rto.
77eabdb feat: new recovery time estimate vs rto.
df45610 chore: add test coverage for the avail budget wiring.
f33f6b4 feat: this is the meat: wire avail budget to the restore, including ux (helpdesk-client and vault incidents).
ea27b4d chore: test coverage dropped after expanding codecov's scope - fixing it now.
3f08329 chore: add test coverage for availability ledger.
23be243 feat: build availability ledger e2e.
d756f98 chore: add test coverage for availability window.
ebd78bc feat: avail window + budget math.
f334e27 chore: edit .codecov.yml in the attempt to improve patch code coverage (often appears as blank).
a17a7ae feat: add new avail classification breakdown by PB series.
25519ed chore: add test coverage for the new lookup/resolver method.
24d18ae feat: every new incident now records which database it is about (DB info was missing before).
ad66753 doc: beef up sample 028 doc, add runs data + links.
b304f2a fix: the insert position fix (the previous attempt didn't take).
aa58025 fix: linter issue with De Morgan + ream record start in a vm fault target.
bc61062 fix: minor issue with the verifier change (discovered in the -lsn test on a vm).
7a8f40f fix: classifier prompt and the pitr PB classes now carry definitions (attribution was off in vm live tests).
2458fb9 fix: the faulttest cli issue with the remed suffix against a vm.
ce463fd fix: two new runs against a target VM showed parsing weakness.
a9cb112 feat: trail check is now wired into the faulttest CLI with faultlib.CheckRecoveryTrail holding all of the logic.
c4b1de7 feat: two new VM faults: `db-pitr-dropped-table-vm` (time target) and `db-pitr-dropped-table-vm-lsn` (LSN target).
7edba9f doc: add a reading path doc to sample transcripts.
ac75c26 (tag: v0.34.0) Merge pull request #36 from borisdali/release/v0.34.0
Full Changelog: v0.34.0...v0.35.0