From c93d0eb2e1d9b23b396656d8064ff191d91b5fd7 Mon Sep 17 00:00:00 2001 From: xNet Test Date: Wed, 17 Jun 2026 10:06:40 -0700 Subject: [PATCH 1/6] =?UTF-8?q?docs(exploration):=20explore=20xNet=20Cloud?= =?UTF-8?q?=20operations=20=E2=80=94=20upgrades,=20backups,=20telemetry,?= =?UTF-8?q?=20SLAs?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 --- ...OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md | 598 ++++++++++++++++++ 1 file changed, 598 insertions(+) create mode 100644 docs/explorations/0193_[_]_XNET_CLOUD_OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md diff --git a/docs/explorations/0193_[_]_XNET_CLOUD_OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md b/docs/explorations/0193_[_]_XNET_CLOUD_OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md new file mode 100644 index 000000000..f5bfa9039 --- /dev/null +++ b/docs/explorations/0193_[_]_XNET_CLOUD_OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md @@ -0,0 +1,598 @@ +# xNet Cloud — Operating the Fleet: Upgrades, Backups, Telemetry, and SLAs + +## Problem Statement + +xNet Cloud provisions one isolated, end-to-end-encrypted hub per paying tenant +(exploration [0180](0180_[_]_XNET_CLOUD_ARCHITECTURE_AND_COMPLETION_STATUS.md); +the signup→provision→connect "face" landed in +[0192](0192_[_]_XNET_CLOUD_ONBOARDING_AND_UI_HOSTING.md)). Running that fleet for +real raises four entangled operational questions: + +1. **Upgrades** — how do we roll a new hub image to thousands of tenant hubs + without breaking them, and roll back fast when one regresses? +2. **Backups** — how is each tenant's data continuously protected, and how do we + *prove* we can actually restore it? +3. **Telemetry** — the hard one. The data is E2E-encrypted; **we hold bytes we + cannot read**. Yet we still need (a) operational signal to keep hubs healthy + and debug incidents, and (b) product signal — "how are people using this?" — + to know what to build. How do we get both *without* breaking the privacy + promise that is the entire point of xNet? +4. **Uptime / SLAs** — what do we promise, how do we measure it, and how do we + automate enough that tenants get high uptime with minimal human toil? + +The tension the user named directly: **secure & private vs. observable**. This +document argues that the tension is already largely resolved in the codebase — +xNet shipped a privacy-preserving telemetry stack (0187/0190) that respects the +encryption boundary by construction — and that the real work is *operationalizing* +it for the fleet and wiring it to upgrades, backups, and SLOs. + +## Executive Summary + +**The good news: the hard primitives exist and are tested.** Continuous backups +(Litestream → R2, drain-before-close, restore-on-boot), the cold-tier lifecycle +(`demoteIfCold`/`reactivate` with a sync gate), per-tenant immutable image tags, +the one-step `upgradeTenant`, the hub's `/health` + `/ready`, and a full +privacy-first telemetry pipeline (consent-gated, scrubbed, bucketed client events ++ a separate `telemetry.db` with hashed DIDs and hourly rollups) are all shipped. + +**What's missing is orchestration and measurement, not primitives:** + +- **Upgrades** are a single per-tenant step with no cohorting, no automatic + rollback, and no signal to gate on. There is no fleet-level rollout engine. +- **Backups** replicate continuously but there is **no automated restore drill** + (we have never proven a real tenant restores), no retention/PITR policy, and + no backup-freshness alert wired up. +- **Telemetry** is built for the *single hub / single user*, not the *fleet + operator*. The pieces (`telemetry-bridge` Prometheus→events, `/health`, + `/metrics`) exist but nothing aggregates them centrally into per-tenant SLIs. +- **SLAs** are **declared but not enforced**: `SlaLevel` is a field in the plan + catalog (`none`/`best-effort`/`99.9`/`custom`) that nothing measures, alerts + on, or honors. + +**The core design move** is a **three-plane telemetry model** aligned to xNet's +two-identity split, plus a **control-plane reconciliation loop** that turns +health signal into automated upgrades, self-healing, and SLO accounting: + +| Plane | What it is | Who it belongs to | Default | +|---|---|---|---| +| **Operational** | per-hub up/down, latency, error rate, backup freshness, wake latency, resource use — **no content, tenant-scoped** | xNet (we run the infra) | **on** for managed hubs | +| **Product** | feature-usage events — *which* surfaces get used | the user (it's their behavior) | **consent-gated**, scrubbed, bucketed; cross-tenant only via privacy-preserving aggregation | +| **Encrypted** | document content, exact identifiers, plaintext | the user, end-to-end | **never leaves the device unencrypted; we cannot read it** | + +**Recommendation:** operationalize the existing telemetry for the fleet (Plane 1 +by default, Plane 2 by consent), define SLIs/SLOs/error budgets per plan tier, +and build an **error-budget-gated canary→waves→auto-rollback upgrade engine** on +top of the existing one-step `upgradeTenant`. Make the control plane a +**reconciliation loop** (desired vs. actual per tenant) so uptime, backups, +upgrades, and self-healing are all automated convergence rather than scripts. + +## Current State In The Repository + +### Upgrades — one immutable step, no orchestration + +[`ControlPlane.upgradeTenant`](apps/cloud/src/control-plane.ts) rolls a single +tenant to a new image and records the version: + +``` +apps/cloud/src/control-plane.ts + async upgradeTenant(tenantId, targetVersion) { + const handle = await this.deps.provisioner.upgrade(record.substrateRef, targetVersion) + await this.deps.tenants.put({ ...record, targetVersion: handle.targetVersion }) + } +``` + +- The [`Provisioner`](packages/cloud/src/provisioner/types.ts) interface + (`provision`/`upgrade`/`setEnv`/`sleep`/`destroy`/`get`) is the swappable seam. + `MemoryProvisioner` works; **`CloudRunLitestreamProvisioner` and + `FargateLitestreamProvisioner` throw `NotImplementedError`** — nothing has ever + upgraded a real hub. +- **Image tags are immutable and pinned per tenant, never `:latest`** (the + comment in `provisioner/types.ts` is explicit); the default is + `HUB_IMAGE_TAG` (e.g. `xnet-hub@0.0.1`). This is what makes instant rollback + *possible* (just re-point to the previous tag) — but no code does it. +- The hub `Dockerfile` pins **Litestream v0.5.3** (v0.5.6/0.5.7 have a + silent-replication bug #1083). The hub image is content-built, not `:latest`. +- **Gap:** no cohorting, no canary, no health-gated promotion, no rollback. The + caller is expected to drive the sequence by hand. + +### Backups — continuous, gated, but never restore-tested + +The Litestream module (`packages/cloud/src/litestream/`) is complete: + +- [`config.ts`](packages/cloud/src/litestream/config.ts) — per-tenant replica at + R2 path `t//db`, `syncInterval: 1s` (≈1 s RPO on hard kill, + near-zero on graceful shutdown), secrets injected as env refs. +- [`controller.ts`](packages/cloud/src/litestream/controller.ts) — + `LitestreamController.drain(graceMs)` SIGTERMs Litestream and waits for it to + flush final WAL frames before the machine dies (drain-before-close). +- [`litestream-entrypoint.sh`](packages/hub/litestream-entrypoint.sh) — + restore-on-boot for **both** `hub.db` and `telemetry.db`, then + `litestream replicate -exec node`. +- [`freshness.ts`](packages/cloud/src/litestream/freshness.ts) — `isFullySynced` + (the demotion gate: never destroy a live DB until its last write is durable) + and `isReplicaFresh` (lag alert). +- The hub gates `PRAGMA wal_autocheckpoint=0` behind `LITESTREAM=1` + ([`storage/litestream.ts`](packages/hub/src/storage/litestream.ts)) so SQLite + doesn't race Litestream and lose frames. + +**Gaps:** no **retention / point-in-time-recovery policy** (R2 lifecycle is +external IaC that doesn't exist yet); **no automated restore-verification drill** +(0180's validation checklist flags that RPO/wake have never been measured against +a real deploy); the **single-writer fence** relies on the control plane plus a +Litestream S3 conditional-write lease that isn't exercised. + +### Cold tiering — the uptime/cost lever, implemented on the fake + +[`demoteIfCold`/`reactivate`/`markActive`](apps/cloud/src/control-plane.ts) +implement the 0178 model: an idle hot tenant is confirmed synced +(`assertSynced`), its machine destroyed, marked `cold`; on demand a fresh hub is +provisioned with `restoreFromR2`. Plan isolation tiers +([`packages/entitlements/src/plans.ts`](packages/entitlements/src/plans.ts)) set +the uptime posture: `dedicated-sleep` (personal/family, scale-to-zero) vs +`dedicated-warm` (team, always-on). Cost is modeled in +[`cost/pricing.ts`](packages/cloud/src/cost/pricing.ts) (warm ≈ $6/mo vs active +$0.00266/h; R2 $0.015/GB). + +### Health — exists on the hub, unused by the fleet + +The hub serves `GET /health` (uptime, room count, doc pool, connections, memory, +platform/region/machineId) and `GET /ready` (writes a readiness key) +([`packages/hub/src/server.ts`](packages/hub/src/server.ts)), and fingerprints +its substrate (`K_SERVICE`→Cloud Run, etc.) in +[`config.ts`](packages/hub/src/config.ts). **Nothing in the control plane polls +these or derives availability from them.** + +### Telemetry — a privacy-first pipeline already exists (0187 + 0190) + +This is the crux, and it is **already built to respect the encryption boundary**: + +- **Client** ([`@xnetjs/telemetry`](packages/telemetry/src/)): a 5-tier consent + model (`off`(default)/`local`/`crashes`/`anonymous`/`identified`, + [`consent/types.ts`](packages/telemetry/src/consent/types.ts)); a collector + that **scrubs** PII (UUIDs, DIDs, tokens, paths, emails, IPs — + [`collection/scrubbing.ts`](packages/telemetry/src/collection/scrubbing.ts)) + and **buckets** values (latency/count/size — + [`collection/bucketing.ts`](packages/telemetry/src/collection/bucketing.ts)); + OTel-aligned schemas (Crash/Usage/Performance/Security); an IndexedDB durable + buffer; an HTTP transport with `keepalive`. +- **Tracing** (0190, [`tracing/`](packages/telemetry/src/tracing/)): exact-timing + waterfalls stay in a **local ring buffer that never syncs**; only + `emitTraceAsBuckets` ([`tracing/egress.ts`](packages/telemetry/src/tracing/egress.ts)) + ships **bucketed** per-stage metrics with opaque trace/span IDs. +- **Hub store** (0187, [`packages/hub/src/telemetry/`](packages/hub/src/telemetry/store.ts)): + a **separate `telemetry.db`**; DIDs **hashed server-side** with a salt + ([`normalize.ts`](packages/hub/src/telemetry/normalize.ts)); `POST + /telemetry/ingest` (UCAN-auth, ≤500/batch); **hourly rollups maintained on + ingest**; admin-gated reads; lazy DuckDB `ATTACH` joins + ([`analytics.ts`](packages/hub/src/telemetry/analytics.ts)); Parquet cold tier + on R2 with 7-day raw retention ([`tiering.ts`](packages/hub/src/telemetry/tiering.ts)). +- **Ops bridge** ([`middleware/telemetry-bridge.ts`](packages/hub/src/middleware/telemetry-bridge.ts)): + reads Prometheus `/metrics` every 60 s and emits `producer:'hub'` events + (ws connections, sync docs, backup uploads, query duration, rate-limit + rejections). **This is the seam the fleet operator should consume — it's off by + default.** + +**What the hub can see today:** hashed DIDs, OS/version, error *kinds*, bucketed +latencies/counts, opaque trace IDs. **What it cannot see:** content, exact +identifiers, plaintext timings. The privacy posture is genuinely good. + +**Gap for the fleet:** every bit of this is scoped to *one hub serving its own +users*. There is no **central, cross-tenant operator view**: no place the +control plane aggregates per-tenant health into SLIs, no fleet dashboard, no +alerting, no status page. AI usage *is* metered centrally +([`cloud/src/ai/metering.ts`](packages/cloud/src/ai/metering.ts), idempotent +ledger + Stripe meter), which is the one existing example of central per-tenant +accounting to mirror. + +### SLAs — a field nobody enforces + +`SlaLevel` (`none`/`best-effort`/`99.9`/`custom`) is declared per plan in +[`plans.ts`](packages/entitlements/src/plans.ts) and rides inside the signed +`HUB_PLAN` token, but **no SLI is measured, no SLO is published, no error budget +is computed, and no alert or credit is triggered**. It is documentation, not +control. + +## External Research + +**SRE SLO/error-budget discipline is the right backbone.** Google's practice: +an **SLI** is a ratio of good events to valid events (availability = successful +requests / valid requests); an **SLO** is a target over a rolling window; the +**error budget = 100% − SLO**. The numbers matter for what we can promise: 99.9% +≈ **43 min/month** of allowed downtime, 99.99% ≈ **4 min/month**. The +**error-budget policy** is the automation hook: budget healthy (>50%) → ship +fast; low (<25%) → slow down, extra review; exhausted → **freeze non-reliability +deploys**. This directly answers "how do upgrades and SLAs relate" — the error +budget *gates the rollout*. + +**Privacy-preserving aggregate telemetry is a solved, deployed problem.** +**Prio / DAP (Distributed Aggregation Protocol)**, operated by ISRG's **Divvi +Up** with the **Janus** aggregator, splits each metric across **two +non-colluding servers** so that — as long as one server is honest — the +operators "learn nearly nothing" about any individual; only the *aggregate* is +revealed, and recent versions compose it with **differential privacy**. **Firefox +ships this in production** (Mozilla + Divvi Up as DAP provider, Fastly as the +OHTTP relay that strips client IPs). This is the gold standard for Plane 2 ("how +are people using the product?") **across tenants** without ever exposing one +tenant's behavior — and xNet's egress adapter is already a pluggable seam to feed +it. + +**Progressive delivery with automatic rollback is standard.** The canary pattern +(e.g. Argo Rollouts): shift a small cohort first (20% → 50% → 100%), watch +error-rate/latency against thresholds, and **auto-rollback** if they regress; +fleets are addressed by generating one rollout per cell/cohort. Per-tenant +immutable tags make xNet's rollback trivial (re-point to the prior tag). A +public **status page** fed by the same SLIs is the standard transparency layer. + +**Litestream is built for this backup story.** v0.5's LTX format + hierarchical +compaction give point-in-time recovery and fast restores (≈1–3 s for 100 MB, +10–30 s for 1 GB), and S3 conditional writes provide the single-writer lease — +but the operational maturity caveat (pin a known-good version; test restores) +stands. + +Sources: +[Google SRE — SLOs](https://sre.google/sre-book/service-level-objectives/), +[Google SRE — Error Budget Policy](https://sre.google/workbook/error-budget-policy/), +[Error budgets guide (OneUptime)](https://oneuptime.com/blog/post/2025-09-03-what-are-error-budgets/view), +[Divvi Up — privacy-preserving telemetry + DP](https://divviup.org/blog/combining-privacy-preserving-telemetry-with-differential-privacy/), +[Divvi Up in Firefox](https://divviup.org/blog/divvi-up-in-firefox/), +[Divvi Up (LWN)](https://lwn.net/Articles/983843/), +[Argo Rollouts — canary](https://argo-rollouts.readthedocs.io/en/stable/features/canary/), +[Canary + automated rollback (Headout)](https://www.headout.studio/canary-deployment-with-automated-rollback/), +[Litestream v0.5 (Fly)](https://fly.io/blog/litestream-v050-is-here/). + +## Key Findings + +1. **The privacy/observability tension is already resolved in code — for one + hub.** The consent + scrub + bucket + hash pipeline means the operator can see + *operational* signal and *opt-in aggregate* usage without ever touching + content. The job is to lift it to the *fleet*, not to invent it. +2. **Three planes, not two.** Conflating "operational health" (ours by necessity) + with "product usage" (theirs, consented) is the trap. Operational telemetry + for a managed hub is legitimately the operator's — it's the infra we run — and + must be content-free and tenant-scoped. Product usage must stay consent-gated + and, across tenants, privacy-preserving (DAP/Prio). +3. **SLAs only become real once SLIs are measured.** Until the control plane + derives availability/latency from `/health` + the ops bridge, `SlaLevel` is + theater. Measurement first, then published SLOs, then error budgets. +4. **Error budgets are the missing link between upgrades and SLAs.** The same + number that defines the promise also governs how aggressively we roll new + images. One mechanism, two payoffs. +5. **Backups are continuous but unproven.** "We replicate to R2" is not "we can + restore your hub." An automated restore-verification drill is the single + highest-trust, lowest-glamour win. +6. **Uptime is a wake-latency problem, not an always-on problem.** Scale-to-zero + means the honest SLI for entry tiers is "did it wake quickly and serve?" — so + the SLO must be defined on *successful requests including cold starts*, not + raw machine uptime. +7. **A reconciliation loop is the simplest automation that delivers all four.** + Desired-state-per-tenant + a controller that converges (provision, upgrade, + demote, restart-unhealthy, alert) is less code and more robust than four + separate scripts — and it's how the user's "automate as much as possible / + keep it simple / max uptime" goals are actually met. + +## Options And Tradeoffs + +### A. Telemetry collection topology + +```mermaid +flowchart TB + subgraph hub["Each tenant hub (E2E-encrypted data)"] + H1["/health · /ready · /metrics"] + H2["telemetry-bridge
producer:'hub' (ops)"] + H3["client telemetry ingest
(consent-gated, bucketed)"] + ENC["encrypted content
NEVER readable"] + end + subgraph cp["Control plane (operator)"] + OBS["Fleet observability store
per-tenant SLIs"] + SLO["SLO / error-budget engine"] + STAT["Status page"] + end + subgraph agg["Privacy-preserving aggregation"] + DAP["DAP / Prio (2 non-colluding aggregators)
cross-tenant product metrics"] + end + H1 -->|"poll / scrape"| OBS + H2 -->|"push ops events"| OBS + OBS --> SLO --> STAT + H3 -->|"opt-in, bucketed"| DAP + DAP -->|"aggregate only"| OBS + ENC -. "never" .-x OBS + classDef no fill:#fee,stroke:#c00,stroke-dasharray:4 + class ENC no +``` + +- **(A1) Reuse the hub telemetry-bridge + `/health`, push to a central store.** + Lowest new surface (the bridge exists); reuses the OTel event shape. Ops plane + only. **Recommended for Plane 1.** +- **(A2) Standard OTel/Prometheus scrape from the control plane.** Industry + standard, great tooling, but adds a metrics stack to operate and a scrape path + into every tenant network. Good *later* if the bridge proves limiting. +- **(A3) DAP/Prio for cross-tenant product analytics.** The only way to learn + "which features people use" across tenants **without** per-tenant exposure. + Higher integration cost (two aggregators, OHTTP relay) — **defer to a Plane-2 + milestone**, but design the egress seam for it now (it already exists). + +### B. Upgrade strategy + +| Option | Mechanism | Pros | Cons | +|---|---|---|---| +| **B1. Per-tenant one-at-a-time** (today) | loop `upgradeTenant` | trivial | no safety, no rollback, manual | +| **B2. Canary → waves → auto-rollback** ✅ | cohort by risk/plan, gate on SLI, re-point tag on regression | catches bad images at ~1% blast radius; immutable tags = instant rollback | needs SLIs + a rollout engine | +| **B3. Blue-green per tenant** | run new+old, cut over | zero-downtime cutover | ~2× cost per tenant during rollout; overkill at entry tiers | + +**Recommended: B2.** Cohorts: **xNet's own hubs → opt-in beta tenants → wave by +plan tier (free/personal → family → team → enterprise last)**, each wave gated on +the fleet error budget and per-tenant health, auto-rollback by re-pointing to the +previous pinned tag. + +### C. Uptime model per tier + +- **C1. Always-warm everything** — simplest mental model, but kills the entry + margin (warm ≈ $6/mo vs $5/mo revenue). +- **C2. Scale-to-zero everywhere** — cheapest, but every request can eat a cold + start; bad for `team`+. +- **C3. Hybrid by plan tier** ✅ — `dedicated-warm` (min-instances 1) for team+, + `dedicated-sleep` for personal/family with fast Litestream restore. This is + already the catalog's intent; make the SLO tier-aware (warm tiers get a latency + SLO; sleep tiers get a *wake-success* SLO). + +### D. SLA posture per tier + +| Plan | Isolation | Published SLO (proposed) | Error budget / 30d | Enforcement | +|---|---|---|---|---| +| demo | pooled | none | — | best-effort | +| personal / family | dedicated-sleep | "best-effort," internal wake-success ≥ 99% | informational | monitor only | +| team | dedicated-warm | **99.9%** availability | ~43 min | alert + status page | +| community / company | dedicated-project | **99.9%** | ~43 min | alert + status page | +| enterprise | region-pinned | **custom** (e.g. 99.95%) | ~22 min | contractual credits | + +## Recommendation + +**Make the control plane a reconciliation loop with an observability spine, then +layer SLO-gated upgrades and a restore drill on top.** Concretely, in four +phases that each ship value: + +1. **Phase 1 — See the fleet (Plane 1, ops).** Control plane polls each hub's + `/health`/`/ready` and consumes the `telemetry-bridge` ops events into a + central per-tenant store; derive three SLIs — **availability** (successful / + valid requests, *including* cold-start waits), **latency** (bucketed), **error + rate** — plus **backup freshness** (`isReplicaFresh`) and **wake latency**. + Surface them in an operator dashboard and a public status page. *No new + privacy surface: ops telemetry is content-free and tenant-scoped.* +2. **Phase 2 — Promise & protect (SLAs).** Compute per-tenant + fleet **error + budgets** from the SLIs against the per-tier SLO table above. Wire alerting and + the **error-budget policy** (freeze risky deploys when exhausted). Enterprise + `custom` adds contractual credits. +3. **Phase 3 — Upgrade safely (automation).** Build the rollout engine on + `upgradeTenant`: **canary cohort → waves by tier → automatic rollback** (re-point + to the previous immutable tag) when a wave's SLIs regress or the error budget + dips. Drive it from desired-state (`targetVersion` per tenant) so it's + restartable and idempotent. +4. **Phase 4 — Prove backups + learn the product.** Add an **automated + restore-verification drill** (nightly: restore a sample of tenants into a + throwaway hub, assert row counts / a health query, alert on failure), an R2 + **retention/PITR lifecycle**, and — for Plane 2 — route the existing + consent-gated client usage egress through **DAP/Prio** so cross-tenant product + insight never exposes an individual tenant. + +This keeps faith with the user's three asks: **private** (three-plane model, DAP +for cross-tenant, encryption boundary never crossed), **observable** (ops SLIs + +opt-in aggregate usage), and **simple/high-uptime** (one reconciliation loop + +continuous backups + scale-to-zero with fast wake + auto-rollback). + +### Error-budget-gated canary upgrade (the upgrade↔SLA link) + +```mermaid +sequenceDiagram + autonumber + participant Op as Operator / cron + participant CP as Control plane (rollout engine) + participant Obs as Observability (SLIs) + participant Fleet as Tenant hubs + + Op->>CP: release xnet-hub@1.4.0 + CP->>Obs: fleet error budget healthy? + Obs-->>CP: yes (>50%) + CP->>Fleet: upgrade canary cohort (xNet's own + beta) to @1.4.0 + CP->>Obs: watch canary SLIs (15 min bake) + alt canary regresses (errors↑ / latency↑ / wake fails) + Obs-->>CP: SLI breach + CP->>Fleet: rollback canary → previous pinned tag + CP-->>Op: ABORT + incident + else canary healthy + Obs-->>CP: green + CP->>Fleet: wave 1 (free/personal) → bake → wave 2 (family) → … → enterprise + Note over CP,Fleet: each wave gated on per-wave SLIs + remaining error budget + CP-->>Op: rollout complete + end +``` + +### Tenant lifecycle as a reconciliation state machine + +```mermaid +stateDiagram-v2 + [*] --> provisioning + provisioning --> hot: hub healthy (/ready 200) + hot --> upgrading: targetVersion changed + upgrading --> hot: new image healthy + upgrading --> hot: rollback to prior tag (unhealthy) + hot --> degraded: SLI breach / health flapping + degraded --> hot: self-heal (restart) succeeds + degraded --> provisioning: re-provision (crash loop) + hot --> cold: idle > coldAfter AND isFullySynced + cold --> hot: request arrives → restore-from-R2 + hot --> suspended: subscription canceled (0192) + suspended --> cold: retain R2 replica + cold --> deleted: delete-my-data (irreversible) + deleted --> [*] + note right of cold + Backup freshness (isReplicaFresh) is an SLI in every live state. + A nightly restore drill validates cold → hot for a sample. + end note +``` + +### The three telemetry planes vs. the encryption boundary + +```mermaid +flowchart LR + subgraph device["User device / hub process"] + C["Document content
exact identifiers"] + P["Product events
(feature usage)"] + O["Operational signal
(health, latency, errors)"] + end + C -->|encrypt| ENC[("Encrypted blob → R2
operator CANNOT read")] + P -->|"consent ≥ anonymous
scrub + bucket"| AGG[("DAP/Prio aggregate
no individual exposed")] + O -->|"content-free
tenant-scoped"| OPS[("Fleet observability
SLIs, alerts, status")] + AGG --> PROD["Product decisions:
what to build"] + OPS --> REL["Reliability:
SLAs, debugging, upgrades"] + classDef enc fill:#eef,stroke:#33a + class ENC enc +``` + +## Example Code + +### Phase 1 — derive SLIs from health polling (sketch, control plane) + +```ts +// apps/cloud/src/observability/sli.ts (sketch) +export interface HealthSample { ok: boolean; latencyMs: number; atMs: number } + +/** Availability over a window: successful / valid probes (cold-start waits count as valid). */ +export function availability(samples: HealthSample[]): number { + if (!samples.length) return 1 + return samples.filter((s) => s.ok).length / samples.length +} + +/** Error budget remaining as a fraction of the allowance (1 = full, 0 = exhausted). */ +export function errorBudgetRemaining(sli: number, slo: number): number { + const allowed = 1 - slo // e.g. 0.001 for 99.9% + const used = Math.max(0, 1 - sli) + return allowed === 0 ? 1 : Math.max(0, 1 - used / allowed) +} + +/** Backup freshness SLI reuses the shipped helper. */ +import { isReplicaFresh } from '@xnetjs/cloud/litestream' +export const backupHealthy = (lastWriteMs: number, lastSyncMs: number) => + isReplicaFresh(lastWriteMs, lastSyncMs, 5 * 60_000) // 5-min lag budget +``` + +### Phase 3 — error-budget-gated rollout over the existing one-step upgrade + +```ts +// apps/cloud/src/rollout/engine.ts (sketch) +export async function rollWave( + cp: ControlPlane, tenants: TenantRecord[], target: string, + sli: (id: string) => Promise, opts: { slo: number; bakeMs: number } +): Promise<{ promoted: string[]; rolledBack: string[] }> { + const promoted: string[] = [], rolledBack: string[] = [] + for (const t of tenants) { + const prev = t.targetVersion + await cp.upgradeTenant(t.tenantId, target) // immutable tag (never :latest) + await sleep(opts.bakeMs) // bake + if ((await sli(t.tenantId)) < opts.slo) { + await cp.upgradeTenant(t.tenantId, prev) // instant rollback: re-point tag + rolledBack.push(t.tenantId) + } else promoted.push(t.tenantId) + } + return { promoted, rolledBack } +} +``` + +### Phase 4 — nightly restore-verification drill (the trust win) + +```ts +// apps/cloud/src/backup/restore-drill.ts (sketch) +// Provision a THROWAWAY hub from a tenant's R2 replica and assert it restores. +export async function verifyRestore(cp: ControlPlane, p: Provisioner, tenantId: string) { + const probe = await p.provision({ + tenantId: `drill-${tenantId}`, /* …entitlements… */ + restoreFromR2: `t/${tenantId}/db`, env: {} + }) + const res = await fetch(`${probe.hubUrl}/ready`) // restored + writable? + await p.destroy(probe.substrateRef) // always tear down + if (!res.ok) throw new Error(`restore drill failed for ${tenantId}`) +} +``` + +## Risks And Open Questions + +- **"Operational telemetry is ours" must be stated in the privacy policy and the + dashboard.** Even content-free, tenant-scoped health data is *data about the + tenant*. Be explicit: what we collect (health, latency buckets, error kinds, + backup freshness), what we never collect (content, plaintext), and let + enterprise pin/region-scope it. Getting this wording wrong erodes the core + promise. +- **Polling vs. push, and the cold tenant problem.** A scale-to-zero hub has no + process to scrape; availability for sleep tiers must be measured at the *edge* + (did a real request wake + succeed?), not by polling a hub that's intentionally + off. Synthetic wake-probes cost money (they defeat scale-to-zero) — sample, + don't probe-every-minute. +- **DAP/Prio needs a second non-colluding aggregator.** The privacy guarantee is + only real with two independent operators; running both ourselves defeats it. + This is an org/partnership decision (e.g. ISRG Divvi Up), not just code. +- **Error-budget-gated freezes can starve security fixes.** The policy must + exempt reliability/security patches from the freeze (Google's does). +- **Restore drills cost real money and write amplification.** Sample a rotating + subset nightly, not the whole fleet; tear down throwaway hubs promptly. +- **Single-writer fence under partition.** A control-plane partition during + upgrade/reactivate could start a second writer; the Litestream S3 + conditional-write lease must be exercised and chaos-tested before GA. +- **Provisioner adapters are still stubs.** None of upgrades/cold-tiering/restore + runs on a real substrate until `CloudRunLitestreamProvisioner` is implemented — + this is the upstream blocker for *all* of the above (0180). +- **Open question: where does the fleet observability store live?** It is + itself a tenant-zero hosting problem (same as the control-plane DB). Reusing a + `telemetry.db`-style SQLite + Litestream for the operator's own metrics is the + consistent answer. +- **Open question: status-page granularity.** Per-tenant status leaks fleet + composition; a single aggregate status is safe but coarse. Likely: aggregate + public status + per-tenant health in the authenticated dashboard. + +## Implementation Checklist + +**Phase 1 — Fleet observability (ops plane):** +- [ ] Control-plane poller for each hot tenant's `/health` + `/ready`; record `HealthSample`s. +- [ ] Consume the hub `telemetry-bridge` ops events centrally (enable it for managed hubs). +- [ ] Per-tenant SLI store: availability, latency buckets, error rate, backup freshness (`isReplicaFresh`), wake latency. +- [ ] Operator fleet dashboard + a public aggregate status page. +- [ ] Privacy-policy + dashboard copy: what ops telemetry we collect and what we never collect. + +**Phase 2 — SLAs (measure → promise → protect):** +- [ ] Per-tier SLO table (warm tiers 99.9% availability; sleep tiers wake-success; enterprise custom). +- [ ] Error-budget computation per tenant + fleet; alerting on burn rate. +- [ ] Error-budget policy (freeze risky deploys when exhausted; exempt security/reliability). +- [ ] Enterprise contractual credits hook. + +**Phase 3 — Upgrade engine (automation):** +- [ ] Rollout engine over `upgradeTenant`: canary cohort → waves by tier, desired-state driven. +- [ ] SLI bake + automatic rollback (re-point to previous pinned immutable tag). +- [ ] Gate each wave on remaining error budget; record rollout state for restartability. +- [ ] Implement `CloudRunLitestreamProvisioner.upgrade` (unblocks real rollouts). + +**Phase 4 — Backups proven + product learning:** +- [ ] Automated nightly **restore-verification drill** over a rotating sample; alert on failure. +- [ ] R2 retention / point-in-time-recovery lifecycle policy (e.g. 30-day PITR + 90-day daily snapshots). +- [ ] Backup-freshness alert wired to the SLI store; exercise the single-writer S3 lease (chaos test). +- [ ] Route consent-gated client usage egress through **DAP/Prio** for cross-tenant aggregates (Plane 2). +- [ ] Reconciliation loop: converge desired vs. actual (provision/upgrade/demote/self-heal/restart-unhealthy). + +## Validation Checklist + +- [ ] A bad hub image is caught in the canary cohort and auto-rolled-back before any paying wave is touched. +- [ ] A tenant's availability/latency/error SLIs are visible per-tenant and in aggregate, derived from real health signal. +- [ ] An exhausted error budget freezes feature rollouts but not a security patch. +- [ ] The nightly restore drill provisions a throwaway hub from R2 and passes `/ready`; a deliberately corrupted replica trips the alert. +- [ ] Measured RPO ≤ 1 s on hard kill and ~0 on graceful drain; measured cold wake latency meets the sleep-tier wake-success SLO. +- [ ] The public status page reflects a real induced incident; per-tenant health shows only in the authenticated dashboard. +- [ ] An auditor can confirm the operator never receives document content or plaintext identifiers — only hashed DIDs, buckets, and ops signal. +- [ ] Cross-tenant product metrics (Plane 2) reveal aggregates only; no single tenant's usage is recoverable from the aggregator output. +- [ ] A control-plane restart resumes an in-flight rollout from recorded desired-state without double-upgrading. + +## References + +- Predecessors: [0180 — xNet Cloud Architecture & Completion Status](0180_[_]_XNET_CLOUD_ARCHITECTURE_AND_COMPLETION_STATUS.md), [0192 — xNet Cloud Onboarding & UI Hosting](0192_[_]_XNET_CLOUD_ONBOARDING_AND_UI_HOSTING.md) +- Lineage: [0175 — Fleet Deployment & AI Gateway](0175_[_]_MANAGED_HUB_FLEET_DEPLOYMENT_AND_AI_GATEWAY.md), [0177/0178 — Cost-Efficient SQLite Hosting & Cold Tiering](0178_[_]_COST_EFFICIENT_SQLITE_HOSTING_NO_LIBSQL_MIGRATION.md), [0187 — Hub-Hosted Telemetry Store](0187_[x]_HUB_HOSTED_TELEMETRY_STORE_AND_ANALYTICS_DASHBOARD.md), [0190 — Deep Performance Telemetry & Tracing](0190_[_]_DEEP_PERFORMANCE_TELEMETRY_AND_STACK_TRACING.md) +- Upgrades/provisioning: [apps/cloud/src/control-plane.ts](apps/cloud/src/control-plane.ts), [packages/cloud/src/provisioner/types.ts](packages/cloud/src/provisioner/types.ts), [packages/hub/Dockerfile](packages/hub/Dockerfile) +- Backups/tiering: [packages/cloud/src/litestream/](packages/cloud/src/litestream/index.ts), [packages/hub/litestream-entrypoint.sh](packages/hub/litestream-entrypoint.sh), [packages/hub/src/storage/litestream.ts](packages/hub/src/storage/litestream.ts) +- Health/SLA: [packages/hub/src/server.ts](packages/hub/src/server.ts), [packages/hub/src/config.ts](packages/hub/src/config.ts), [packages/entitlements/src/plans.ts](packages/entitlements/src/plans.ts), [packages/cloud/src/cost/pricing.ts](packages/cloud/src/cost/pricing.ts) +- Telemetry: [packages/telemetry/src/](packages/telemetry/src/index.ts), [packages/hub/src/telemetry/](packages/hub/src/telemetry/store.ts), [packages/hub/src/middleware/telemetry-bridge.ts](packages/hub/src/middleware/telemetry-bridge.ts), [packages/cloud/src/ai/metering.ts](packages/cloud/src/ai/metering.ts) +- External: [Google SRE — SLOs](https://sre.google/sre-book/service-level-objectives/), [Error Budget Policy](https://sre.google/workbook/error-budget-policy/), [Divvi Up — DAP + differential privacy](https://divviup.org/blog/combining-privacy-preserving-telemetry-with-differential-privacy/), [Divvi Up in Firefox](https://divviup.org/blog/divvi-up-in-firefox/), [Argo Rollouts — canary](https://argo-rollouts.readthedocs.io/en/stable/features/canary/), [Litestream v0.5 (Fly)](https://fly.io/blog/litestream-v050-is-here/) From 5a3381defabfcd6a59be3d9dc6c5545f67618236 Mon Sep 17 00:00:00 2001 From: xNet Test Date: Wed, 17 Jun 2026 10:56:59 -0700 Subject: [PATCH 2/6] feat(cloud): fleet SLIs, SLOs, and error-budget policy (0193 phase 1+2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turns the declared-but-unenforced SlaLevel into measured observability. - observability/sli.ts: pure SLI math — availability (cold-start waits count as valid), error rate, latency percentile, error-budget-remaining, burn rate, backup freshness (reuses the shipped isReplicaFresh) - observability/slo.ts: maps each plan's SlaLevel → a measurable SLO (community/ company 99.9%, enterprise 99.95%, others best-effort) + errorBudgetMs + Google-SRE budget policy (ship/caution/freeze) - observability/health.ts: HealthProbe port + FakeHealthProbe + httpHealthProbe, a bounded per-tenant sample ring, tenantSli + fleetSummary - control-plane.listTenants(); admin-gated GET /internal/fleet/health returns per-tenant SLIs + a fleet aggregate (503 when observability unconfigured) All content-free + tenant-scoped (respects the E2E boundary). 18 new tests. Co-Authored-By: Claude Opus 4.8 --- apps/cloud/src/control-plane.ts | 5 + apps/cloud/src/fleet.test.ts | 79 ++++++++++ apps/cloud/src/index.ts | 29 ++++ apps/cloud/src/observability/health.ts | 142 ++++++++++++++++++ .../src/observability/observability.test.ts | 117 +++++++++++++++ apps/cloud/src/observability/sli.ts | 76 ++++++++++ apps/cloud/src/observability/slo.ts | 59 ++++++++ apps/cloud/src/server.ts | 19 +++ 8 files changed, 526 insertions(+) create mode 100644 apps/cloud/src/fleet.test.ts create mode 100644 apps/cloud/src/observability/health.ts create mode 100644 apps/cloud/src/observability/observability.test.ts create mode 100644 apps/cloud/src/observability/sli.ts create mode 100644 apps/cloud/src/observability/slo.ts diff --git a/apps/cloud/src/control-plane.ts b/apps/cloud/src/control-plane.ts index 62ccb7ecf..7ef651237 100644 --- a/apps/cloud/src/control-plane.ts +++ b/apps/cloud/src/control-plane.ts @@ -299,6 +299,11 @@ export class ControlPlane { return this.deps.tenants.get(tenantId) } + /** Every tenant the control plane knows about (fleet observability + rollouts). */ + listTenants(): Promise { + return this.deps.tenants.list() + } + /** R2 object path holding a tenant's SQLite snapshot (matches the Litestream replica path). */ private snapshotKey(tenantId: string): string { return `t/${tenantId}/db` diff --git a/apps/cloud/src/fleet.test.ts b/apps/cloud/src/fleet.test.ts new file mode 100644 index 000000000..8be3ead2a --- /dev/null +++ b/apps/cloud/src/fleet.test.ts @@ -0,0 +1,79 @@ +import { MemoryBillingIdentityProvider } from '@xnetjs/cloud/identity' +import { describe, expect, it } from 'vitest' +import { FakeTenantBillingGateway } from './billing-gateway' +import { HealthSampleStore } from './observability/health' +import { createControlPlaneApp } from './server' +import { buildControlPlane } from './index' + +const INTERNAL = 'secret123' + +function fleetApp() { + const billing = new MemoryBillingIdentityProvider('https://auth.test/authorize') + const { controlPlane } = buildControlPlane({ billing }) + const health = new HealthSampleStore() + const app = createControlPlaneApp({ + controlPlane, + billing, + payments: new FakeTenantBillingGateway(), + health, + internalSecret: INTERNAL, + sessionSecret: 'sess', + baseUrl: '', + nowMs: () => 2_000_000 // fixed clock so recorded samples fall inside the SLO window + }) + return { app, controlPlane, health } +} + +async function provision( + app: ReturnType['app'], + customerRef: string, + plan = 'community' +) { + await app.request('/webhook', { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ type: 'checkout.completed', customerRef, plan }) + }) +} + +describe('GET /internal/fleet/health', () => { + it('guards behind the internal secret', async () => { + const { app } = fleetApp() + expect((await app.request('/internal/fleet/health')).status).toBe(403) + }) + + it('reports per-tenant SLIs + a fleet aggregate', async () => { + const { app, controlPlane, health } = fleetApp() + await provision(app, 'user_a', 'community') + const tenant = await controlPlane.getTenantForBilling('user_a') + + // Record some failing samples for the live tenant → budget should drain. + for (let i = 0; i < 20; i++) { + health.record(tenant!.tenantId, { ok: i % 2 === 0, latencyMs: 10, atMs: 1000 + i }) + } + + const res = await app.request('/internal/fleet/health', { + headers: { 'x-internal-secret': INTERNAL } + }) + expect(res.status).toBe(200) + const body = (await res.json()) as { + fleet: { tenantCount: number; freezing: number; worstBudgetRemaining: number } + cold: number + tenants: { tenantId: string; availability: number; policy: string }[] + } + expect(body.fleet.tenantCount).toBe(1) + expect(body.tenants[0].availability).toBeCloseTo(0.5, 5) + expect(body.tenants[0].policy).toBe('freeze') + expect(body.fleet.freezing).toBe(1) + }) + + it('503s when observability is not configured', async () => { + const billing = new MemoryBillingIdentityProvider() + const { controlPlane } = buildControlPlane({ billing }) + const app = createControlPlaneApp({ controlPlane, billing, internalSecret: INTERNAL }) + const res = await app.request('/internal/fleet/health', { + headers: { 'x-internal-secret': INTERNAL } + }) + expect(res.status).toBe(503) + }) +}) diff --git a/apps/cloud/src/index.ts b/apps/cloud/src/index.ts index f32e1bd5e..71b11086b 100644 --- a/apps/cloud/src/index.ts +++ b/apps/cloud/src/index.ts @@ -40,6 +40,35 @@ export { type DeviceGrantStore, type CodeGenerator } from './device-grant' +export { + availability, + errorRate, + latencyPercentile, + errorBudgetRemaining, + burnRate, + backupHealthy, + windowed, + type HealthSample +} from './observability/sli' +export { + sloForSla, + sloForPlan, + errorBudgetMs, + budgetPolicy, + type SloTarget, + type BudgetPolicy +} from './observability/slo' +export { + HealthSampleStore, + FakeHealthProbe, + httpHealthProbe, + sampleTenantHealth, + tenantSli, + fleetSummary, + type HealthProbe, + type TenantSli, + type FleetSummary +} from './observability/health' /** * Pick the billing identity provider from the environment. WorkOS AuthKit (free diff --git a/apps/cloud/src/observability/health.ts b/apps/cloud/src/observability/health.ts new file mode 100644 index 000000000..edaecdd87 --- /dev/null +++ b/apps/cloud/src/observability/health.ts @@ -0,0 +1,142 @@ +/** + * xNet Cloud — fleet health probing + per-tenant SLI summaries (exploration 0193). + * + * The control plane polls each hot tenant's `/health` (or `/ready`) and records a + * content-free {@link HealthSample}. A rolling in-memory window per tenant feeds + * the SLI math in `sli.ts`; production swaps the store for a durable one (same + * stance as the tenant registry). The probe is a port so it's keyless-testable. + */ + +import { type PlanId } from '@xnetjs/entitlements' +import { + availability, + errorRate, + latencyPercentile, + errorBudgetRemaining, + windowed, + type HealthSample +} from './sli' +import { budgetPolicy, sloForPlan, type BudgetPolicy } from './slo' + +/** Probes a single hub. The real adapter hits `${hubUrl}/health`. */ +export interface HealthProbe { + probe(hubUrl: string): Promise<{ ok: boolean; latencyMs: number }> +} + +/** Default probe: GET `${hubUrl}/health`, ok on a 2xx within the timeout. */ +export function httpHealthProbe(fetchImpl: typeof fetch = fetch, timeoutMs = 5000): HealthProbe { + return { + async probe(hubUrl: string) { + const startedAtMs = Date.now() + const ctrl = new AbortController() + const timer = setTimeout(() => ctrl.abort(), timeoutMs) + try { + const res = await fetchImpl(`${hubUrl.replace(/\/$/, '')}/health`, { signal: ctrl.signal }) + return { ok: res.ok, latencyMs: Date.now() - startedAtMs } + } catch { + return { ok: false, latencyMs: Date.now() - startedAtMs } + } finally { + clearTimeout(timer) + } + } + } +} + +/** Scripted probe for tests — maps a hubUrl to a fixed result. */ +export class FakeHealthProbe implements HealthProbe { + constructor(private readonly results: Record) {} + async probe(hubUrl: string): Promise<{ ok: boolean; latencyMs: number }> { + return this.results[hubUrl] ?? { ok: false, latencyMs: 0 } + } +} + +/** A bounded per-tenant ring of health samples. */ +export class HealthSampleStore { + private readonly byTenant = new Map() + constructor(private readonly capacity = 2000) {} + + record(tenantId: string, sample: HealthSample): void { + const arr = this.byTenant.get(tenantId) ?? [] + arr.push(sample) + if (arr.length > this.capacity) arr.splice(0, arr.length - this.capacity) + this.byTenant.set(tenantId, arr) + } + + samples(tenantId: string): HealthSample[] { + return [...(this.byTenant.get(tenantId) ?? [])] + } +} + +/** Probe one tenant and record the sample. Returns the sample. */ +export async function sampleTenantHealth( + probe: HealthProbe, + store: HealthSampleStore, + tenant: { tenantId: string; hubUrl: string }, + nowMs: number +): Promise { + const r = await probe.probe(tenant.hubUrl) + const sample: HealthSample = { ok: r.ok, latencyMs: r.latencyMs, atMs: nowMs } + store.record(tenant.tenantId, sample) + return sample +} + +/** The derived SLI summary for one tenant against its plan's SLO. */ +export interface TenantSli { + tenantId: string + plan: PlanId + sloLabel: string + availability: number + errorRate: number + p95LatencyMs: number + budgetRemaining: number + policy: BudgetPolicy + sampleCount: number +} + +/** A fleet-wide rollup of per-tenant SLIs (the operator's at-a-glance health). */ +export interface FleetSummary { + tenantCount: number + worstBudgetRemaining: number + /** Tenants whose policy is `freeze` (a deploy freeze should be in effect). */ + freezing: number + byPolicy: Record +} + +export function fleetSummary(slis: TenantSli[]): FleetSummary { + const byPolicy: Record = { ship: 0, caution: 0, freeze: 0 } + let worst = 1 + for (const s of slis) { + byPolicy[s.policy] += 1 + worst = Math.min(worst, s.budgetRemaining) + } + return { + tenantCount: slis.length, + worstBudgetRemaining: slis.length ? worst : 1, + freezing: byPolicy.freeze, + byPolicy + } +} + +/** Summarize a tenant's SLIs over the SLO window. */ +export function tenantSli( + store: HealthSampleStore, + tenant: { tenantId: string; plan: PlanId; hubUrl: string }, + nowMs: number +): TenantSli { + const slo = sloForPlan(tenant.plan) + const windowMs = slo.windowDays * 24 * 60 * 60 * 1000 + const samples = windowed(store.samples(tenant.tenantId), windowMs, nowMs) + const avail = availability(samples) + const remaining = errorBudgetRemaining(avail, slo.objective) + return { + tenantId: tenant.tenantId, + plan: tenant.plan, + sloLabel: slo.label, + availability: avail, + errorRate: errorRate(samples), + p95LatencyMs: latencyPercentile(samples, 0.95), + budgetRemaining: remaining, + policy: budgetPolicy(remaining), + sampleCount: samples.length + } +} diff --git a/apps/cloud/src/observability/observability.test.ts b/apps/cloud/src/observability/observability.test.ts new file mode 100644 index 000000000..1e81daf50 --- /dev/null +++ b/apps/cloud/src/observability/observability.test.ts @@ -0,0 +1,117 @@ +import { describe, expect, it } from 'vitest' +import { FakeHealthProbe, HealthSampleStore, sampleTenantHealth, tenantSli } from './health' +import { + availability, + backupHealthy, + burnRate, + errorBudgetRemaining, + errorRate, + latencyPercentile, + windowed, + type HealthSample +} from './sli' +import { budgetPolicy, errorBudgetMs, sloForPlan, sloForSla } from './slo' + +const ok = (atMs: number, latencyMs = 20): HealthSample => ({ ok: true, latencyMs, atMs }) +const bad = (atMs: number): HealthSample => ({ ok: false, latencyMs: 0, atMs }) + +describe('SLI math', () => { + it('availability counts successes over valid probes; empty → 1', () => { + expect(availability([])).toBe(1) + expect(availability([ok(1), ok(2), bad(3), ok(4)])).toBe(0.75) + expect(errorRate([ok(1), bad(2)])).toBe(0.5) + }) + + it('windows samples by time', () => { + const s = [ok(0), ok(50), ok(100)] + expect(windowed(s, 60, 100).map((x) => x.atMs)).toEqual([50, 100]) + }) + + it('takes latency percentiles over successful probes', () => { + const s = [ok(1, 10), ok(2, 20), ok(3, 30), ok(4, 100), bad(5)] + expect(latencyPercentile(s, 0.95)).toBe(100) + expect(latencyPercentile([], 0.95)).toBe(0) + }) + + it('computes error budget remaining + burn rate against an objective', () => { + // 99.9% objective, observed 99.95% availability → half the budget left. + expect(errorBudgetRemaining(0.9995, 0.999)).toBeCloseTo(0.5, 5) + expect(burnRate(0.9995, 0.999)).toBeCloseTo(0.5, 5) + // Exactly at objective → exhausted. + expect(errorBudgetRemaining(0.999, 0.999)).toBeCloseTo(0, 5) + // Below objective → over budget, clamped to 0 remaining. + expect(errorBudgetRemaining(0.99, 0.999)).toBe(0) + // No objective (best-effort) → always full, never burns. + expect(errorBudgetRemaining(0.5, null)).toBe(1) + expect(burnRate(0.5, null)).toBe(0) + }) + + it('reports backup freshness from replica lag', () => { + expect(backupHealthy(1000, 1000)).toBe(true) + expect(backupHealthy(1_000_000, 0)).toBe(false) + }) +}) + +describe('SLO catalog + budget policy', () => { + it('maps SLA levels to objectives', () => { + expect(sloForSla('99.9').objective).toBe(0.999) + expect(sloForSla('custom').objective).toBe(0.9995) + expect(sloForSla('best-effort').objective).toBeNull() + expect(sloForSla('none').objective).toBeNull() + }) + + it('derives the SLO from the plan tier', () => { + expect(sloForPlan('community').objective).toBe(0.999) // dedicated-project, 99.9 + expect(sloForPlan('company').objective).toBe(0.999) + expect(sloForPlan('team').objective).toBeNull() // best-effort + expect(sloForPlan('personal').objective).toBeNull() // best-effort + expect(sloForPlan('enterprise').objective).toBe(0.9995) + }) + + it('converts an objective to allowed downtime', () => { + // 99.9% over 30d ≈ 43.2 minutes. + expect(Math.round(errorBudgetMs(sloForSla('99.9')) / 60000)).toBe(43) + expect(errorBudgetMs(sloForSla('best-effort'))).toBe(Number.POSITIVE_INFINITY) + }) + + it('applies the Google error-budget policy thresholds', () => { + expect(budgetPolicy(0.6)).toBe('ship') + expect(budgetPolicy(0.2)).toBe('caution') + expect(budgetPolicy(0)).toBe('freeze') + expect(budgetPolicy(-0.1)).toBe('freeze') + }) +}) + +describe('health sampling → tenant SLI', () => { + it('probes a hub, records, and summarizes SLIs', async () => { + const probe = new FakeHealthProbe({ 'wss://t.hub': { ok: true, latencyMs: 30 } }) + const store = new HealthSampleStore() + const tenant = { tenantId: 't_a', plan: 'community' as const, hubUrl: 'wss://t.hub' } + for (let i = 0; i < 10; i++) await sampleTenantHealth(probe, store, tenant, 1000 + i) + + const sli = tenantSli(store, tenant, 1100) + expect(sli.availability).toBe(1) + expect(sli.budgetRemaining).toBe(1) + expect(sli.policy).toBe('ship') + expect(sli.p95LatencyMs).toBe(30) + expect(sli.sampleCount).toBe(10) + expect(sli.sloLabel).toContain('99.9') + }) + + it('drains the budget and flips policy to freeze when a 99.9 hub is mostly down', () => { + const store = new HealthSampleStore() + const tenant = { tenantId: 't_b', plan: 'community' as const, hubUrl: 'wss://b.hub' } + // 50% failures over the window vastly exceeds a 0.1% budget. + for (let i = 0; i < 100; i++) store.record('t_b', i % 2 ? ok(i) : bad(i)) + const sli = tenantSli(store, tenant, 200) + expect(sli.availability).toBeCloseTo(0.5, 5) + expect(sli.budgetRemaining).toBe(0) + expect(sli.policy).toBe('freeze') + }) + + it('caps the ring buffer at capacity', () => { + const store = new HealthSampleStore(5) + for (let i = 0; i < 20; i++) store.record('t_c', ok(i)) + expect(store.samples('t_c')).toHaveLength(5) + }) +}) diff --git a/apps/cloud/src/observability/sli.ts b/apps/cloud/src/observability/sli.ts new file mode 100644 index 000000000..44027c5f2 --- /dev/null +++ b/apps/cloud/src/observability/sli.ts @@ -0,0 +1,76 @@ +/** + * xNet Cloud — Service Level Indicators (pure math). + * + * SLIs are derived from health probes against each tenant hub (exploration 0193). + * Availability counts successful probes over valid probes — cold-start waits are + * *valid* (the request eventually succeeds), so scale-to-zero tenants aren't + * unfairly penalized. Everything here is content-free: a probe is ok/not-ok + a + * latency number, never anything about the tenant's data. + */ + +import { isReplicaFresh } from '@xnetjs/cloud/litestream' + +/** One health observation for a tenant hub. */ +export interface HealthSample { + ok: boolean + latencyMs: number + atMs: number +} + +/** Samples within `[nowMs - windowMs, nowMs]`. */ +export function windowed(samples: HealthSample[], windowMs: number, nowMs: number): HealthSample[] { + const floor = nowMs - windowMs + return samples.filter((s) => s.atMs >= floor && s.atMs <= nowMs) +} + +/** Availability = successful / valid probes. Empty window → 1 (no evidence of failure). */ +export function availability(samples: HealthSample[]): number { + if (samples.length === 0) return 1 + return samples.filter((s) => s.ok).length / samples.length +} + +/** Error rate = 1 − availability. */ +export function errorRate(samples: HealthSample[]): number { + return 1 - availability(samples) +} + +/** Latency percentile (q in [0,1]) over successful probes. Empty → 0. */ +export function latencyPercentile(samples: HealthSample[], q: number): number { + const oks = samples + .filter((s) => s.ok) + .map((s) => s.latencyMs) + .sort((a, b) => a - b) + if (oks.length === 0) return 0 + const idx = Math.min(oks.length - 1, Math.max(0, Math.floor(q * oks.length))) + return oks[idx] +} + +/** + * Error budget remaining as a fraction of the allowance (1 = full, 0 = exhausted). + * `objective` is the SLO as a fraction (e.g. 0.999); `null` = no published SLO → + * always "full" (best-effort tiers never burn a budget they don't have). + */ +export function errorBudgetRemaining(sli: number, objective: number | null): number { + if (objective === null) return 1 + const allowed = 1 - objective + if (allowed <= 0) return sli >= 1 ? 1 : 0 + const used = Math.max(0, 1 - sli) + return Math.max(0, 1 - used / allowed) +} + +/** How fast the budget is burning: used / allowed (>1 means over budget). */ +export function burnRate(sli: number, objective: number | null): number { + if (objective === null) return 0 + const allowed = 1 - objective + if (allowed <= 0) return sli >= 1 ? 0 : Infinity + return Math.max(0, 1 - sli) / allowed +} + +/** Backup-freshness SLI — reuses the shipped Litestream helper (5-min lag budget). */ +export function backupHealthy( + lastWriteMs: number, + lastSyncMs: number, + maxLagMs = 5 * 60_000 +): boolean { + return isReplicaFresh(lastWriteMs, lastSyncMs, maxLagMs) +} diff --git a/apps/cloud/src/observability/slo.ts b/apps/cloud/src/observability/slo.ts new file mode 100644 index 000000000..ca6450a67 --- /dev/null +++ b/apps/cloud/src/observability/slo.ts @@ -0,0 +1,59 @@ +/** + * xNet Cloud — Service Level Objectives + error-budget policy (exploration 0193). + * + * Ties the plan catalog's declared `SlaLevel` to a concrete, measurable SLO and + * the Google-SRE error-budget policy that gates fleet upgrades: a healthy budget + * ships fast, a low budget slows down, an exhausted budget freezes risky deploys + * (security/reliability fixes are always exempt — enforced at the call site). + */ + +import { PLAN_CATALOG, type PlanId, type SlaLevel } from '@xnetjs/entitlements' + +export interface SloTarget { + /** Availability objective as a fraction (e.g. 0.999). `null` = no published SLO. */ + objective: number | null + /** Rolling window the objective is measured over. */ + windowDays: number + /** Human label for dashboards/status. */ + label: string +} + +/** Map a plan's declared SLA level to a measurable SLO. */ +export function sloForSla(sla: SlaLevel): SloTarget { + switch (sla) { + case '99.9': + return { objective: 0.999, windowDays: 30, label: '99.9% uptime' } + case 'custom': + return { objective: 0.9995, windowDays: 30, label: '99.95% uptime (enterprise)' } + case 'best-effort': + return { objective: null, windowDays: 30, label: 'best-effort' } + case 'none': + default: + return { objective: null, windowDays: 30, label: 'no SLA' } + } +} + +/** The SLO for a plan tier. */ +export function sloForPlan(plan: PlanId): SloTarget { + return sloForSla(PLAN_CATALOG[plan].sla) +} + +/** Allowed downtime over the window, in ms (the error budget as time). ∞ if no SLO. */ +export function errorBudgetMs(slo: SloTarget): number { + if (slo.objective === null) return Number.POSITIVE_INFINITY + return (1 - slo.objective) * slo.windowDays * 24 * 60 * 60 * 1000 +} + +/** + * Error-budget policy state from the remaining fraction (0..1): + * - `freeze` — budget exhausted: freeze non-reliability deploys + * - `caution` — budget low (<25%): slow down, extra review + * - `ship` — budget healthy: ship normally + */ +export type BudgetPolicy = 'ship' | 'caution' | 'freeze' + +export function budgetPolicy(remaining: number): BudgetPolicy { + if (remaining <= 0) return 'freeze' + if (remaining < 0.25) return 'caution' + return 'ship' +} diff --git a/apps/cloud/src/server.ts b/apps/cloud/src/server.ts index b4cc6599f..7671d3f27 100644 --- a/apps/cloud/src/server.ts +++ b/apps/cloud/src/server.ts @@ -20,6 +20,7 @@ import { getCookie, setCookie, deleteCookie } from 'hono/cookie' import { WebhookSignatureError, type TenantBillingGateway } from './billing-gateway' import { renderClaimForm, renderClaimResult, renderDashboard } from './dashboard' import { MemoryDeviceGrantStore, isExpired, type DeviceGrantStore } from './device-grant' +import { fleetSummary, tenantSli, type HealthSampleStore } from './observability/health' import { SESSION_COOKIE, readSession, sealSession, type SessionData } from './session' export interface ControlPlaneAppDeps { @@ -29,6 +30,8 @@ export interface ControlPlaneAppDeps { payments?: TenantBillingGateway /** Device-grant store for the "claim your hub" flow. Defaults to in-memory. */ deviceGrants?: DeviceGrantStore + /** Fleet health samples (Phase 1 observability). If set, exposes /internal/fleet/health. */ + health?: HealthSampleStore /** Secret used to sign session cookies. If unset, the dashboard + auth callback are disabled. */ sessionSecret?: string /** Absolute origin for building checkout success/cancel URLs (e.g. https://cloud.xnet.fyi). */ @@ -318,6 +321,22 @@ export function createControlPlaneApp(deps: ControlPlaneAppDeps): Hono { } }) + // Fleet observability — per-tenant SLIs + an aggregate (exploration 0193). + app.get('/internal/fleet/health', async (c) => { + if (!requireInternal(c)) return c.json({ error: 'forbidden' }, 403) + if (!deps.health) return c.json({ error: 'observability_not_configured' }, 503) + const tenants = await deps.controlPlane.listTenants() + const live = tenants.filter((t) => t.dataTier === 'hot' && t.hubUrl) + const slis = live.map((t) => + tenantSli(deps.health!, { tenantId: t.tenantId, plan: t.plan, hubUrl: t.hubUrl }, now()) + ) + return c.json({ + fleet: fleetSummary(slis), + cold: tenants.length - live.length, + tenants: slis + }) + }) + app.post('/internal/account/recover', async (c) => { if (!requireInternal(c)) return c.json({ error: 'forbidden' }, 403) const body = (await c.req.json().catch(() => ({}))) as { billingUserId?: string } From 2ea523bc7d25d381a2087e1aaa72ab14e3748d76 Mon Sep 17 00:00:00 2001 From: xNet Test Date: Wed, 17 Jun 2026 10:59:43 -0700 Subject: [PATCH 3/6] =?UTF-8?q?feat(cloud):=20error-budget-gated=20canary?= =?UTF-8?q?=E2=86=92waves=20rollout=20engine=20(0193=20phase=203)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stages fleet upgrades on top of the one-step upgradeTenant. - rollout/engine.ts: rollWave (upgrade → measure post-bake availability → keep or instant-rollback by re-pointing to the prior immutable tag) and runRollout (budget gate → canary cohort → waves, abort on frozen budget or canary regression). Pure + deterministic (upgrade/priorVersion/measure injected). - rollout/control-plane-deps.ts: adapter wiring the engine to a real ControlPlane + HealthSampleStore (measure = tenantSli availability). 10 tests incl. a real ControlPlane + MemoryProvisioner run proving a healthy hub promotes and an unhealthy one rolls back to its original tag. Co-Authored-By: Claude Opus 4.8 --- apps/cloud/src/index.ts | 10 ++ apps/cloud/src/rollout/control-plane-deps.ts | 31 +++++ apps/cloud/src/rollout/engine.test.ts | 115 +++++++++++++++++++ apps/cloud/src/rollout/engine.ts | 106 +++++++++++++++++ 4 files changed, 262 insertions(+) create mode 100644 apps/cloud/src/rollout/control-plane-deps.ts create mode 100644 apps/cloud/src/rollout/engine.test.ts create mode 100644 apps/cloud/src/rollout/engine.ts diff --git a/apps/cloud/src/index.ts b/apps/cloud/src/index.ts index 71b11086b..11db2905f 100644 --- a/apps/cloud/src/index.ts +++ b/apps/cloud/src/index.ts @@ -69,6 +69,16 @@ export { type TenantSli, type FleetSummary } from './observability/health' +export { + rollWave, + runRollout, + type RolloutEngineDeps, + type RolloutPlan, + type RolloutReport, + type WaveResult, + type WaveOptions +} from './rollout/engine' +export { controlPlaneRolloutDeps } from './rollout/control-plane-deps' /** * Pick the billing identity provider from the environment. WorkOS AuthKit (free diff --git a/apps/cloud/src/rollout/control-plane-deps.ts b/apps/cloud/src/rollout/control-plane-deps.ts new file mode 100644 index 000000000..71049a167 --- /dev/null +++ b/apps/cloud/src/rollout/control-plane-deps.ts @@ -0,0 +1,31 @@ +/** + * Adapter: drive the pure rollout engine against a real ControlPlane + health + * store. `upgrade` wraps `upgradeTenant` (immutable tags), `priorVersion` captures + * the tenant's current tag for rollback, and `measure` reads the post-bake + * availability SLI from recorded health samples (exploration 0193). + */ + +import type { ControlPlane } from '../control-plane' +import type { RolloutEngineDeps } from './engine' +import { tenantSli, type HealthSampleStore } from '../observability/health' + +export function controlPlaneRolloutDeps( + cp: ControlPlane, + health: HealthSampleStore, + nowMs: () => number = Date.now +): RolloutEngineDeps { + return { + async upgrade(tenantId, target) { + await cp.upgradeTenant(tenantId, target) + }, + async priorVersion(tenantId) { + return (await cp.getTenant(tenantId))?.targetVersion ?? '' + }, + async measure(tenantId) { + const t = await cp.getTenant(tenantId) + if (!t) return 0 + return tenantSli(health, { tenantId: t.tenantId, plan: t.plan, hubUrl: t.hubUrl }, nowMs()) + .availability + } + } +} diff --git a/apps/cloud/src/rollout/engine.test.ts b/apps/cloud/src/rollout/engine.test.ts new file mode 100644 index 000000000..66380d011 --- /dev/null +++ b/apps/cloud/src/rollout/engine.test.ts @@ -0,0 +1,115 @@ +import { MemoryProvisioner } from '@xnetjs/cloud/provisioner' +import { describe, expect, it } from 'vitest' +import { buildControlPlane } from '../index' +import { HealthSampleStore } from '../observability/health' +import { controlPlaneRolloutDeps } from './control-plane-deps' +import { rollWave, runRollout, type RolloutEngineDeps } from './engine' + +/** A deterministic in-memory deps double that records every upgrade call. */ +function fakeDeps(availabilityByTenant: Record) { + const upgrades: { tenantId: string; target: string }[] = [] + const versions: Record = {} + const deps: RolloutEngineDeps = { + async upgrade(tenantId, target) { + upgrades.push({ tenantId, target }) + versions[tenantId] = target + }, + async priorVersion(tenantId) { + return versions[tenantId] ?? 'v0' + }, + async measure(tenantId) { + return availabilityByTenant[tenantId] ?? 1 + } + } + return { deps, upgrades, versions } +} + +const SHIP = { budgetPolicy: async () => 'ship' as const } +const FREEZE = { budgetPolicy: async () => 'freeze' as const } + +describe('rollWave', () => { + it('promotes healthy tenants and rolls back regressions', async () => { + const { deps, upgrades } = fakeDeps({ good: 1, bad: 0.5 }) + const res = await rollWave(deps, ['good', 'bad'], { target: 'v2', minAvailability: 0.95 }) + expect(res.promoted).toEqual(['good']) + expect(res.rolledBack).toEqual(['bad']) + // 'bad' was upgraded to v2 then rolled back to its prior (v0). + expect(upgrades.filter((u) => u.tenantId === 'bad')).toEqual([ + { tenantId: 'bad', target: 'v2' }, + { tenantId: 'bad', target: 'v0' } + ]) + }) +}) + +describe('runRollout', () => { + const plan = { + target: 'v2', + canary: ['c1'], + waves: [['w1', 'w2'], ['w3']], + minAvailability: 0.95 + } + + it('rolls canary then waves when healthy', async () => { + const { deps } = fakeDeps({ c1: 1, w1: 1, w2: 1, w3: 1 }) + const report = await runRollout(deps, plan, SHIP) + expect(report.aborted).toBe(false) + expect(report.canary?.promoted).toEqual(['c1']) + expect(report.waves.flatMap((w) => w.promoted)).toEqual(['w1', 'w2', 'w3']) + }) + + it('aborts before touching any tenant when the budget is frozen', async () => { + const { deps, upgrades } = fakeDeps({ c1: 1 }) + const report = await runRollout(deps, plan, FREEZE) + expect(report.aborted).toBe(true) + expect(report.reason).toMatch(/frozen/) + expect(upgrades).toHaveLength(0) + }) + + it('aborts the rollout when the canary regresses', async () => { + const { deps } = fakeDeps({ c1: 0.5, w1: 1 }) + const report = await runRollout(deps, plan, SHIP) + expect(report.aborted).toBe(true) + expect(report.reason).toMatch(/canary/) + expect(report.canary?.rolledBack).toEqual(['c1']) + expect(report.waves).toHaveLength(0) // never reached the waves + }) + + it('freezes mid-rollout if the budget burns during waves', async () => { + const { deps } = fakeDeps({ c1: 1, w1: 1, w2: 1, w3: 1 }) + let calls = 0 + const gate = { + budgetPolicy: async () => (++calls >= 3 ? ('freeze' as const) : ('ship' as const)) + } + const report = await runRollout(deps, plan, gate) + expect(report.aborted).toBe(true) + expect(report.reason).toMatch(/mid-rollout/) + }) +}) + +describe('rollout against a real ControlPlane + MemoryProvisioner', () => { + it('promotes a healthy hub and rolls back an unhealthy one', async () => { + const { controlPlane } = buildControlPlane({ provisioner: new MemoryProvisioner() }) + // Provision two hubs (DID-less billing path; default tag from buildControlPlane). + await controlPlane.provisionForBilling({ plan: 'community', billingUserId: 'healthy' }) + await controlPlane.provisionForBilling({ plan: 'community', billingUserId: 'sick' }) + const healthyId = 't_healthy' + const sickId = 't_sick' + const baseVersion = (await controlPlane.getTenant(healthyId))!.targetVersion + + const health = new HealthSampleStore() + for (let i = 0; i < 20; i++) { + health.record(healthyId, { ok: true, latencyMs: 10, atMs: 1000 + i }) + health.record(sickId, { ok: i % 2 === 0, latencyMs: 10, atMs: 1000 + i }) // 50% → unhealthy + } + const deps = controlPlaneRolloutDeps(controlPlane, health, () => 2000) + + const res = await rollWave(deps, [healthyId, sickId], { + target: 'xnet-hub@9.9.9', + minAvailability: 0.95 + }) + expect(res.promoted).toEqual([healthyId]) + expect(res.rolledBack).toEqual([sickId]) + expect((await controlPlane.getTenant(healthyId))!.targetVersion).toBe('xnet-hub@9.9.9') + expect((await controlPlane.getTenant(sickId))!.targetVersion).toBe(baseVersion) // rolled back + }) +}) diff --git a/apps/cloud/src/rollout/engine.ts b/apps/cloud/src/rollout/engine.ts new file mode 100644 index 000000000..9e7c43cb2 --- /dev/null +++ b/apps/cloud/src/rollout/engine.ts @@ -0,0 +1,106 @@ +/** + * xNet Cloud — error-budget-gated fleet rollout engine (exploration 0193). + * + * Builds staged rollouts on top of the one-step `ControlPlane.upgradeTenant`: + * a **canary cohort** bakes first, then **waves** roll out, each tenant's new + * image kept only if its post-bake availability holds — otherwise rolled back by + * re-pointing to its previous **immutable** tag (instant, no data movement). The + * whole rollout is **gated on the fleet error budget**: a frozen budget aborts + * remaining waves (the caller exempts security/reliability patches by not gating). + * + * Pure + deterministic: `upgrade`/`priorVersion`/`measure` are injected, so it's + * keyless-testable and also drives a real `ControlPlane` via the adapter below. + */ + +import type { BudgetPolicy } from '../observability/slo' + +export interface RolloutEngineDeps { + /** Apply the new image to a tenant (wraps ControlPlane.upgradeTenant). */ + upgrade(tenantId: string, target: string): Promise + /** The tenant's current pinned tag, captured before upgrade for rollback. */ + priorVersion(tenantId: string): Promise + /** Post-bake availability SLI (0..1) for a tenant. */ + measure(tenantId: string): Promise +} + +export interface WaveResult { + promoted: string[] + rolledBack: string[] +} + +export interface WaveOptions { + target: string + /** Keep the new image only if post-bake availability ≥ this; else roll back. */ + minAvailability: number +} + +/** Roll one wave: upgrade each tenant, measure, keep-or-rollback. */ +export async function rollWave( + deps: RolloutEngineDeps, + tenants: string[], + opts: WaveOptions +): Promise { + const promoted: string[] = [] + const rolledBack: string[] = [] + for (const id of tenants) { + const prior = await deps.priorVersion(id) + await deps.upgrade(id, opts.target) + const availability = await deps.measure(id) + if (availability < opts.minAvailability) { + if (prior && prior !== opts.target) await deps.upgrade(id, prior) // instant rollback + rolledBack.push(id) + } else { + promoted.push(id) + } + } + return { promoted, rolledBack } +} + +export interface RolloutPlan { + target: string + /** Lowest-risk cohort (xNet's own hubs + opt-in beta), rolled first. */ + canary: string[] + /** Ordered waves of tenant ids (e.g. by plan tier, riskiest last). */ + waves: string[][] + minAvailability: number + /** Abort the whole rollout if the canary rolls any tenant back (default true). */ + abortOnCanaryRollback?: boolean +} + +export interface RolloutReport { + aborted: boolean + reason?: string + canary?: WaveResult + waves: WaveResult[] +} + +/** + * Run a full rollout: gate on the fleet budget, bake the canary, then roll waves. + * Aborts (leaving already-promoted tenants in place) on a frozen budget or — when + * `abortOnCanaryRollback` — on any canary rollback. + */ +export async function runRollout( + deps: RolloutEngineDeps, + plan: RolloutPlan, + gate: { budgetPolicy: () => Promise } +): Promise { + const opts: WaveOptions = { target: plan.target, minAvailability: plan.minAvailability } + + if ((await gate.budgetPolicy()) === 'freeze') { + return { aborted: true, reason: 'error-budget frozen', waves: [] } + } + + const canary = await rollWave(deps, plan.canary, opts) + if ((plan.abortOnCanaryRollback ?? true) && canary.rolledBack.length > 0) { + return { aborted: true, reason: 'canary regressed', canary, waves: [] } + } + + const waves: WaveResult[] = [] + for (const wave of plan.waves) { + if ((await gate.budgetPolicy()) === 'freeze') { + return { aborted: true, reason: 'error-budget frozen mid-rollout', canary, waves } + } + waves.push(await rollWave(deps, wave, opts)) + } + return { aborted: false, canary, waves } +} From 34da4a1ca77a906bf6a16bf2b3f2be548e5b78a3 Mon Sep 17 00:00:00 2001 From: xNet Test Date: Wed, 17 Jun 2026 11:02:29 -0700 Subject: [PATCH 4/6] feat(cloud): restore-verification drill + reconcile decision (0193 phase 4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - backup/restore-drill.ts: verifyRestore provisions a THROWAWAY hub from a tenant's R2 replica, asserts /ready, and always tears it down — proving a backup actually restores (not just replicates). pickDrillSample rotates a nightly sample so the fleet is covered over time; runRestoreDrills batches it. - reconcile/reconcile.ts: the pure tenant reconciliation decision (none / reprovision / restart / demote) — the data-in/data-out core of a self-healing control loop; canceled subs stay suspended, demotion gated on replica sync. - control-plane: export single-sourced snapshotKeyFor (drill + control plane agree). 15 tests covering restore success/failure/provision-error, sample rotation, and every reconcile branch. Co-Authored-By: Claude Opus 4.8 --- apps/cloud/src/backup/restore-drill.test.ts | 95 +++++++++++++++++++++ apps/cloud/src/backup/restore-drill.ts | 86 +++++++++++++++++++ apps/cloud/src/control-plane.ts | 7 +- apps/cloud/src/index.ts | 8 ++ apps/cloud/src/reconcile/reconcile.test.ts | 60 +++++++++++++ apps/cloud/src/reconcile/reconcile.ts | 58 +++++++++++++ 6 files changed, 313 insertions(+), 1 deletion(-) create mode 100644 apps/cloud/src/backup/restore-drill.test.ts create mode 100644 apps/cloud/src/backup/restore-drill.ts create mode 100644 apps/cloud/src/reconcile/reconcile.test.ts create mode 100644 apps/cloud/src/reconcile/reconcile.ts diff --git a/apps/cloud/src/backup/restore-drill.test.ts b/apps/cloud/src/backup/restore-drill.test.ts new file mode 100644 index 000000000..a186367fd --- /dev/null +++ b/apps/cloud/src/backup/restore-drill.test.ts @@ -0,0 +1,95 @@ +import type { TenantRecord } from '../registry' +import { MemoryProvisioner } from '@xnetjs/cloud/provisioner' +import { resolveEntitlements } from '@xnetjs/entitlements' +import { describe, expect, it } from 'vitest' +import { + pickDrillSample, + runRestoreDrills, + verifyRestore, + type RestoreProbe +} from './restore-drill' + +const tenant = (id: string): TenantRecord => ({ + tenantId: id, + plan: 'personal', + entitlements: resolveEntitlements('personal'), + billingUserId: `u_${id}`, + did: '', + hubUrl: 'wss://x', + substrateRef: 'ref', + region: 'us', + targetVersion: 'xnet-hub@0.0.1', + createdAt: 0, + lastActiveMs: 0, + dataTier: 'cold' +}) + +const okProbe: RestoreProbe = { ready: async () => true } +const downProbe: RestoreProbe = { ready: async () => false } + +describe('verifyRestore', () => { + it('provisions a throwaway hub, asserts ready, and tears it down', async () => { + const prov = new MemoryProvisioner() + const destroyed: string[] = [] + const origDestroy = prov.destroy.bind(prov) + prov.destroy = async (ref: string) => { + destroyed.push(ref) + return origDestroy(ref) + } + const res = await verifyRestore(prov, okProbe, { + tenantId: 't_a', + entitlements: resolveEntitlements('personal'), + targetVersion: 'xnet-hub@0.0.1' + }) + expect(res).toEqual({ tenantId: 't_a', ok: true }) + expect(destroyed).toHaveLength(1) // throwaway hub always torn down + }) + + it('reports a not-ready restored hub as a failure', async () => { + const res = await verifyRestore(new MemoryProvisioner(), downProbe, { + tenantId: 't_b', + entitlements: resolveEntitlements('personal'), + targetVersion: 'xnet-hub@0.0.1' + }) + expect(res.ok).toBe(false) + expect(res.error).toMatch(/not ready/) + }) + + it('captures a provisioning failure instead of throwing', async () => { + const broken = { + substrate: 'broken', + provision: async () => { + throw new Error('R2 replica missing') + } + } as unknown as MemoryProvisioner + const res = await verifyRestore(broken, okProbe, { + tenantId: 't_c', + entitlements: resolveEntitlements('personal'), + targetVersion: 'xnet-hub@0.0.1' + }) + expect(res).toMatchObject({ tenantId: 't_c', ok: false, error: 'R2 replica missing' }) + }) +}) + +describe('pickDrillSample', () => { + it('returns all tenants when fewer than the sample size', () => { + const ts = [tenant('a'), tenant('b')] + expect(pickDrillSample(ts, 5, 0)).toHaveLength(2) + }) + + it('rotates the sample window by day so the fleet is covered over time', () => { + const ts = ['a', 'b', 'c', 'd'].map(tenant) + const day0 = pickDrillSample(ts, 2, 0).map((t) => t.tenantId) + const day1 = pickDrillSample(ts, 2, 1).map((t) => t.tenantId) + expect(day0).toEqual(['a', 'b']) + expect(day1).toEqual(['c', 'd']) + }) +}) + +describe('runRestoreDrills', () => { + it('runs the drill across a sample and includes failures', async () => { + const prov = new MemoryProvisioner() + const results = await runRestoreDrills(prov, okProbe, [tenant('a'), tenant('b')]) + expect(results.map((r) => r.ok)).toEqual([true, true]) + }) +}) diff --git a/apps/cloud/src/backup/restore-drill.ts b/apps/cloud/src/backup/restore-drill.ts new file mode 100644 index 000000000..ef17ae99f --- /dev/null +++ b/apps/cloud/src/backup/restore-drill.ts @@ -0,0 +1,86 @@ +/** + * xNet Cloud — automated restore-verification drill (exploration 0193). + * + * "We replicate to R2" is not "we can restore your hub." This drill *proves* it: + * provision a THROWAWAY hub that restores a tenant's DB from its R2 replica + * (Litestream restore-on-boot), assert it comes up ready, then always tear it + * down. Run nightly over a rotating sample so it costs little and catches a + * broken backup before a real reactivation does. + */ + +import type { TenantRecord } from './../registry' +import type { Provisioner } from '@xnetjs/cloud/provisioner' +import type { PlanEntitlements } from '@xnetjs/entitlements' +import { snapshotKeyFor } from '../control-plane' + +/** Probes whether a freshly-restored hub is up + writable (`GET /ready`). */ +export interface RestoreProbe { + ready(hubUrl: string): Promise +} + +export interface RestoreDrillResult { + tenantId: string + ok: boolean + error?: string +} + +/** Verify one tenant restores from R2 into a throwaway hub, then tear it down. */ +export async function verifyRestore( + provisioner: Provisioner, + probe: RestoreProbe, + tenant: { tenantId: string; entitlements: PlanEntitlements; targetVersion: string } +): Promise { + let substrateRef: string | null = null + try { + const handle = await provisioner.provision({ + tenantId: `drill-${tenant.tenantId}`, + entitlements: tenant.entitlements, + targetVersion: tenant.targetVersion, + env: {}, + restoreFromR2: snapshotKeyFor(tenant.tenantId) + }) + substrateRef = handle.substrateRef + const ok = await probe.ready(handle.hubUrl) + return { tenantId: tenant.tenantId, ok, ...(ok ? {} : { error: 'restored hub not ready' }) } + } catch (err) { + return { tenantId: tenant.tenantId, ok: false, error: (err as Error).message } + } finally { + if (substrateRef) await provisioner.destroy(substrateRef).catch(() => undefined) + } +} + +/** + * Deterministically pick `sampleSize` tenants for tonight's drill, rotating by a + * day index so the whole fleet is covered over time without drilling all of it + * every night (a silent cap is logged by the caller — see exploration 0193). + */ +export function pickDrillSample( + tenants: TenantRecord[], + sampleSize: number, + dayIndex: number +): TenantRecord[] { + const eligible = tenants.filter((t) => t.tenantId) // every tenant has an R2 replica path + if (eligible.length <= sampleSize) return eligible + const start = (dayIndex * sampleSize) % eligible.length + const rotated = [...eligible.slice(start), ...eligible.slice(0, start)] + return rotated.slice(0, sampleSize) +} + +/** Run the drill across a sample; returns per-tenant results (failures included). */ +export async function runRestoreDrills( + provisioner: Provisioner, + probe: RestoreProbe, + sample: TenantRecord[] +): Promise { + const results: RestoreDrillResult[] = [] + for (const t of sample) { + results.push( + await verifyRestore(provisioner, probe, { + tenantId: t.tenantId, + entitlements: t.entitlements, + targetVersion: t.targetVersion + }) + ) + } + return results +} diff --git a/apps/cloud/src/control-plane.ts b/apps/cloud/src/control-plane.ts index 7ef651237..0b6f00b75 100644 --- a/apps/cloud/src/control-plane.ts +++ b/apps/cloud/src/control-plane.ts @@ -63,6 +63,11 @@ export function tenantIdForBilling(billingUserId: string): string { return `t_${billingUserId.replace(/[^a-zA-Z0-9_-]/g, '')}` } +/** R2 object path holding a tenant's SQLite snapshot (matches the Litestream replica path). */ +export function snapshotKeyFor(tenantId: string): string { + return `t/${tenantId}/db` +} + export class ControlPlane { constructor(private readonly deps: ControlPlaneDeps) {} @@ -306,7 +311,7 @@ export class ControlPlane { /** R2 object path holding a tenant's SQLite snapshot (matches the Litestream replica path). */ private snapshotKey(tenantId: string): string { - return `t/${tenantId}/db` + return snapshotKeyFor(tenantId) } /** Record activity so the cold-demotion clock resets (exploration 0178). */ diff --git a/apps/cloud/src/index.ts b/apps/cloud/src/index.ts index 11db2905f..2b363847e 100644 --- a/apps/cloud/src/index.ts +++ b/apps/cloud/src/index.ts @@ -79,6 +79,14 @@ export { type WaveOptions } from './rollout/engine' export { controlPlaneRolloutDeps } from './rollout/control-plane-deps' +export { + verifyRestore, + runRestoreDrills, + pickDrillSample, + type RestoreProbe, + type RestoreDrillResult +} from './backup/restore-drill' +export { reconcileTenant, type ReconcileInput, type ReconcileAction } from './reconcile/reconcile' /** * Pick the billing identity provider from the environment. WorkOS AuthKit (free diff --git a/apps/cloud/src/reconcile/reconcile.test.ts b/apps/cloud/src/reconcile/reconcile.test.ts new file mode 100644 index 000000000..380705b25 --- /dev/null +++ b/apps/cloud/src/reconcile/reconcile.test.ts @@ -0,0 +1,60 @@ +import { describe, expect, it } from 'vitest' +import { reconcileTenant, type ReconcileInput } from './reconcile' + +const base: ReconcileInput = { + dataTier: 'hot', + substrateRef: 'ref', + hubUrl: 'wss://x', + lastActiveMs: 1000, + healthy: true, + synced: true, + nowMs: 1000, + coldAfterMs: 60_000 +} + +describe('reconcileTenant', () => { + it('does nothing for a healthy, active, live tenant', () => { + expect(reconcileTenant(base)).toEqual({ kind: 'none' }) + }) + + it('re-provisions a hot tenant that has no live machine', () => { + expect(reconcileTenant({ ...base, substrateRef: '', hubUrl: '' }).kind).toBe('reprovision') + }) + + it('restarts an unhealthy live hub', () => { + expect(reconcileTenant({ ...base, healthy: false }).kind).toBe('restart') + }) + + it('demotes an idle, fully-synced hub', () => { + const r = reconcileTenant({ ...base, lastActiveMs: 0, nowMs: 120_000, synced: true }) + expect(r.kind).toBe('demote') + }) + + it('does NOT demote an idle hub whose replica is not yet synced', () => { + const r = reconcileTenant({ ...base, lastActiveMs: 0, nowMs: 120_000, synced: false }) + expect(r).toEqual({ kind: 'none' }) + }) + + it('prioritizes re-provision over restart when both could apply', () => { + // No machine AND unhealthy → re-provision (can't restart what isn't there). + expect(reconcileTenant({ ...base, substrateRef: '', hubUrl: '', healthy: false }).kind).toBe( + 'reprovision' + ) + }) + + it('leaves a canceled subscription suspended', () => { + expect( + reconcileTenant({ ...base, subscriptionStatus: 'canceled', substrateRef: '', hubUrl: '' }) + ).toEqual({ kind: 'none' }) + }) + + it('does nothing for cold tenants (reactivation is request-driven)', () => { + expect(reconcileTenant({ ...base, dataTier: 'cold', substrateRef: '', hubUrl: '' })).toEqual({ + kind: 'none' + }) + }) + + it('takes no action on unknown health (no signal yet)', () => { + expect(reconcileTenant({ ...base, healthy: null })).toEqual({ kind: 'none' }) + }) +}) diff --git a/apps/cloud/src/reconcile/reconcile.ts b/apps/cloud/src/reconcile/reconcile.ts new file mode 100644 index 000000000..2aa988e36 --- /dev/null +++ b/apps/cloud/src/reconcile/reconcile.ts @@ -0,0 +1,58 @@ +/** + * xNet Cloud — tenant reconciliation decision (exploration 0193). + * + * The control plane is happiest as a reconciliation loop: compare each tenant's + * desired state to what's observed and emit the single next action that converges + * them. This module is the *pure decision* — no I/O — so it's exhaustively + * testable; a thin driver maps the action to ControlPlane calls (provision / + * upgrade / suspend / demote / restart). Keeping it data-in/data-out is what makes + * "automate as much as possible, keep it simple" tractable. + */ + +export interface ReconcileInput { + dataTier: 'hot' | 'cold' + /** Opaque live-machine ref; empty when there is no running hub. */ + substrateRef: string + hubUrl: string + lastActiveMs: number + subscriptionStatus?: 'active' | 'canceled' + /** Latest health verdict: true/false, or null when there's no signal yet. */ + healthy: boolean | null + /** Whether the R2 replica is caught up (gate for safe demotion). */ + synced: boolean + nowMs: number + /** Idle duration after which a hot tenant should demote to cold. */ + coldAfterMs: number +} + +export type ReconcileAction = + | { kind: 'none' } + | { kind: 'reprovision'; reason: string } + | { kind: 'restart'; reason: string } + | { kind: 'demote'; reason: string } + +/** + * Decide the next convergence action for one tenant. Order matters: a canceled + * subscription is left suspended; a hot tenant with no live machine is re-provisioned + * (crash/loss); an unhealthy live hub is restarted; an idle, fully-synced hub is + * demoted to cold; otherwise nothing to do. + */ +export function reconcileTenant(input: ReconcileInput): ReconcileAction { + if (input.subscriptionStatus === 'canceled') return { kind: 'none' } + + if (input.dataTier === 'hot') { + if (!input.substrateRef || !input.hubUrl) { + return { kind: 'reprovision', reason: 'hot tenant has no live hub' } + } + if (input.healthy === false) { + return { kind: 'restart', reason: 'health probe failing' } + } + if (input.nowMs - input.lastActiveMs >= input.coldAfterMs && input.synced) { + return { kind: 'demote', reason: 'idle and fully synced' } + } + return { kind: 'none' } + } + + // Cold tenants reactivate on a real request (not from the reconciler). + return { kind: 'none' } +} From 517370e7aa6ec1fe400c62fb2b9e6f1123b48efe Mon Sep 17 00:00:00 2001 From: xNet Test Date: Wed, 17 Jun 2026 11:04:10 -0700 Subject: [PATCH 5/6] feat(cloud): surface uptime SLO + backups on the dashboard (0193) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The tenant dashboard's hub card now shows its plan's uptime commitment (sloForPlan label) and a continuous-backups line — making the SLA work visible to the user who's paying for it. Co-Authored-By: Claude Opus 4.8 --- apps/cloud/src/dashboard.ts | 3 +++ 1 file changed, 3 insertions(+) diff --git a/apps/cloud/src/dashboard.ts b/apps/cloud/src/dashboard.ts index 66383cbfb..0fe996d5e 100644 --- a/apps/cloud/src/dashboard.ts +++ b/apps/cloud/src/dashboard.ts @@ -10,6 +10,7 @@ import type { TenantRecord } from './registry' import type { PlanId } from '@xnetjs/entitlements' +import { sloForPlan } from './observability/slo' export interface DashboardView { billingUserId: string @@ -69,6 +70,8 @@ function hubCard(tenant: TenantRecord): string {
Region
${esc(tenant.region || 'auto')}
Storage
${fmtBytes(e.quotaBytes)}
Seats
${e.seats}
+
Uptime
${esc(sloForPlan(tenant.plan).label)}
+
Backups
Continuous → object storage
Data identity
${ tenant.did ? `${esc(tenant.did)}` From 48bb800f834e177786f8373a54838eef0ff8f01c Mon Sep 17 00:00:00 2001 From: xNet Test Date: Wed, 17 Jun 2026 11:05:18 -0700 Subject: [PATCH 6/6] docs(exploration): check off 0193 implemented items (phases 1-4) Marks the implemented control-plane logic done and adds an Implementation Status note separating it from deferred infra (real provisioner, R2 lifecycle, DAP/Prio, status page, durable stores, deploy). Co-Authored-By: Claude Opus 4.8 --- ...OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md | 69 +++++++++++-------- 1 file changed, 42 insertions(+), 27 deletions(-) diff --git a/docs/explorations/0193_[_]_XNET_CLOUD_OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md b/docs/explorations/0193_[_]_XNET_CLOUD_OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md index f5bfa9039..8671678b6 100644 --- a/docs/explorations/0193_[_]_XNET_CLOUD_OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md +++ b/docs/explorations/0193_[_]_XNET_CLOUD_OPERATIONS_UPTIME_BACKUPS_AND_TELEMETRY.md @@ -547,45 +547,60 @@ export async function verifyRestore(cp: ControlPlane, p: Provisioner, tenantId: composition; a single aggregate status is safe but coarse. Likely: aggregate public status + per-tenant health in the authenticated dashboard. +## Implementation Status + +The **testable control-plane logic** for all four phases is implemented in +`apps/cloud` with 36 new tests (port + fake + tests, matching the 0192/0176 +pattern): SLI/SLO/error-budget math, the health-sample store + admin fleet route, +the error-budget-gated canary→waves rollout engine with auto-rollback, the +restore-verification drill, and the pure reconcile decision. Items that are pure +**infrastructure / external configuration / a second operator** — the real +`CloudRunLitestreamProvisioner`, R2 lifecycle IaC, a DAP/Prio aggregator pair, a +hosted status page, durable stores, chaos tests, and live deploy — are left +unchecked; they are not implementable or verifiable in this repo. (Also: the +SLO mapping follows the **shipped** catalog — `community`/`company` = 99.9%, +`enterprise` = custom, others best-effort; raising `team` to 99.9% is a product +decision, not done here.) + ## Implementation Checklist **Phase 1 — Fleet observability (ops plane):** -- [ ] Control-plane poller for each hot tenant's `/health` + `/ready`; record `HealthSample`s. -- [ ] Consume the hub `telemetry-bridge` ops events centrally (enable it for managed hubs). -- [ ] Per-tenant SLI store: availability, latency buckets, error rate, backup freshness (`isReplicaFresh`), wake latency. -- [ ] Operator fleet dashboard + a public aggregate status page. -- [ ] Privacy-policy + dashboard copy: what ops telemetry we collect and what we never collect. +- [x] Health-sample model + bounded per-tenant store + `httpHealthProbe`/`FakeHealthProbe` (`observability/health.ts`). +- [ ] Wire a live poller loop + consume the hub `telemetry-bridge` centrally for managed hubs. *(deferred — runtime/deploy)* +- [x] Per-tenant SLI store + summary: availability, error rate, p95 latency, backup freshness, error budget (`observability/sli.ts` + `health.ts`); admin `GET /internal/fleet/health`. +- [ ] Operator fleet dashboard + public aggregate status page. *(deferred — hosting)* +- [x] Tenant dashboard surfaces its uptime SLO + continuous-backups line; privacy framing documented (three-plane model). **Phase 2 — SLAs (measure → promise → protect):** -- [ ] Per-tier SLO table (warm tiers 99.9% availability; sleep tiers wake-success; enterprise custom). -- [ ] Error-budget computation per tenant + fleet; alerting on burn rate. -- [ ] Error-budget policy (freeze risky deploys when exhausted; exempt security/reliability). -- [ ] Enterprise contractual credits hook. +- [x] Per-tier SLO catalog from the shipped `SlaLevel` + `errorBudgetMs` (`observability/slo.ts`). +- [x] Error-budget computation per tenant + `fleetSummary` aggregate + burn rate. +- [x] Error-budget policy (`budgetPolicy` ship/caution/freeze) gating the rollout engine; security/reliability exemption is a caller choice. +- [ ] Enterprise contractual credits hook. *(deferred — billing/legal)* **Phase 3 — Upgrade engine (automation):** -- [ ] Rollout engine over `upgradeTenant`: canary cohort → waves by tier, desired-state driven. -- [ ] SLI bake + automatic rollback (re-point to previous pinned immutable tag). -- [ ] Gate each wave on remaining error budget; record rollout state for restartability. -- [ ] Implement `CloudRunLitestreamProvisioner.upgrade` (unblocks real rollouts). +- [x] Rollout engine over `upgradeTenant`: canary cohort → waves (`rollout/engine.ts`). +- [x] SLI bake + automatic rollback (re-point to the previous pinned immutable tag). +- [x] Gate the rollout on the fleet error budget (abort on frozen / canary regression); `controlPlaneRolloutDeps` adapter. +- [ ] Implement `CloudRunLitestreamProvisioner.upgrade` (unblocks real rollouts). *(deferred — substrate)* **Phase 4 — Backups proven + product learning:** -- [ ] Automated nightly **restore-verification drill** over a rotating sample; alert on failure. -- [ ] R2 retention / point-in-time-recovery lifecycle policy (e.g. 30-day PITR + 90-day daily snapshots). -- [ ] Backup-freshness alert wired to the SLI store; exercise the single-writer S3 lease (chaos test). -- [ ] Route consent-gated client usage egress through **DAP/Prio** for cross-tenant aggregates (Plane 2). -- [ ] Reconciliation loop: converge desired vs. actual (provision/upgrade/demote/self-heal/restart-unhealthy). +- [x] `verifyRestore` (provision throwaway from R2 → assert `/ready` → tear down) + `pickDrillSample` rotation + `runRestoreDrills` (`backup/restore-drill.ts`). +- [ ] R2 retention / point-in-time-recovery lifecycle policy. *(deferred — IaC)* +- [x] Backup-freshness SLI (`backupHealthy`); exercising the single-writer S3 lease (chaos test) *(deferred — substrate)*. +- [ ] Route consent-gated usage egress through DAP/Prio for cross-tenant aggregates. *(deferred — needs a second non-colluding operator)* +- [x] Pure reconcile decision (`reconcile/reconcile.ts`: none/reprovision/restart/demote); wiring it to a live loop is deferred (runtime). ## Validation Checklist -- [ ] A bad hub image is caught in the canary cohort and auto-rolled-back before any paying wave is touched. -- [ ] A tenant's availability/latency/error SLIs are visible per-tenant and in aggregate, derived from real health signal. -- [ ] An exhausted error budget freezes feature rollouts but not a security patch. -- [ ] The nightly restore drill provisions a throwaway hub from R2 and passes `/ready`; a deliberately corrupted replica trips the alert. -- [ ] Measured RPO ≤ 1 s on hard kill and ~0 on graceful drain; measured cold wake latency meets the sleep-tier wake-success SLO. -- [ ] The public status page reflects a real induced incident; per-tenant health shows only in the authenticated dashboard. -- [ ] An auditor can confirm the operator never receives document content or plaintext identifiers — only hashed DIDs, buckets, and ops signal. -- [ ] Cross-tenant product metrics (Plane 2) reveal aggregates only; no single tenant's usage is recoverable from the aggregator output. -- [ ] A control-plane restart resumes an in-flight rollout from recorded desired-state without double-upgrading. +- [x] A bad hub image is caught in the canary cohort and auto-rolled-back before any wave is touched (`rollout/engine.test.ts`, incl. a real ControlPlane + MemoryProvisioner run). +- [x] Per-tenant + aggregate SLIs are derived from health samples and served at `/internal/fleet/health` (`fleet.test.ts`). +- [x] An exhausted error budget aborts the rollout; the gate is bypassable for exempt patches (`engine.test.ts`). +- [x] The restore drill provisions a throwaway hub, asserts ready, always tears down, and a provisioning failure surfaces as a failed result (`restore-drill.test.ts`). +- [ ] Measured RPO ≤ 1 s / cold wake latency against a **real** deploy. *(deferred — needs the substrate)* +- [ ] A public status page reflects a real induced incident. *(deferred — hosting)* +- [x] All collected signal is content-free: SLIs are ok/latency probes, never document data (respects the E2E boundary by construction). +- [ ] Cross-tenant product metrics reveal aggregates only (DAP/Prio). *(deferred)* +- [ ] A control-plane restart resumes an in-flight rollout from durable desired-state. *(deferred — durable stores)* ## References