feat: xNet Cloud operations — SLIs/SLOs, error-budget rollouts, restore drill (0193) - #146
Merged
Conversation
… telemetry, SLAs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turns the declared-but-unenforced SlaLevel into measured observability. - observability/sli.ts: pure SLI math — availability (cold-start waits count as valid), error rate, latency percentile, error-budget-remaining, burn rate, backup freshness (reuses the shipped isReplicaFresh) - observability/slo.ts: maps each plan's SlaLevel → a measurable SLO (community/ company 99.9%, enterprise 99.95%, others best-effort) + errorBudgetMs + Google-SRE budget policy (ship/caution/freeze) - observability/health.ts: HealthProbe port + FakeHealthProbe + httpHealthProbe, a bounded per-tenant sample ring, tenantSli + fleetSummary - control-plane.listTenants(); admin-gated GET /internal/fleet/health returns per-tenant SLIs + a fleet aggregate (503 when observability unconfigured) All content-free + tenant-scoped (respects the E2E boundary). 18 new tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…se 3) Stages fleet upgrades on top of the one-step upgradeTenant. - rollout/engine.ts: rollWave (upgrade → measure post-bake availability → keep or instant-rollback by re-pointing to the prior immutable tag) and runRollout (budget gate → canary cohort → waves, abort on frozen budget or canary regression). Pure + deterministic (upgrade/priorVersion/measure injected). - rollout/control-plane-deps.ts: adapter wiring the engine to a real ControlPlane + HealthSampleStore (measure = tenantSli availability). 10 tests incl. a real ControlPlane + MemoryProvisioner run proving a healthy hub promotes and an unhealthy one rolls back to its original tag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ase 4) - backup/restore-drill.ts: verifyRestore provisions a THROWAWAY hub from a tenant's R2 replica, asserts /ready, and always tears it down — proving a backup actually restores (not just replicates). pickDrillSample rotates a nightly sample so the fleet is covered over time; runRestoreDrills batches it. - reconcile/reconcile.ts: the pure tenant reconciliation decision (none / reprovision / restart / demote) — the data-in/data-out core of a self-healing control loop; canceled subs stay suspended, demotion gated on replica sync. - control-plane: export single-sourced snapshotKeyFor (drill + control plane agree). 15 tests covering restore success/failure/provision-error, sample rotation, and every reconcile branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The tenant dashboard's hub card now shows its plan's uptime commitment (sloForPlan label) and a continuous-backups line — making the SLA work visible to the user who's paying for it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Marks the implemented control-plane logic done and adds an Implementation Status note separating it from deferred infra (real provisioner, R2 lifecycle, DAP/Prio, status page, durable stores, deploy). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
|
Preview removed for PR #146. |
crs48
added a commit
that referenced
this pull request
Jun 18, 2026
…193/0194) (#201) The `changelog-section` required check was only added in #164, so the 0192/0193/0194 feature batch merged before it never got changelog entries. This backfills the 12 user-facing features that were missing, each dated to its merge day with its PR number and contributors: | PR | Entry | |----|-------| | #138 | A safer foundation for plugins | | #145 | One trust model across plugins and Labs | | #146 | Reliability you can see for managed hubs | | #148 | Drive your own coding agent from xNet | | #149 | AI that can act on your workspace | | #150 | Your Labs become AI tools | | #152 | Your agent can use your workspace | | #154 | AI edits, right inside the editor | | #155 | An agentic dev loop in your terminal | | #158 | Review AI edits before they apply | | #159 | Kick off agentic code tasks from xNet | | #162 | Plugins run on the Labs runtime | Skipped: PRs already covered by umbrella entries (#142 plugin ecosystem, #144 extensibility fabric, #147 automated changelog, #163 agent panel, #180 changelog gallery), internal-only changes (#139 schema authz — zero user-facing effect), and meta/test/docs PRs. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements docs/explorations/0193 — the operational spine for the managed fleet (upgrades, backups, telemetry, SLAs). Turns the declared-but-unenforced
SlaLevelinto measured observability and connects it to upgrades via error budgets.The central reframe: xNet's privacy-preserving telemetry (0187/0190) already respects the E2E boundary, so this PR is about operationalizing it for the fleet — everything collected here is content-free and tenant-scoped (ok/latency probes, never document data).
Phase 1+2 — SLIs / SLOs / error budgets (
observability/)sli.ts: availability (cold-start waits count as valid), error rate, p95 latency, error-budget-remaining, burn rate, backup freshness (reuses shippedisReplicaFresh).slo.ts: maps each plan'sSlaLevel→ a measurable SLO (community/company 99.9%, enterprise 99.95%, others best-effort) +errorBudgetMs+ Google-SREbudgetPolicy(ship/caution/freeze).health.ts:HealthProbeport +Fake/httpimpls, bounded per-tenant sample ring,tenantSli+fleetSummary.GET /internal/fleet/healthreturns per-tenant SLIs + a fleet aggregate.Phase 3 — Error-budget-gated rollout engine (
rollout/)engine.ts:rollWave(upgrade → measure post-bake availability → keep or instant-rollback by re-pointing to the prior immutable tag) andrunRollout(budget gate → canary cohort → waves, abort on frozen budget or canary regression).control-plane-deps.ts: adapter driving it against a realControlPlane+ health store.Phase 4 — Backups proven + self-healing (
backup/,reconcile/)restore-drill.ts:verifyRestoreprovisions a throwaway hub from R2, asserts/ready, always tears down — proving a backup restores, not just replicates;pickDrillSamplerotates nightly coverage.reconcile.ts: the pure tenant reconciliation decision (none/reprovision/restart/demote) — the data-in/data-out core of a self-healing loop.Verification
apps/cloudtests pass (36 new), incl. a realControlPlane+MemoryProvisionerrollout that promotes a healthy hub and rolls back an unhealthy one.apps/cloudtypecheck + eslint + prettier clean; dashboard screenshotted.Deferred (pure infra / external — see the doc's Implementation Status)
Real
CloudRunLitestreamProvisioner, R2 lifecycle/PITR IaC, a DAP/Prio aggregator pair (needs a second non-colluding operator), a hosted status page, durable stores, chaos tests, and live deploy. The live poller/reconcile loops are wired to tested pure cores but not run as daemons here.🤖 Generated with Claude Code