feat(sandbox): tenant warm pools — pods already running the dev server

[pedrofrxncx]

A member of an opted-in org opens a project and the dev server is already
up — no clone, no install, no Vite boot. Spec (updated to match what shipped):
packages/sandbox/tenant-warm-pools-spec.md.

A pool is keyed (org, repo). Studio bootstraps its pods while they are
unbound; a claim from that org binds one instead of a generic empty warm pod.
Off everywhere by default: no pool configured → not a single code path changes.

What's in it

Provider (packages/sandbox/server/provider/agent-sandbox/)

• tenant-pools.ts — pool config (STUDIO_SANDBOX_TENANT_POOLS, a JSON array)
  and resolution. The pool is resolved from the authenticated user's org,
  never from a request field: the operator has no notion of a tenant and binds
  whatever pool a claim names, so the server-built claim is the boundary.
• runner.ts — the claim swaps warmpool to the pool's name; a reconciler
  (60s, no-op with no pools) clones/installs/starts each unbound pool pod and
  refreshes it every 30 min. That refresh is the clone-credential refresh,
  so no pod outlives its ~1h token. Bound pods are never touched.
• Runs on every replica with no leader election: the daemon classifies each
  config post itself and its setup queue collapses concurrent requests — so
  duplicated effort, never duplicated clones.

Daemon (daemon-go)

• claimed now flips only for a config carrying a user identity. Without
  this, bootstrapping a pool pod makes the housekeeper reap it 15 min later —
  warming a pool would be what kills it. Unbound pods report
  claimed:false, prewarmed:true.
• env-change now maps to a dev-server restart. It was a silent no-op: the
  merged env only reached the dev server if some other transition happened to
  restart it.

Chart — tenantPools: renders a SandboxWarmPool per entry against the
shared SandboxTemplate.

Webhook (optional) — POST /api/_github/webhook makes a refresh immediate
on a push. 503 without GITHUB_WEBHOOK_SECRET; nothing here needs it —
pools refresh on their own schedule regardless.

Testing

Unit: pool parsing/resolution (including "a user of another org never resolves
this pool"), webhook HMAC. Daemon e2e (real binary, black-box):
identity-less bootstrap stays unclaimed, an identity-bearing config claims,
env-only change restarts dev. Full daemon-e2e green (209/209).

End-to-end on a local rancher-desktop cluster (real operator, real pods, real
daemon built from this branch):

1. reconciler bootstraps the pool pod → repo cloned;
2. it reports claimed:false, prewarmed:true (housekeeper-safe);
3. a claim from the pool's org gets warmpool: tenant-local-hello and binds
   that pod; claiming flips claimed:true;
4. releasing the claim destroys the used pod (never reused — it holds the
   previous user's credential, identity and working tree) and the pool refills;
   the refill self-warms;
5. a signed push payload through the webhook handler refreshes the warm pod,
   and a push on another branch/repo matches nothing.

Notes

• One template, not two: the claim swaps only warmpool. A per-pool template
  exists solely to carry per-pool node placement (on-demand / do-not-disrupt),
  which this drops — reinstate both together.
• connectionId is optional; without it pods clone anonymously (public repos).
• Every path still works at pool size 0 — the pool is an optimization, never a
  dependency.

---

Summary by cubic

Adds tenant warm pools so members of opted-in orgs open a project with the dev server already running. Pools are per org/repo, bootstrapped by Studio, scoped per release, and off by default.

• New Features

  • Tenant warm pools resolved from the authenticated user's org; Studio bootstraps unbound pods (clone/install/start) and refreshes them every 30 min. Claims bind a warm pod by setting warmpool to the pool name; no pools configured → no behavior change. Added studio.sandbox.pool.pods gauge (ready/bound/pending/failed).
  • Server reads pools from STUDIO_SANDBOX_TENANT_POOLS (strict parser); when set, the sandbox provider starts eagerly at boot so pods pre-warm before the first request.
  • Optional GitHub push webhook POST /api/_github/webhook (HMAC via GITHUB_WEBHOOK_SECRET) marks pools dirty to refresh immediately; periodic refresh still runs without it.
  • Helm chart: tenantPools: renders a SandboxWarmPool per entry against the shared SandboxTemplate (chart v0.12.0). Names must end with -<envName> and size must be ≥1. Server reads pools from STUDIO_SANDBOX_TENANT_POOLS.

• Bug Fixes

  • Runner Role grants sandboxwarmpools: get; the reconciler re-checks a pod is unbound immediately before config and clone, treating unreadable pods as bound to avoid hard resets on live working trees.
  • Failure cap now backs off 30 minutes and retries once, preventing a transient failure from permanently blackholing a slot.
  • Daemon sets claimed:true only when a user identity is present; warm pool pods report claimed:false, prewarmed:true. env-change now restarts the dev server.
  • Housekeeper sweep fixes claimed parsing across sed variants and adds tests for probe classification; added tests for tenant-pool binding, push matching, and daemon e2e (branch change keeps deps; different-repo config is refused).

^{Written for commit 785580f. Summary will update on new commits.}