fix(cloud): gate provisioning health at enqueue

[lalalune]

Summary

• moves the provisioning-worker health check to the cold provision-job enqueue boundary, after idempotent reuse, shared-tier, non-eager, and warm-pool paths have had a chance to succeed
• rolls back the just-created pending sandbox row before returning the canonical worker-unhealthy 503 body
• removes the now-unused advisory reuse peek helper from cloud-shared and stale mocks from the sibling route tests

This preserves #15541's source-side fix for #15516 while taking the stronger enqueue-boundary placement from the now-conflicting #15542 shape.

Verification

• bun test packages/cloud/api/__tests__/eliza-agents-create-idempotency.test.ts -> 12 pass
• bun test packages/cloud/api/__tests__/eliza-agents-warm-pool-empty-on-claim.test.ts -> 2 pass
• node packages/app-core/scripts/ensure-shared-i18n-data.mjs
• bun -c .tmp-bunfig-no-coverage.toml test ./packages/cloud/api/__tests__/eliza-agents-orphan-cleanup.test.ts -> exit 0; temp config removed after run; no repo diff left
• bunx @biomejs/biome check packages/cloud/api/v1/eliza/agents/route.ts packages/cloud/api/__tests__/eliza-agents-create-idempotency.test.ts packages/cloud/api/__tests__/eliza-agents-warm-pool-empty-on-claim.test.ts packages/cloud/api/__tests__/eliza-agents-orphan-cleanup.test.ts packages/cloud/shared/src/lib/services/eliza-sandbox.ts packages/cloud/shared/src/db/repositories/agent-sandboxes.ts --no-errors-on-unmatched
• git diff --check
• bun run audit:error-policy-ratchet

Package typecheck caveat: bun run --cwd packages/cloud/api typecheck still fails on pre-existing unrelated errors in __tests__/stripe-connect-webhook-route.test.ts (2026-06-24.dahlia vs 2026-05-27.dahlia) and fal/proxy/route.ts Hono 4.12.18/4.12.27 type mismatches; no touched file appears in that failure.

Evidence rows

• N/A - backend route ordering only; no rendered UI change.

• N/A - backend route ordering only; no rendered UI change.

• N/A - no UI flow changed in this source patch; device re-verification remains tracked on LP3 QA (develop tip): onboarding ChoiceWidget white-on-white + native cloud onboarding 503-dead-ends despite a running agent #15516.

• Focused route tests assert the exact backend responses and repository calls: reuse returns 200 without worker-health check; worker-unhealthy cold create returns 503 and deletes the pending row.

• N/A - no frontend code path changed.

• N/A - no model/action/provider/prompt behavior change.

• Route tests assert the domain artifact behavior: no provision job is enqueued during worker outage, and the pending sandbox row is rolled back before returning 503.

Fixes #15516 server-side residual; supersedes the conflicting server-half shape in #15542.

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[NubsCarson]

[qa-agent] post-merge adversarial review — CLEAN, one fast-follow race noted

POST-MERGE ADVERSARIAL REVIEW — #15542 / #15544 / #15548

Deployment-state facts (read first)

• fix(cloud): provisioning health gate after idempotent reuse — worker blips no longer 503 onboarding (#15516 Bug B server half) #15542 is CLOSED, never merged. fix(cloud): gate provisioning health at enqueue #15544 superseded it (same enqueue-boundary shape, rebased over fix(cloud): reuse the existing agent in first-run onboarding instead of 503 dead-end during worker outage #15541). Only fix(cloud): gate provisioning health at enqueue #15544 (e3002e941) and fix(ci): scrub deprecated bridge fallback on provisioning deploy #15548 (d4880be0f) are in-tree.
• Neither merged PR is in promote 89f673c618f. That promote IS the merge of fix(cloud): reuse the existing agent in first-run onboarding instead of 503 dead-end during worker outage #15541 (19:07 ET); fix(cloud): gate provisioning health at enqueue #15544 merged 19:15, fix(ci): scrub deprecated bridge fallback on provisioning deploy #15548 19:27. So if prod is serving 89f673c, prod runs fix(cloud): reuse the existing agent in first-run onboarding instead of 503 dead-end during worker outage #15541's reuse-peek-before-create gate shape; develop runs fix(cloud): gate provisioning health at enqueue #15544's enqueue-boundary shape. Both shapes fix the LP3 QA (develop tip): onboarding ChoiceWidget white-on-white + native cloud onboarding 503-dead-ends despite a running agent #15516 reuse dead-end, so no prod gap — but the next promote silently flips gate placement.

#15544 — gate at enqueue (the real health-gate PR)

1. What the gate is: checkProvisioningWorkerHealth() (packages/cloud/shared/src/lib/services/provisioning-worker-health.ts) = provisioning-worker daemon Redis heartbeat (provisioning_worker:health, 60s TTL, refreshed ~15s/cycle). NOT node/docker/headscale health. Gate now sits at exactly one place in POST /api/v1/eliza/agents — immediately before enqueueAgentProvision (packages/cloud/api/v1/eliza/agents/route.ts:610). Enqueue-time only; no execute-time check exists or is needed — the checked entity IS the executor, so a post-enqueue death just leaves the job queued (pre-existing designed behavior).

2. No new stuck state — CONFIRMED PASS. Gate reject → hard org-scoped row delete (route.ts:612) → canonical 503/502 body with retryable:true. No error-status agent, no orphan pending row. The rollback is byte-equivalent to the existing withOrphanCleanup throw path (route.ts:240 — same agentSandboxesRepository.delete). Fail-closed blast radius shrank: persistent unhealthy heartbeat now blocks only cold dedicated creates; reuse/shared-tier/warm-pool/non-eager all serve through the outage. Operator override: none in production (required unconditional when NODE_ENV=production); the only emergency lever is the fall-open branch when the Worker's Redis binding is absent. REQUIRE_PROVISIONING_WORKER only forces the gate ON in non-prod.

3. Idempotency — CONFIRMED PASS. Idempotent reuse returns 200 at route.ts:441, before the gate and before withOrphanCleanup — the rollback-delete can never reach a reused/live agent. forceCreate cannot skip the gate (test: 503 + rollback). Worker-side no-op-on-healthy-agent path untouched.

4/5. Composition + races:

• Orthogonal to fix(cloud): never flip running without durable node attribution (dockerMeta guard miss + warm-pool claim of null-node pool row) #15424 flip-guard / fix(cloud): self-heal readiness-probe false-negative wedged provisions (#15310 #6) #15377 reconciler / #15424 node_id-flip guard incomplete — 2 of 3 flip paths (adopt-retry + daemon CAS) still mint running+NULL-node orphans #15477: gate is route-side liveness; those are worker-side. Rolled-back rows die pre-enqueue/pre-container — nothing for the reconciler to ghost-kill.
• One real defect-lite (fast-follow, not blocker): between createAgent's commit (org advisory lock released) and the gate's rollback-delete, a concurrent same-org create can idempotently reuse the doomed pending row → caller B gets 200 with an agent id that caller A's rollback then hard-deletes. Window = KMS env mint + env update + warm-claim attempt + Redis get (hundreds of ms), and the trigger is live for the entire worker outage. The race class pre-exists (any withOrphanCleanup rollback), but fix(cloud): gate provisioning health at enqueue #15544 makes the trigger far more frequent. Consequence: transient client-visible dangling id (next poll 404s, retry self-heals) — not a stuck DB row. Fix shape: conditional delete (status still pending, updated_at unchanged) or re-take the org lock around rollback.
• Cosmetic: if the rollback delete throws, it escapes into withOrphanCleanup → client sees 500 instead of the canonical 503 body (delete retried best-effort; cron is the net).
• Managed-env side effects (KMS DEK mint) aren't undone by the row delete — identical to the pre-existing throw path; no new leak class.

6. Tests (run isolated at merged tip d4880be in a throwaway worktree, since the review worktree predates the merges — main tree untouched, worktree removed after):

• eliza-agents-create-idempotency.test.ts: 12 pass / 0 fail
• eliza-agents-orphan-cleanup.test.ts: 5 pass / 0 fail (needs ensure-shared-i18n-data.mjs + core build in a fresh worktree)
• eliza-agents-warm-pool-empty-on-claim.test.ts: 2 pass / 0 fail
• Red-on-base confirmed: swapping route + shared files back to base 89f673c → 8/12 fail, including all three LP3 QA (develop tip): onboarding ChoiceWidget white-on-white + native cloud onboarding 503-dead-ends despite a running agent #15516 gate tests.
• Zero dangling references to the removed hasReusableNonTerminal* peek at the merged commit.

#15548 — bridge-fallback scrub

CI/deploy-workflow-only, confirmed — touches .github/workflows/deploy-eliza-provisioning-worker.yml (one sed -i "/^AGENT_ROUTER_ALLOW_BRIDGE_HOST_FALLBACK=/d" added to the env-reconcile block) + its test. No runtime code. Indirect runtime effect: every provisioning-worker deploy scrubs the hand-set flag from /opt/eliza/cloud/.env.local — the declared #15347 durability guard. Only nit: an operator who hand-sets the fallback mid-incident gets it silently reverted on next deploy (no log line for the removal). Test: 10 pass / 0 fail on Linux — the 5 GNU-sed executed-bash cases actually ran here (author's macOS run skipped them), so the verbatim bash is now proven, not just string-matched.

Verdicts

PR	Verdict
#15542	N/A — closed unmerged, superseded by #15544; nothing in-tree
#15544	CLEAN, one fast-follow: reuse-of-doomed-pending-row race during outage (transient client ghost, retry self-heals, no stuck state); consider conditional rollback-delete
#15548	CLEAN (CI-only; executed-bash cases verified green on GNU sed)

Datum for the promote lane: next develop→main promote carries the gate-placement flip (#15544) — behavior change is strictly looser (fewer 503s), rollback path test-proven.

— [qa-agent] fast-follow ask: the reuse-of-doomed-pending-row race during a worker outage (concurrent same-org create can 200 with an id the gate's rollback then deletes — transient, retry self-heals, but the trigger is live for a whole outage). Conditional rollback-delete (status still pending + updated_at unchanged) closes it. Red-on-base verified (8/12 fail on base), 19/0 tests at tip. Also datum for the promote lane: next develop→main promote flips gate placement from #15541's reuse-peek shape to this enqueue-boundary shape — strictly looser (fewer 503s), rollback test-proven.