fix(cloud): reuse the existing agent in first-run onboarding instead of 503 dead-end during worker outage

[NubsCarson]

What & why (#15516 Bug B)

On a real LP3 device, native cloud onboarding provisioned + 503'd and dead-ended for a user whose one healthy agent (running/ready/pairable) needed no provisioning at all. Two stacked defects, both fixed here:

Client — the reuse pick refused every non-running agent

#15491 changed pickPreferredCloudAgent to running-only. Any transiently non-running row (starting/stopped/status churn during the very outage that 503s creates) produced zero reuse candidates → first-run fell through to a fresh create. The reuse branch's cold-boot wait (waitForCloudAgentRunning) had become unreachable dead code — a pick was always running, so the wait never fired.

Fix: pick prefers running, then falls back to the newest non-terminal row, routed through waitForCloudAgentRunning — which kicks a resume and resolves with the fresh post-wake record (post-wake URLs). That is the correct guard against #15491's stale-pointer concern (never bind a stale row directly), instead of refusing reuse outright. Safe because shared rows are born running (eliza-sandbox.ts create: shared → "running"), so a non-running pick always has a container to wake. A remembered preferAgentId is honored whenever non-terminal — silently binding a different agent would swap the user's conversations/memories.

Server — the worker-health gate ran before its own reuse safety-net

POST /api/v1/eliza/agents ran checkProvisioningWorkerHealth() before createAgent's idempotent reuse — so during a heartbeat gap (real outage in the CP journal: Redis socket timeouts 20:26–20:40, key TTL 60s; zero provision jobs 20:00–21:33, all pre-enqueue rejects) even callers whose create would be served by reuse got 503.

Fix: the route peeks reuse candidacy first (read-only repository mirror of the guard predicate, documented as such) and only gates a create that would genuinely enqueue. forceCreate always stays gated (it opts out of reuse by definition). Fail-closed behavior for no-agent orgs is unchanged.

Proof

The existing mock-cloud connect suite was RED on develop — 3 tests fail on the unfixed code with the exact bug signature (create POST fired instead of reusing the stopped agent):

× waits out a dedicated cold boot: resume kick, starting progress, fresh post-wake base…
× fails fast with the agent's error message on a terminal error status
× times out with an actionable message when the container never reports running
  → Cloud request failed (404): no mock route for POST /api/v1/eliza/agents

With this fix: 9/9 pass (incl. 2 new: remembered-agent wake priority; all-terminal → create fall-through).

Route suites (bun): 12/12 idempotency (3 new #15516 ordering tests), 5/5 orphan-cleanup, 2/2 warm-pool (per-file, matching CI; the combined-single-process run has pre-existing cross-suite mock.module interference on develop too — verified 0-pass/2-fail on untouched develop). New ordering coverage:

• worker outage + reusable agent → 200 reuse, gate skipped entirely (checkProvisioningWorkerHealth not called)
• worker outage + no reusable agent → 503 fail-closed, createAgent never runs
• worker outage + forceCreate + reusable agent → 503 (forceCreate never skips the gate)

Biome clean; typecheck clean for packages/ui, packages/cloud/shared, packages/cloud/api (filtered to touched files).

Evidence rows

• Backend logs: CP journal forensics + prod DB job-table evidence in the root-cause comment on LP3 QA (develop tip): onboarding ChoiceWidget white-on-white + native cloud onboarding 503-dead-ends despite a running agent #15516 (heartbeat outage windows, zero-jobs gap). N/A for staging traces — the fix is proven by the red→green suite + route tests above.
• Screenshots / video: N/A — no rendered UI change; the flow fix is client-logic + API. The on-device re-verify (fresh LP3 pass through onboarding, including during a simulated heartbeat gap) is @qa-agent's device lane per the issue.
• LLM trajectory: N/A — no agent/prompt/model behavior change.
• Domain artifacts: route tests assert the exact wire shapes (200 reuse envelope, 503 PROVISIONING_WORKER_UNHEALTHY body).

Residual (separate, flagged on #15516): the /provision route's healthy-agent 200 fast-path requires bridge_url AND health_url — a healthy agent missing either still falls into the gate; that's #15477-adjacent (provisioning-durability lane).

Fixes #15516 (Bug B) · root cause + tracer evidence: #15516 (comment) · [cloud-agent]

Evidence rows

• Before screenshots N/A - no rendered UI change (client flow-logic + API route ordering); on-device before/after is the qa device-lane re-verify per #15516.

• After screenshots N/A - no UI surface changed.

• Walkthrough video N/A - no user-visible web/desktop flow change; the LP3 on-device walkthrough is the qa device lane's re-verify step tracked on #15516.

• Backend logs: CP journal + prod jobs-table forensics (heartbeat outage windows, zero-jobs gap) in the root-cause comment: LP3 QA (develop tip): onboarding ChoiceWidget white-on-white + native cloud onboarding 503-dead-ends despite a running agent #15516 (comment)

• Frontend logs: the mock-cloud suite drives the REAL client HTTP path against a real HTTP server and asserts the request log (red on develop, 9/9 green here); original device logcat: LP3 QA (develop tip): onboarding ChoiceWidget white-on-white + native cloud onboarding 503-dead-ends despite a running agent #15516

• Real-LLM trajectory N/A - no agent/action/provider/prompt/model behavior change; control-plane transport only.

• Domain artifacts: route tests assert the exact wire shapes (200 reuse envelope, 503 PROVISIONING_WORKER_UNHEALTHY fail-closed, forceCreate gated) — see test output in the PR description; suite files: https://github.com/elizaOS/eliza/pull/15541/files

• OCR visual text review N/A - no media/screenshot artifacts in this change.

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.