bug(v6.0.2): shared RELAY_API_KEY across machines — broker exits at WebSocket subscription with 'Unable to connect'

[prefrontalsys]

Bug: shared RELAY_API_KEY across machines — broker exits at WebSocket subscription with "Unable to connect"

Summary

In v6.0.2, the documented "set the same RELAY_API_KEY on multiple machines so brokers join the same workspace" pattern (which #789 builds on as a precondition) silently fails. The HTTP registration step succeeds — the cloud accepts the key and registers the agent — but the WebSocket subscription step that follows exits with "Unable to connect. Is the computer able to access the url?", killing the broker.

This blocks every multi-machine use case, including the messaging premise that #789 assumes works.

Environment

• agent-relay: v6.0.2 (standalone binary install via install.sh)
• Mac: macOS arm64, agent_name sc, launchd-managed broker, ran agent-relay cloud login (token in ~/.agent-relay/cloud-auth.json)
• Linux VPS: Ubuntu aarch64, agent_name ubuntu, systemd-user managed broker, same cloud-auth.json copied via scp from Mac
• agent-relay cloud whoami returns identical user/org/workspace on both: sr4001@gmail.com, Scot Campbell's Workspace, Default

Reproduction

1. On Mac, agent-relay up --no-dashboard — broker starts, creates workspace, prints Workspace Key: rk_live_4165c2f458fc2976c0dd0ad092050afb
2. Verify the key is live from VPS:

   curl -H "Authorization: Bearer rk_live_4165c2f..." https://api.relaycast.dev/v1/channels
   → 200 OK, channels listed (`general`, `engineering`)
   

3. On VPS, set the env var and start:

   RELAY_API_KEY=rk_live_4165c2f... agent-relay up --no-dashboard
   

4. Observed: Broker started. is logged, then the broker exits within ~1s with status code 1 and stderr Failed to start broker: Unable to connect. Is the computer able to access the url?
5. Cross-check on the cloud during step 4 (via api.relaycast.dev /v1/agents):
   • ubuntu agent does appear in the list, status offline, with a released_at timestamp matching the broker exit
   • i.e., the agent registration HTTP call succeeded — the failure is downstream

Why I think this is the WS-subscription step

• VPS network is fine (HTTP 200 to api.relaycast.dev/v1/channels)
• Bearer auth works (the same key returns 200 from VPS via curl)
• The broker logs Broker started. (after connect_relay completed) before the error appears
• The "Unable to connect" message is identical to what surfaces from tokio_tungstenite failures elsewhere in the binary's strings (tokio_tungstenite::tls::encryption::rustls)

What works (control)

Without RELAY_API_KEY, both brokers run fine (each auto-creates its own workspace). Each can talk to its own agents on its own host. The bug is only triggered when a broker is told to join a workspace it didn't create.

Hypothesis on root cause

The WebSocket subscription requires an auth context tied to the creator of the workspace — perhaps the cloud token used at workspace creation gets bound to subsequent WS connections. A broker that joins via RELAY_API_KEY only has its own cloud token, which the WS endpoint won't accept for that workspace's channel.

If correct, the fix is either (a) accept any cloud-auth token from the same org for WS auth on any workspace owned by that org, or (b) provide a workspace.export / workspace.invite flow that issues a transferable WS-eligible credential.

Why this matters for #789

#789 ("remote spawn") explicitly states: "multiple brokers sharing a workspace key can exchange messages, DMs, and channel posts in real-time." That assumption is currently false in v6.0.2 — sharing a workspace key crashes the broker. Spawning across machines presupposes the messaging fabric works first; this bug must be fixed (or worked around) before #789 becomes meaningful.

Asks

1. Confirm whether multi-machine RELAY_API_KEY sharing is an intended supported configuration in v6 or a regression from v5.
2. If supported: surface the actual WS error (current "Unable to connect" is misleading — HTTP auth is fine).
3. If not yet supported: document this clearly in the README and add a workspace export / workspace join <key> flow that produces the right credential bundle.
4. Consider whether RELAY_WORKSPACES_JSON / RELAY_DEFAULT_WORKSPACE (visible in binary strings, undocumented) is the intended path here.

Logs / evidence available

Happy to attach:

• journalctl --user -u agent-relay from VPS during the failing window
• ~/Library/Logs/agent-relay.{out,err}.log from Mac (success case)
• ~/.agent-relay/identity-debug.txt from both machines showing distinct agent_id / default_workspace values that converge when RELAY_API_KEY is shared (proving HTTP-side resolution works)
• api.relaycast.dev/v1/agents listing from before/after the VPS attempt

[prefrontalsys]

Investigation summary + fix plan

I traced the call path through the cloned repo. Posting the orientation here so anyone picking this up has a head start. Happy to take the fix myself (already assigned to #789, this is its prerequisite).

Key finding: the misleading "Unable to connect..." error is not from the Rust broker — it's from the JS launcher. The actual Rust failure is being swallowed by the SDK's broker-readiness probe.

• "Failed to start broker: ..." lives at packages/sdk/src/client.ts:636.
• "Unable to connect. Is the computer able to access the url?" is Bun's runtime error wrapping ECONNREFUSED on fetch (cross-confirmed in opencode #16122, #7278, grimoire Fix trajectory viewer race condition with request counter #171).
• Chain: Rust broker exits → JS launcher's HTTP probe to 127.0.0.1:<port>/health gets ECONNREFUSED → Bun wraps it → SDK prefixes "Failed to start broker:".

Call path traced (verified by Read on repo at main branch):

1. main.rs:768 → run_init → connect_relay (main.rs:478)
2. connect_relay → AuthClient::startup_session_set_with_options (auth.rs:254)
3. With RELAY_API_KEY env, register_agent_with_workspace_key (auth.rs:597-651) succeeds — registration HTTP returns 2xx, both agents visible in /v1/agents. Returns workspace_id=None because CreateAgentResponse doesn't include it.
4. MultiWorkspaceSession::new (multi_workspace.rs:38) spawns per-workspace WS task → RelaycastWsClient::run (relaycast_ws.rs:57).
5. WS connect at relaycast_ws.rs:74-79:

   let mut ws = WsClient::new(
       WsClientOptions::new(self.workspace_http.api_key.clone())  // workspace key only
           .with_base_url(self.ws_base_url.clone()),
   );

   Only the workspace API key is passed to the WS handshake. The agent's bearer token (session.token, at_live_*) — already cached via seed_agent_token in multi_workspace.rs:87-93 for HTTP — is not given to the WS client.

However: WS connect failures here are tracing::warn! only with a 3s retry loop — they don't exit the process. So the fatal exit is somewhere between connect_relay completed (main.rs:942) and the API server spawn (main.rs:1031). Step zero to actually nail this is to reproduce with AGENT_RELAY_STARTUP_DEBUG=1 RUST_LOG=relay_broker=trace,relaycast=trace — I'll do that next session and post the exit phase here.

Closed-crate caveat: Cargo.toml pins relaycast = "=1.0.0" (private/internal). Whether WsClientOptions exposes a way to attach an agent bearer token is unknowable from this repo alone.

Fix branches (ranked, conditional on step zero output):

• A (likely): WS handshake needs the agent bearer token. Patch relaycast_ws.rs:74-79 to pass agent_token from seed_agent_token cache. May require an upstream PR to the closed relaycast crate first to add WsClientOptions::with_agent_token.
• B: CreateAgentResponse should include workspace_id so register_agent_with_workspace_key can populate it (currently None). Cloud-side schema coordination.
• C: Rate-limit / race on second-broker registration. Less likely given deterministic repro.

Open questions for the team:

1. Is WsClientOptions::with_agent_token (or equivalent) exposed in relaycast = 1.0.0? If not, where do agent tokens enter WS auth today?
2. Is the cloud-side WS endpoint expected to accept the workspace API key alone, or workspace-key + agent-token together?
3. Should register_agent_with_workspace_key populate workspace_id in its response?

If you can confirm #1/#2, the fix surface narrows drastically. Otherwise I'll proceed via step-zero log capture.