Skip to content

fix(hub): single-flight local agent daemon connection + level-gated debug logging#53

Merged
oorabona merged 1 commit into
mainfrom
fix/local-agent-connection-lifecycle
Jun 9, 2026
Merged

fix(hub): single-flight local agent daemon connection + level-gated debug logging#53
oorabona merged 1 commit into
mainfrom
fix/local-agent-connection-lifecycle

Conversation

@oorabona

@oorabona oorabona commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

What & why

Local terminals failed in the prod SEA build with SPAWN timeout / CHANNEL_STATE timeout. Root
cause: the hub opened multiple hub→agent connections — a throwaway socket probe during agent launch
plus un-coalesced attaches — which churned the agent's single-connection (last-writer-wins) policy until
its daemon wedged; a displaced connection's CHANNEL_STATE never arrived and the hub hung for the
full timeout instead of failing fast
. Surfaced on the first real prod-SEA local-terminal test.

Validated end-to-end (headless harness): clean state → SPAWN_OK in ~130ms, repeatedly.

Fix (hub + agent)

  • Single-flight local daemon connection (agent-connection-manager.ts): reuse a live ctx.agents
    agent, otherwise join the one in-flight attach promise per host; publish to ctx.agents only after
    the AUTH + CHANNEL_STATE handshake completes; evict on disconnect/handshake failure.
  • No throwaway socket probe (agent-launcher.ts): connectOrLaunch/connectWhenReady poll with the
    real connectLocal connection — the probe was being counted by the daemon as a hub connection and
    displaced the real one.
  • Fail fast, don't hang (termora-agent.ts): waitForChannelState rejects on socket close/error
    before CHANNEL_STATE_END (instead of always running to the 5s timeout).
  • Restart channel-id reconciliation (channel-lifecycle-manager.ts / session-manager.ts): restart
    consumes the real SPAWN_OK.channelId; an ATTACH returning CHANNEL_NOT_FOUND retires the stale hub
    channel rather than serving a cached ATTACH_OK.

Observability (level-gated debug logging)

  • Fixed a HubLogger import cycle that prevented [logging] level = "debug" from gating output
    (LOG_SEVERITY moved to a leaf levels.ts). Now debug records actually surface at debug and are
    suppressed at info.
  • Converted the ad-hoc traces added during diagnosis into permanent, level-gated logging on the
    existing stacks — hubLogger.debug (hub) and tracing::debug! (agent, with a connection_id) across
    the connection lifecycle (accept / displace / auth / handshake / per-message). Off by default.

Headless test harness (committed)

scripts/dev/headless-hub-test.sh (isolated XDG state, PID-file lifecycle — no ps/fuser) +
headless-spawn-probe.mts (WS client: AUTH + SPAWN, logs the message sequence + timing). Documented in
the README. This is the repro that pinned the bug; kept as the canonical local-spawn E2E.

Verification

  • TS: vitest packages/hub packages/shared → 1673 passed / 2 skipped / 0 failed; biome + tsc clean; no
    new skips. New HubLogger debug-gating tests + TermoraAgent/session tests.
  • Rust: cargo fmt --check, cargo clippy -p termora-agent --all-targets -- -D warnings,
    cargo clippy --target x86_64-pc-windows-msvc -p termora-agent --all-targets -- -D warnings,
    cargo test -p termora-agent → all green.
  • Harness smoke: SPAWN_OK at +127ms; debug lifecycle records visible at level=debug.

Also hardens a few agent tests for sandbox reliability (isolated state dirs; ASKPASS fallback when
XDG_RUNTIME_DIR is unwritable, preserving O_EXCL + 0700).

Gate

Orthogonal pre-PR gate: codex = CLEAN; copilot exogenously unavailable (rate-limited). Degraded
GATE_OK per set-union semantics.

…ebug logging

Local terminals failed in the prod SEA build with SPAWN/CHANNEL_STATE timeouts. Root cause: the hub opened multiple hub->agent connections (a throwaway socket probe during launch + un-coalesced attaches), churning the agent's single-connection policy until its daemon wedged; a displaced connection's CHANNEL_STATE never arrived and the hub hung instead of failing fast.

Fix (validated end-to-end, ~130ms spawn on clean state): single-flight attachDaemon (reuse live agent / join one in-flight attach per host, publish to ctx.agents only after the AUTH+CHANNEL_STATE handshake, evict on disconnect/failure); connectOrLaunch no longer uses a throwaway socket probe (polls with the real connectLocal); waitForChannelState rejects on socket close before CHANNEL_STATE_END; restart consumes the real SPAWN_OK channelId and ATTACH CHANNEL_NOT_FOUND retires the stale hub channel.

Observability: fixed a HubLogger import-cycle that prevented [logging] level=debug from gating output (LOG_SEVERITY moved to a leaf module); converted ad-hoc debug traces to level-gated hubLogger.debug (hub) / tracing::debug (agent) across the connection lifecycle, off by default. Added a committed headless test harness (scripts/dev/headless-hub-test.sh + headless-spawn-probe.mts) with isolated XDG state and PID-file lifecycle.
@oorabona oorabona merged commit d67e3d5 into main Jun 9, 2026
8 checks passed
@oorabona oorabona deleted the fix/local-agent-connection-lifecycle branch June 9, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant