fix(hub): single-flight local agent daemon connection + level-gated debug logging#53
Merged
Merged
Conversation
…ebug logging Local terminals failed in the prod SEA build with SPAWN/CHANNEL_STATE timeouts. Root cause: the hub opened multiple hub->agent connections (a throwaway socket probe during launch + un-coalesced attaches), churning the agent's single-connection policy until its daemon wedged; a displaced connection's CHANNEL_STATE never arrived and the hub hung instead of failing fast. Fix (validated end-to-end, ~130ms spawn on clean state): single-flight attachDaemon (reuse live agent / join one in-flight attach per host, publish to ctx.agents only after the AUTH+CHANNEL_STATE handshake, evict on disconnect/failure); connectOrLaunch no longer uses a throwaway socket probe (polls with the real connectLocal); waitForChannelState rejects on socket close before CHANNEL_STATE_END; restart consumes the real SPAWN_OK channelId and ATTACH CHANNEL_NOT_FOUND retires the stale hub channel. Observability: fixed a HubLogger import-cycle that prevented [logging] level=debug from gating output (LOG_SEVERITY moved to a leaf module); converted ad-hoc debug traces to level-gated hubLogger.debug (hub) / tracing::debug (agent) across the connection lifecycle, off by default. Added a committed headless test harness (scripts/dev/headless-hub-test.sh + headless-spawn-probe.mts) with isolated XDG state and PID-file lifecycle.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Local terminals failed in the prod SEA build with
SPAWN timeout/CHANNEL_STATE timeout. Rootcause: the hub opened multiple hub→agent connections — a throwaway socket probe during agent launch
plus un-coalesced attaches — which churned the agent's single-connection (last-writer-wins) policy until
its daemon wedged; a displaced connection's
CHANNEL_STATEnever arrived and the hub hung for thefull timeout instead of failing fast. Surfaced on the first real prod-SEA local-terminal test.
Validated end-to-end (headless harness): clean state →
SPAWN_OKin ~130ms, repeatedly.Fix (hub + agent)
agent-connection-manager.ts): reuse a livectx.agentsagent, otherwise join the one in-flight attach promise per host; publish to
ctx.agentsonly afterthe
AUTH+CHANNEL_STATEhandshake completes; evict on disconnect/handshake failure.agent-launcher.ts):connectOrLaunch/connectWhenReadypoll with thereal
connectLocalconnection — the probe was being counted by the daemon as a hub connection anddisplaced the real one.
termora-agent.ts):waitForChannelStaterejects on socket close/errorbefore
CHANNEL_STATE_END(instead of always running to the 5s timeout).channel-lifecycle-manager.ts/session-manager.ts): restartconsumes the real
SPAWN_OK.channelId; anATTACHreturningCHANNEL_NOT_FOUNDretires the stale hubchannel rather than serving a cached
ATTACH_OK.Observability (level-gated debug logging)
[logging] level = "debug"from gating output(
LOG_SEVERITYmoved to a leaflevels.ts). Now debug records actually surface atdebugand aresuppressed at
info.existing stacks —
hubLogger.debug(hub) andtracing::debug!(agent, with aconnection_id) acrossthe connection lifecycle (accept / displace / auth / handshake / per-message). Off by default.
Headless test harness (committed)
scripts/dev/headless-hub-test.sh(isolated XDG state, PID-file lifecycle — nops/fuser) +headless-spawn-probe.mts(WS client: AUTH + SPAWN, logs the message sequence + timing). Documented inthe README. This is the repro that pinned the bug; kept as the canonical local-spawn E2E.
Verification
vitest packages/hub packages/shared→ 1673 passed / 2 skipped / 0 failed; biome + tsc clean; nonew skips. New HubLogger debug-gating tests + TermoraAgent/session tests.
cargo fmt --check,cargo clippy -p termora-agent --all-targets -- -D warnings,cargo clippy --target x86_64-pc-windows-msvc -p termora-agent --all-targets -- -D warnings,cargo test -p termora-agent→ all green.SPAWN_OKat +127ms; debug lifecycle records visible atlevel=debug.Also hardens a few agent tests for sandbox reliability (isolated state dirs; ASKPASS fallback when
XDG_RUNTIME_DIRis unwritable, preservingO_EXCL+0700).Gate
Orthogonal pre-PR gate: codex = CLEAN; copilot exogenously unavailable (rate-limited). Degraded
GATE_OK per set-union semantics.