feat(deploy): Slice B1 — ProgressEmitter + SSE on hdb_deployment lifecycle#657
Conversation
Wires the ProgressEmitter (resurrected from the paused #531) into the new DeploymentRecorder so every deploy_component lifecycle phase is captured on the row's event_log AND streamable live via SSE. Same content-negotiated branch serves get_deployment, letting Studio (or any client) replay a deploy's history and tail in-flight events through a single endpoint. What's new - DeploymentRecorder subscribes to a ProgressEmitter and coalesces writes: every emit appends to a bounded event_log (200 cap, head+tail retention so the lifecycle spine survives a noisy install); chained puts collapse a burst into one round trip. Emits a `_recorder_finished` sentinel on finish() so SSE tailers terminate cleanly even on crash paths. - deployComponent emits prepare/load/replicate/restart/success phase events around their respective steps. Strips req.progress before replicateOperation so peers see a clean payload. Skips recording entirely on replicated (peer-side) executions — origin owns the canonical row. - An in-memory activeEmitters Map keyed by deployment_id lets get_deployment SSE locate the live emitter and tail it. - handlePostRequest gains a content-negotiated SSE branch (req.headers.accept includes text/event-stream + op in SSE_PROGRESS_OPERATIONS). Prime write on the PassThrough so Fastify starts piping immediately — empirically Fastify buffers a returned Readable until end-of-stream without it, collapsing all intermediate writes into a single flush. - get_deployment with SSE subscribes to the live emitter BEFORE reading the row, then replays the historical event_log and dedupes by timestamp so no event is lost in the stitching gap. A polling fallback resolves the SSE promise even if the deploy disappears without signaling a terminal event. - CLI sends Accept: text/event-stream for deploy_component; consumes the SSE response via parseSSE; renders phase/install/error events through DeployRenderer. - httpRequest gains a streamResponse option that yields the raw IncomingMessage as a Readable instead of buffering — what the SSE consumer needs. Ported from #531 (with the multi-line data spec fix, StringDecoder, and disconnect cleanup already applied earlier in the session): - server/serverHelpers/progressEmitter.ts (+ tests) - bin/sseConsumer.ts (+ tests) - bin/deployRenderer.ts (+ tests) Integration coverage: integrationTests/deploy/deploy-tracking-events.test.ts asserts event_log shape on success, SSE replay+done on get_deployment, and the failure path emits an error event into the log. Refs #641 (Slice B1). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| if (liveEmitter && !TERMINAL_STATUSES.has(row.status)) { | ||
| await new Promise<void>((resolve) => { | ||
| resolveLive = resolve; | ||
| // Flush anything that arrived during replay, filtering to events the replay missed. | ||
| for (const buffered of liveBuffer) { | ||
| if (buffered.t > lastReplayedTs) forwardLive(buffered.event); | ||
| } | ||
| if (liveDone) resolve(); | ||
| // Safety net — if the in-memory emitter is dropped (recorder finished or | ||
| // the process recycled) before signaling, poll the row's status as a | ||
| // fallback so the client never hangs indefinitely. | ||
| const pollTimer = setInterval(async () => { | ||
| if (liveDone) { | ||
| clearInterval(pollTimer); | ||
| return; | ||
| } | ||
| const live = getActiveEmitter(req.deployment_id); | ||
| if (!live || live !== liveEmitter) { | ||
| clearInterval(pollTimer); | ||
| const latest = await table.get(req.deployment_id); | ||
| if (latest && TERMINAL_STATUSES.has(latest.status) && !liveDone) { | ||
| liveDone = true; | ||
| resolve(); | ||
| } | ||
| } | ||
| }, 500); | ||
| }); | ||
| } |
There was a problem hiding this comment.
Missing test coverage for the live-tail branch.
The integration test suite (deploy-tracking-events.test.ts) only exercises the terminal/replay path — get_deployment SSE is called after the deploy has already completed, so liveEmitter is always undefined and this whole block is skipped.
The live-tail code is the most complex part of this PR: subscribe-first ordering, liveBuffer dedup by timestamp, the resolveLive closure, and the 500 ms poll fallback. Per the review layer: "if the PR introduces a runtime-shape branch, both legs need coverage — a test that only lands in the fallback gives false confidence on the production path."
A minimal integration test for this branch: start the SSE watcher before the deploy completes (e.g., start the deploy, immediately attach get_deployment with Accept: text/event-stream from a parallel request, and assert that live phase events appear in the stream before the done event). The poll fallback can be exercised by wrapping the emitter deletion and verifying the client still closes cleanly.
| emit('error', { | ||
| message: err?.message ?? String(err), | ||
| code: err?.statusCode ?? err?.code, | ||
| phase: recorder?.row.phase, | ||
| }); | ||
| if (recorder) await recorder.finish('failed', err); |
There was a problem hiding this comment.
Double error event on failed SSE deploy.
emit('error', ...) writes the error into the PassThrough stream. Then throw err causes createSSEResponseStream's .catch handler to write a second error event. The CLI renders both — the user sees the error line twice. The second event also overwrites sseError in the consumer loop, dropping the phase field from the first.
One fix: after emitting the error here, signal completion through the emitter (e.g., emit _recorder_finished) so createSSEResponseStream knows the operation's error is already accounted for, then return a sentinel instead of throwing. Alternatively, suppress the framework-level catch event when an application-level error event was already written.
|
Reviewed; no blockers found. |
…ation test When operations.js emits an `error` event through the ProgressEmitter before throwing, createSSEResponseStream's .catch handler was writing a second error SSE record — dropping the phase context from the first. Fix: track whether the subscriber already forwarded an error event and skip the framework fallback if so. Adds a unit test for the dedup behavior and an integration test covering the live-tail SSE branch (liveEmitter && !TERMINAL_STATUSES) which was previously unexercised — the new test opens get_deployment SSE against an in-flight deploy using a sleep 3 install command. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Both findings addressed in commit pushed just now: Finding 1 (missing live-tail test): Added Finding 2 (double error event): Added an — Claude |
…ires
Without a package.json Harper skips install entirely ("no package.json; skipping install"), so the deploy completes in <100ms and the polling loop
never catches an in-flight row. The other install-command tests in this file
already include package.json for the same reason.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Ethan-Arrowood
left a comment
Ethan-Arrowood
left a comment
There was a problem hiding this comment.
this is my favorite new feature. tests look good.
kriszyp
merged commit 054ae0a
into
feat/deployment-tracking-slice-a
2 participants
Summary
Slice B1 of #641, stacked on #655 (Slice A). Brings the paused #531 SSE work back, but now wired through the
DeploymentRecorderfrom Slice A — so every deploy lifecycle phase is captured both as a persistentevent_logon thehdb_deploymentrow AND as a live SSE stream to any client requestingAccept: text/event-stream. The same content-negotiated branch servesget_deployment, letting Studio replay a deploy's history and tail in-flight events through a single endpoint.Where to look
server/serverHelpers/progressEmitter.ts— ported from feat(deploy): live SSE progress for deploy_component #531 with the multi-line SSE spec fix, StringDecoder, disconnect cleanup, and the new prime write (: stream open\n\n). The prime is the result of an empirical investigation: without it, Fastify buffers the returned PassThrough until end-of-stream and only flushes the final chunk. The prime forces the response to start streaming immediately.components/deploymentRecorder.ts— subscribes to aProgressEmitter, coalesces writes (one in-flight put at a time + dirty flag), boundsevent_logto 200 entries with head+tail retention so a noisy install doesn't purge the lifecycle spine. Emits a_recorder_finishedsentinel onfinish()before unsubscribing.components/deploymentOperations.ts—handleGetDeploymentSSE branch subscribes to the live emitter before reading the row (closing the stitching gap between historical replay and live tail), dedupes by timestamp, and has a polling fallback so the SSE promise always resolves even on crash paths.components/operations.js—deployComponentemits phase events aroundprepare/load/replicate/restart/success, stripsreq.progressbeforereplicateOperation(peers can't deserialize functions), skips recording on replicated executions.server/serverHelpers/serverHandlers.js— adds the SSE_PROGRESS_OPERATIONS branch.Acceptparsing uses.includes('text/event-stream')to play nice with RFC 7231 multi-type accepts.bin/cliOperations.ts+bin/sseConsumer.ts+bin/deployRenderer.ts— CLI consumes SSE, renders phase/install/error events live.Findings from cross-model review (gemini)
All BLOCKER/MAJOR findings addressed in this PR:
_recorder_finishedsentinel + 500ms polling fallback===Accept-header check fragile under multi-type accepts.includes('text/event-stream')Test plan
integrationTests/deploy/deploy-tracking-events.test.ts— 4/4 passdeploy-tracking.test.tsregression — 4/4 passdeploy-multipart-stream.test.ts— 3/3 passnpm run format:check+npm run lint:requiredcleanScope deliberately deferred
npm installstdout/stderr isn't yet forwarded into the emitter. That's Slice B2 alongside peer-side blob reads.Stacked on top of #655 (Slice A). Will retarget to main once #655 lands.
🤖 Generated with Claude Code