feat(kap-server): add the WS v2 realtime protocol stack - #3538
feat(kap-server): add the WS v2 realtime protocol stack#3538liruifengv wants to merge 34 commits into
Conversation
Single source of truth for the v2 wire contract: 25-type ServerMessage discriminated union, ClientMessage control frames, parseServerMessage, entityId/entityKey. Exposed as @moonshot-ai/kap-server/protocol/v2; code-app's app-core ws2/messages becomes a pure re-export.
Event2 -> ServerMessage projection core: per-agent timeline projector (turn/step/text/tool/prompt domains), session.state composer, and a script harness that deep-equals projected streams against the vendored 24-tab example fixtures (basic and tool tabs covered).
submitSteer enqueues through the same path as regular submissions, so a steer-bound prompt was indistinguishable from a queue-bound one on the event stream. The new optional steer flag lets projections hold the queued emission and place the user message directly into the active turn when prompt.steered arrives.
multi-tool, todo, queue-abort, tool-error and llm-retry streams now deep-equal the example fixtures. The projector gains: tools.update_store driven todo entities with projector-allocated td_* ids and the TodoWrite back-link, prompt.aborted driven system interruption frames (m_* ids), interrupted-step usage passthrough, error tool results without output, retry-step text reset with placeholder re-announce, and progress cleanup on tool completion. Turn usage now sums plain input_other/output only. ProjectionEvent declares its payload fields explicitly to satisfy noPropertyAccessFromIndexSignature.
…eams Six more fixture tabs now deep-equal: approval (both branches), question, background-task, compaction, undo and big-output. The projector gains a generic interaction path (pending frames with tool_call approval_id back-link, terminal frames preserving the original request), bus-driven approval events (permission.approval.requested/resolved), compaction system frames with summarized_through_turn derivation, undo frames from context.undone, shell.output driven task output_tail refreshes, and a turnId hint on prompt.submitted for engine-known turn clocks.
…treams Five more fixture tabs deep-equal (19 total). plan mode enter/exit edges from agent.status.updated with revision-tracked payloads, goal.updated system frames and goal-continuation turns without user frames, sidechat agents sharing the session turn clock, attachment_ids flowing from prompt submissions into user and turn frames, and cron.fired normalized through the prompt submission path. applyFacts now flushes in-flight text before composing session.state, and modes becomes a per-frame fact instead of sticky status state.
…live tabs injection (steer, busy-cron, busy-task-notify) and subagent (foreground, background, foreground-to-background, plus the on-demand subagent channel) now deep-equal the fixtures, covering every engine-synthesizable live tab. Busy cron injections take the system-steer path with pre-emptive completion ordering, subagent lifecycle drives task entities with child_agent_id and agent_refs on the parent tool call, and the subagent channel projects the child agent's own stream under an r-prefixed turn namespace.
recordRevision now derives a summary from the first markdown heading of the snapshotted plan so wire consumers can label revisions without fetching the blob.
buildColdHistory folds wire.jsonl into terminal v2 entities: turn groups of step groups in wire order, protocol ids reproduced cold, in-flight turns without a cover but with their completed steps, undo kept with a marker, clear/compaction as fold floors, and keyset pagination over before_turn/after_step. A 100k-line wire rebuild lands in ~240ms. The route is served through defineRoute and shares readColdWireRecords with the transcript service. Thirteen REST fixture sections now deep-equal the builder output alongside the twenty-one live streams.
A session binder feeds engine events to the v2 projector and routes the emitted messages to subscribed connections. WsConnectionV2 implements hello on connect, subscribe/ack with ErrorCode acks, per-type omit, a bounded outbound queue that closes slow consumers with backpressure_overflow, protocol-layer heartbeats, and on-demand subagent channels. Recovery is the same sequence as live: in-flight turn cover, current step, accumulated streaming entities, state entities, and a fresh session.state, synthesized from the projector's new recoveryEntities API. recovery A/B, omit, backpressure-reconnect and the subagent channel now deep-equal the fixtures end to end.
App-scope events now project to v2 global messages broadcast to every v2 connection: session index frames (meta.updated with SessionInfo and changed_fields, created/archived), workspace created/updated/deleted via toWireWorkspace, sanitized config with changed_fields, config.warning, and the thin model_catalog/plugin/capability notifications. Remaining system subtypes (hook, skill, notice, clear, swarm enter/exit) are projected, and the binder learns profile.bind as a model fact. The session-changes and global tabs deep-equal, and the full kap-server suite passes at 1374 tests.
Real-engine e2e (scripted provider, stub model) covers eight scenarios: basic text, yolo Bash, manual approval with the approval_id back-link, TodoWrite entities, abort with interruption, mid-turn disconnect recovery, REST history after completion, and v1 protocol coexistence. Fixes from the integration: engine step ordinals are 1-based so the projection normalizes to the 0-based wire ids (with a phase clamp before the first step starts), TodoList calls bridge to the TodoWrite wire vocabulary at both the live and cold boundaries, tools.update_store is now observable so live todo entities flow, and the wire manifest is regenerated for the plan.revision summary addition.
The session settles to idle before prompt.aborted projects the interruption marker; fixtures and the queue-abort script follow.
Durable interaction.request records and the interaction hub both carry engine-shaped payloads (toolName/action, question items with object options). Cold history and the live question path now pass them through a shared toWireInteractionRequest/Response mapper (approval requests recover their input from the linked tool call, question items get ids and string options), instead of leaking engine shapes onto the wire and tripping the client's contract guard.
PromptReservation.submit took no options, so a caller could not mark a submission as a steer at enqueue time and had to steer in a separate call afterwards. submit now accepts an optional steer flag and passes it through to enqueue.
POST /sessions/{id}/prompts accepts steer: true. The reservation is
submitted with the steer flag and, when the handle lands pending, is
steered immediately within the same request; a PROMPT_NOT_FOUND race
against an already-launched prompt is tolerated. Clients no longer
issue a separate steer call that rendered the steered user message
twice.
- skip empty thinking parts in the agent loop (live stream, interrupted drain, cold history) so no empty thinking shells are emitted - emit the steered prompt's user message when submission races with turn completion, assigning it to the new turn exactly once - emit goal system entries only on goal status transitions, and mirror that in cold history (goal.create produces the active entry, goal.update/clear produce none) - let before_turn history paging reach past the compaction floor so loading earlier messages keeps returning pages - refresh the v2 example fixtures and add a steer-race projection case
🦋 Changeset detectedLatest commit: d7f7f64 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bc3c7ed71
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } | ||
| const queued = status === 'queued'; | ||
| if (queued) this.queue.push(promptId); | ||
| const predictedEngineTurn = (event.turnId as number | undefined) ?? this.maxTurnId + this.queue.length + (queued ? 0 : 1); |
There was a problem hiding this comment.
When a persisted session is resumed, this projector still starts with maxTurnId === -1, and prompt.submitted does not carry a turnId, so the first new prompt is emitted as t1.u0 even if the actual subsequent turn.started is for turn 15. Because onTurnStarted does not reassign an already-emitted prompt, the new user frame collides with the first user entity loaded from REST history and the new turn references the wrong message ID.
Useful? React with 👍 / 👎.
| let binding = this.bindings.get(source.sessionId); | ||
| if (!binding) { | ||
| binding = new SessionV2Binding(source, this.clock); | ||
| this.bindings.set(source.sessionId, binding); | ||
| } |
There was a problem hiding this comment.
After a session is closed or archived and then restored in the same server process, the map still contains its old binding, so attach returns an object subscribed to the disposed session and agent buses instead of the new source. There is no production call to SessionV2Binder.detach, which means restored sessions stop producing v2 live frames and the retained projectors also accumulate indefinitely.
Useful? React with 👍 / 👎.
| detached: info.detached === true, | ||
| description: info.description as string | undefined, | ||
| output_tail: (info.outputTail as string) ?? '', | ||
| started_at: (info.startedAt as string) ?? iso(event.time), |
There was a problem hiding this comment.
Real AgentTaskInfo values expose startedAt and endedAt as epoch-millisecond numbers, but these casts leave the runtime values numeric while taskMessageSchema requires strings. Consequently both running and terminal task frames fail serverMessageSchema.safeParse in WsConnectionV2.send and are silently dropped; convert these values with new Date(...).toISOString() rather than casting them.
Useful? React with 👍 / 👎.
| const patch = factsPatchForEvent(event); | ||
| if (patch) this.emitFacts(patch, event.time ?? this.clock()); |
There was a problem hiding this comment.
Every watched agent feeds agent.activity.updated, agent.status.updated, goal, and permission patches into the single session composer. When a subagent starts or finishes, its phase/model/goal can therefore overwrite the main agent's values, producing states such as an idle phase while main_turn_active is still true; only main-agent facts should populate this shared session snapshot.
AGENTS.md reference: packages/kap-server/AGENTS.md:L33-L33
Useful? React with 👍 / 👎.
| started_at: (info.startedAt as string) ?? iso(event.time), | ||
| model: info.model as string | undefined, | ||
| thinking_effort: info.thinkingEffort as string | undefined, | ||
| child_agent_id: info.childAgentId as string | undefined, |
There was a problem hiding this comment.
The real subagent task record uses info.agentId, not info.childAgentId, so this field is always omitted for kind: 'agent' tasks. A client receiving the parent task consequently cannot obtain the agent ID needed to subscribe to that subagent's v2 channel; the terminated projection repeats the same mismatch.
Useful? React with 👍 / 👎.
| const usage = event.usage as StatusFactEvent['usage']; | ||
| if (model !== undefined) status.model = model; | ||
| if (thinkingEffort !== undefined) status.thinkingEffort = thinkingEffort; | ||
| if (contextTokens !== undefined) status.contextTokens = contextTokens; | ||
| if (maxContextTokens !== undefined) status.maxContextTokens = maxContextTokens; | ||
| if (usage !== undefined) status.usage = usage; | ||
| return Object.keys(status).length > 0 ? { status } : undefined; |
There was a problem hiding this comment.
agent.status.updated carries planMode and swarmMode, but factsPatchForEvent only copies model, effort, token, and usage fields. No other production path supplies SessionFactsPatch.modes, so session.state.modes can never report either active mode even though the schema and composer expose it.
Useful? React with 👍 / 👎.
| const turnUnits = units.filter((unit) => unit.turnOrdinal !== undefined); | ||
| const startIndex = Math.max(0, turnUnits.length - pageSize); | ||
| const kept = new Set(turnUnits.slice(startIndex)); | ||
| const items = units.filter((unit) => unit.turnOrdinal === undefined || kept.has(unit)).flatMap((unit) => unit.items); |
There was a problem hiding this comment.
For the default page, this predicate retains every non-turn unit, including every historical task, regardless of the last page_size turn window. Sessions with many background tasks therefore receive an effectively unbounded first page, and those old tasks are returned again on subsequent before_turn pages; standalone units need to be bounded by the same cursor window rather than admitted solely because they lack turnOrdinal.
Useful? React with 👍 / 👎.
Approval requests in the v2 stream dropped the engine's ToolInputDisplay,
so an ExitPlanMode approval reached clients as an empty {} payload instead
of the plan review. The approval request schema gains a display field, the
live projector and the cold-history wire mapper pass it through, and cold
history backfills a tool frame's display from its linked approval so plan
blocks survive a refresh.
A completed background task delivers its notice into the conversation
as a user message with origin {kind: task}; the v2 stream dropped it,
so clients showed nothing. The projector now maps task.notified into
a user message carrying a structured notification payload — into the
running turn when busy, or the predicted delivery turn when idle —
and cold history rebuilds the same shape from turn.prompt/turn.steer
records by parsing the notification XML. The TaskNotified engine
event carries the source agent id so subagent completions stay
distinguishable.
A task notice delivered into a running turn persists as a context.append_message record, not turn.steer, so a refresh lost the notification user message. Cold history now surfaces task-origin appends (other origins stay hidden).
…tory Steered and task-notification user messages materialize at step boundaries, landing in the gap between one step's end and the next step's begin, where the strict per-step window silently dropped them. Timed users are now assigned to the first step sealed by their time, falling back to the last visible step.
The wire notification keeps the small structured fields and gains the raw XML envelope so clients render the card exactly like v1: clean body, output file chip, output preview, and the raw payload disclosure.
…ages
A skill activation is conversation content, not a system marker. The
system(skill) mark is gone; activations now surface as user messages
with origin {kind: skill}: model-tool and nested activations are
synthesized from skill.activated, user-slash injections arrive
through turn.steer, and prompts with bundled /skill mentions
populate skill_activations on the user message. Cold history mirrors
all three paths (skill turn origins are displayable, bundled
activations rebuild from turn.prompt).
prompt.submitted fires only for user-kind origins and carried no origin, so bundled /skill prompts lost their skill_activations and idle slash activations (skill_activation origin) produced a turn with no opening user message. The event now carries the origin, and the projector synthesizes the turn's opening user message from turn.started when no prompt message exists (cron/task fallbacks are deduped by the per-turn message count).
A model Skill tool call both fires skill.activated and steers the skill instructions into the turn, so synthesizing from the event duplicated the message. All activation paths are covered by the submitted/steered/turn-start messages, so the event emits nothing.
Model-tool and nested-skill activations are engine internals (v1 hides them); only user-slash activations surface as skill-origin user messages, in both the live projector and cold history.
A prompt with bundled /skill mentions persists the engineered skill blocks as leading content parts; cold history rendered them raw into the user bubble. The turn.prompt user text now slices them off like the engine's stripBundledSkillBlocks.
1 participant
Related Issue
Internal protocol refactor — no linked issue. Client adoption: MoonshotAI/kimi-code-app#503.
Problem
The v1 realtime channel carries two coexisting protocol families (legacy event frames and transcript frames), so one logical change has more than one source of truth. Clients cannot consume WS frames directly: every frame has to be applied through the transcript package before it becomes usable, and the frame granularity (off/turn/block/delta) is too low-level, forcing each client to rebuild message semantics on its own. Cold history is served in a different shape from the live stream, so refresh and reconnect recovery are ad hoc.
What changed
Adds a WS v2 protocol stack to kap-server, per
packages/kap-server/docs/2026-09-04-v2-protocol-refactor-plan.md:src/protocol/v2/— zod schemas for the full wire union: turn/step lifecycle, user/assistant/thinking/tool_call/system messages, task/plan/goal/sidechat/cron/attachment/compaction/undo frames, each carrying explicitstatustransitions. This is the only definition of the v2 wire shape; the code-app client re-exports it.src/services/v2Projection/projects engine loop events into wire messages —agentProjectorfor the live stream,coldHistoryfor a rebuild from persisted records. Parity-tested against 24 instance fixtures (test/v2Projection.test.ts).GET /sessions/{id}/historyspeaks the same wire shape as the live stream, paginates viabefore_turn, and reaches past compaction points./api/v2/wswith the hello/subscribe/ack handshake, omit-based unsubscribe, recovery payloads for refresh/reconnect, backpressure, and per-subagent channels.test/v2Engine.e2e.test.ts); v1 stays untouched and serves current clients.steerflag through prompt submission, and the prompt route steers a pending handle within the same request, so the steered user message no longer renders twice); empty thinking parts are skipped in the loop and the projection; goal entries emit only on goal status transitions;before_turnhistory paging crosses compaction points.Checklist
gen-changesetsskill, or this PR needs no changeset (the v2 stack is an internal protocol change; the four user-visible fixes each carry a changeset).gen-docsskill, or this PR needs no doc update (protocol decisions are recorded in the refactor plan doc in this branch).