Chat Runtime, Conversation & Cost

Chat Runtime, Conversation & Cost

This page covers the provider-agnostic core of chat: src/core/chat/runtime.ts, src/core/chat/conversation.ts, and src/core/chat/cost.ts. None of these files import Electron, touch the filesystem, or know which provider is in use. The same createChatRuntime() is instantiated by the Electron main process and by the Server Edition, each supplying its own ChatRuntimeDeps (provider resolution, event broadcast, tool set, driver factory). Everything provider-specific lives behind the injected ChatDriver, implemented by openai.ts / anthropic.ts and documented on the Model Provider Adapters & Streaming page; the loop that consumes those drivers lives in tools.ts (see Chat Tool Calling & Project Tools).

Run mechanics: one active run per node

createChatRuntime(deps) returns a ChatRuntime with four methods: send, stop, approve, isBusy. The state behind them is deliberately small:

• activeRuns is a module-level Map<string, AbortController> keyed by nodeId. It is not per-runtime-instance, so two runtimes created in the same JS realm (it happens in tests) share the busy map.
• pendingApprovals is a closure-local Map<string, resolver> keyed by `${nodeId}:${callId}`.

send(req) is the only entry point that starts work. Its order of operations matters:

1. If activeRuns.has(req.nodeId), return { ok: false, error: 'chat already running for this node' }. There is no queueing and no coalescing — concurrency is rejected.
2. deps.resolveProvider(req) must return a ChatProviderConfig. null returns { ok: false, error: 'no chat provider configured …' }. The runtime does not read settings or env itself; the contract (per the file header) is that env wins over settings, the same rule the Telegram token uses.
3. Register an AbortController in activeRuns before any I/O, so stop() and isBusy() are correct from the first tick.
4. Build the driver (deps.driverFor?.(cfg) ?? driverFor(cfg)) and the tool list (deps.toolsFor?.(req) ?? [] — no tools by default).
5. Resolve the model with req.model ?? cfg.model ?? '' and hand [...req.messages] to runChatLoop. Note the shallow copy: the loop may append tool/assistant messages without mutating the caller's array, but individual ChatMessage objects are shared by reference.

Note that the two early-exit paths return before the try, so they broadcast no done event. A caller that already rendered an "in progress" state must surface those errors itself.

The send/approve/tool-result cycle

sequenceDiagram
  participant UI as Chat node UI
  participant RT as ChatRuntime.send
  participant Loop as runChatLoop (tools.ts)
  participant Drv as ChatDriver (openai/anthropic)
  UI->>RT: send(req)
  RT->>RT: busy check, resolveProvider, activeRuns.set(nodeId, controller)
  RT->>Loop: runChatLoop(driver, model, messages, tools, hooks)
  Loop->>Drv: stream(opts)
  Drv-->>Loop: streamed chunks / usage
  Loop-->>RT: onEvent(e) — every kind except 'done'
  RT-->>UI: deps.broadcast(nodeId, e)
  Loop->>RT: requestApproval(call)
  RT->>RT: pendingApprovals.set(nodeId:callId, resolve)
  RT-->>UI: broadcast toolCall (permission card)
  UI->>RT: approve(nodeId, callId, decision)
  RT->>Loop: resolver(decision) — entry deleted from the map
  Loop->>Drv: stream again with the tool result re-entered
  Loop-->>RT: { stopReason }
  RT-->>UI: broadcast done(reason)
  RT->>RT: finally: activeRuns.delete(nodeId)

Loading

Key nodes:

• onEvent filter. The runtime forwards every event kind except done, because it emits its own done after the loop settles, with a reason it computes itself. The ChatEvent union itself is declared in ./types alongside the message/tool contracts.
• Approval is a promise, not a callback. requestApproval returns a Promise<'approve' | 'deny'> whose resolver is parked in pendingApprovals; the loop blocks until approve() is called. The runtime broadcasts the toolCall event itself, and the comment at the call site warns that the loop also re-broadcasts the call — the renderer is expected to dedupe by call id. approve() is a no-op for an unknown key, so duplicate or late clicks are harmless.
• Tool-result re-entry is entirely the loop's business: the runtime never inspects tool calls or results, it only shuttles the approval decision back.

Terminal outcomes and cancellation

stop(nodeId) only calls abort() on the controller; the activeRuns entry survives until send's finally. A user stop is treated as a normal outcome, not an error:

Situation	done reason broadcast	ChatSendResult
Signal aborted after the loop returned	stopped	{ ok: true, stopReason }
Loop returned stopReason === 'max_iterations'	max_iterations	{ ok: true, stopReason }
Loop returned normally	end_turn	{ ok: true, stopReason }
Driver threw AbortError	stopped	{ ok: true, stopReason: 'stopped' }
Any other throw	error	{ ok: false, error: message }

The finally block deletes the run entry and logs chat <nodeId> finished in <ms> through deps.log. Two ordering details are worth knowing before you build on this:

• The done broadcast happens inside the try, before finally removes the entry. A UI that reacts to done by immediately calling send can observe isBusy(nodeId) === true for a brief window and get the "chat already running" error.
• pendingApprovals entries are only removed by approve(). If a run ends (abort, error, or loop return) while an approval card is still outstanding, its resolver stays in the map; a later approve() for the same node/call id resolves an already-settled promise and is silently dropped. There is no sweep on run teardown.

stateDiagram-v2
  [*] --> Idle
  Idle --> Running: send() stores AbortController in activeRuns
  Running --> AwaitingApproval: requestApproval(call) parks resolver
  AwaitingApproval --> Running: approve()/deny() deletes resolver and resolves
  Running --> Terminal: loop returns
  AwaitingApproval --> Terminal: loop settles after abort
  Terminal --> Idle: finally deletes activeRuns entry
  note right of AwaitingApproval
    Resolvers are only removed by approve();
    send()'s finally does not purge them.
  end note
  note right of Terminal
    done is broadcast before the finally runs,
    so isBusy can still be true right after done.
  end note

Loading

Conversation state

conversation.ts owns one chat session's message list: Conversation { id, messages, createdAt }, created by createConversation(). It has zero dependencies and no I/O, which is what makes it usable from both the runtime side and the renderer-side owner of the message list.

Mutation API. appendMessage(role, content) pushes a fully formed message. The streaming appliers — appendDelta, appendThinking, setUsage, markStopped — all route through ensureMessage, which creates a placeholder { role: 'assistant', content: '', ts } when the id is unknown. That means event ordering is not a correctness requirement: a delta can arrive before the message it belongs to exists, and it will be materialized. setUsage stamps msg.usage and, when supplied, msg.model — the two fields cost.ts later reads.

The note role. appendNote writes local-only annotations (the comment names /cost output as the example). Notes persist with the transcript but are never mapped onto the wire — the comment records the reason: a system-role note silently vanished from Anthropic's context (audit B4). isValidRole accepts user | assistant | system | tool | note, and deserializeConversation uses the same predicate, so any new role must be added in both places.

Slash commands. detectSlashCommand is a pure parser for /clear, /model <id>, /system <text>, /cost. The command word is case-insensitive; a missing leading slash or an unknown word returns null. It only detects — execution (clearing the list, switching model, inserting a system prompt) belongs to the caller.

Persistence and the history window. serializeConversation emits { v: 1, messages }; capConversationMessages is exported separately so node-data persistence (project.json) can apply the same window without going through the serializer (audit B5). The cap drops oldest user+assistant pairs and always keeps a leading system message.

flowchart TD
  A[messages] --> B{JSON.stringify v:1 messages fits maxBytes?}
  B -- yes --> C[return messages unchanged]
  B -- no --> D[start = 1 if messages[0] is system, else 0]
  D --> E{start + 1 < capped.length?}
  E -- no --> F[return capped, still over budget]
  E -- yes --> G{capped[start] is user and capped[start+1] is assistant?}
  G -- yes --> H[drop the pair]
  G -- no --> I[start += 1]
  H --> J{fits maxBytes?}
  I --> J
  J -- yes --> K[return capped]
  J -- no --> E

Loading

Consequences you should know before changing the cap:

• The budget is compared against String.length of the serialized JSON, i.e. UTF-16 code units, despite the maxBytes name and the 200 KiB default. Non-ASCII transcripts consume more real bytes than the check accounts for.
• If nothing is droppable (a lone system message, or a leading run that never lines up as a user+assistant pair at start), the loop advances start to the end and returns an over-budget array rather than looping forever.
• serializeConversation writes only { v, messages } — it does not persist conv.id or conv.createdAt. A serialize→deserialize round-trip therefore synthesizes fresh values when they are absent.

Deserialization. deserializeConversation throws TypeError for invalid JSON, non-object payloads, a non-array messages, and messages missing id/valid role/content/ts. Everything else is copied verbatim so unknown fields survive the round-trip. That tolerance is one-directional: nested values such as usage, thinking, or tool-call payloads are not deep-validated, and cost.ts assumes usage.inputTokens/outputTokens are numbers.

Token and cost accounting

cost.ts is pure and has no knowledge of providers. DEFAULT_PRICES maps model prefixes to USD per million tokens (gpt-, gpt-4o, claude-, claude-3-5-haiku, deepseek). priceFor(model, overrides) tries an exact override hit first, then scans overrides and defaults together and keeps the longest matching prefix. Two implications:

• Overrides participate in prefix matching, so a short override prefix (say claude-) loses to the longer built-in claude-3-5-haiku entry; a specific model override needs a prefix at least as long as the longest competing default.
• No match returns null. costOf turns that into { usd: 0, estimated: true } so the UI can render "n/a" instead of a fabricated number.

conversationCost(messages, model, overrides) sums assistant messages that carry usage, using m.model ?? model per message. Be precise about the flag: estimated is true only when no message had a known price. A conversation mixing priced and unpriced models reports estimated: false with a total that undercounts the unknown ones.

Model price overrides are an input parameter here; the settings key that supplies them is outside these three files.

File responsibilities and how they connect

• runtime.ts is the orchestrator and the only stateful piece: run registry, abort, approval plumbing, provider/driver/tool resolution, terminal event emission. It knows about ChatEvent, ChatMessage, and the ChatDriver/ChatToolDef interfaces but not about HTTP, OpenAI shapes, or tool schemas.
• conversation.ts is the session model and the persistence format. It is where streamed bytes become messages, where the byte-capped window is enforced, and where slash commands are recognized. It is the authority on what a message may contain.
• cost.ts is a pure function library over { usage, model } pairs produced by the conversation model and populated from provider usage reports.

They connect in one direction: a caller passes messages into ChatRuntime.send; the runtime broadcasts stream events; the message-list owner applies them with the conversation.ts appliers, which stamp usage/model; cost.ts reads those stamped fields (directly, or via /cost, which lands as a note message). Provider adapters and the tool loop are the two injected seams that keep the runtime provider-agnostic.

Extension points

• New provider shape. Extend driverFor's api switch, or inject deps.driverFor to bypass the built-in adapters entirely. The new adapter must honor signal and surface aborts as either a done event or an AbortError — the runtime explicitly handles both.
• New tool set. Supply deps.toolsFor(req); returning [] disables tool calling for a send. Approval prompts only appear for tools the loop decides to gate through requestApproval.
• New slash command. Add to SlashCommandName, the switch in detectSlashCommand, and the caller that executes commands; nothing in the runtime needs to change.
• Different history budget. Pass maxBytes to serializeConversation/capConversationMessages. Remember persistence paths that call the helper directly.
• New pricing. Add a prefix to DEFAULT_PRICES or pass an override table long enough to win the longest-prefix comparison.
• Testing. deps.driverFor, deps.toolsFor, deps.resolveProvider, and deps.broadcast are all injectable, so the whole loop can be driven by a fake driver with no network and no filesystem.

Limits of this page

tools.ts, types.ts, openai.ts, and anthropic.ts are visible here only through the interfaces runtime.ts imports (runChatLoop, ChatDriver, ChatToolDef, ChatError, ChatEvent, ChatMessage, streamOpenAI, streamAnthropic). The exact ChatEvent union, the iteration/max_iterations policy, the tool schema shape, and SSE/thinking parsing are covered on the Chat Tool Calling & Project Tools and Model Provider Adapters & Streaming pages. The renderer-side owner that calls the conversation appliers and reacts to the approval-card dedupe rule is covered by Chat Node UI. The source of price overrides in settings is not determinable from these files.

Sources: src/core/chat/runtime.ts, src/core/chat/conversation.ts, src/core/chat/cost.ts