feat(cli): CLI feature parity phase 2 - usage tracking and streaming (AI-assisted)

[rmorse]

Summary

This PR continues CLI backend improvements from #1921, adding accurate token usage tracking and real-time streaming support.

Concurrency Hardening

Fixes race conditions in CLI transcript writes:

• Uses { flag: 'wx' } for atomic file creation (TOCTOU fix)
• Adds session-level locking for concurrent-safe writes
• New async API: appendMessageToTranscriptAsync, appendAssistantMessageToTranscriptAsync
• Tests for partial failure (orphaned user message) and concurrent writes

Token Usage Tracking

⚠️ WIP: Token display is improved but not fully resolved - some edge cases remain

Fixes incorrect token display in UI (showed 558k/200k when actual was ~120k):

• Adds usage parameter to appendMessageToTranscript functions
• Passes result.meta.agentMeta?.usage when persisting assistant messages
• Creates CliSessionManager class with SDK-aligned API for future use
• Transcript entries now contain actual input/output/cache token counts

Configurable Usage Fields

Enables per-backend token field parsing:

• Adds usageFields config to CliBackendConfig
• Handles different API response formats (Anthropic vs OpenAI field names)
• Correctly parses cache_creation_input_tokens (previously missed)
• Maintains backwards compatibility via fallback defaults

Streaming NDJSON Support

⚠️ TODO: Non-streaming path (streaming: false) is currently broken - needs fix before merge

Adds real-time output for CLI backends:

• New cli-runner/streaming.ts module with readline-based NDJSON parsing
• Emits events as they arrive instead of waiting for full response
• Config options: streaming?: boolean, streamingEventTypes?: string[]
• Event mapping for Claude CLI (text, tool_use, result) and Codex CLI (item.*, turn.*)
• Debug logging throughout pipeline for production diagnostics

Reply Directives for Streaming

Matches embedded flow's text processing:

• Applies parseReplyDirectives to streaming text events
• Extracts media URLs, cleans directives, computes delta from cleaned text

Test plan

• Unit tests for CliSessionManager (17 tests)
• Unit tests for session-utils.fs usage parameter
• Unit tests for CLI streaming module
• Unit tests for agent-runner-execution CLI persistence
• Tested locally with claude-cli backend - tokens display improved
• Verified streaming events emit in real-time
• TODO: Fix and test non-streaming path
• TODO: Verify token display edge cases resolved

AI-assisted

This PR was developed with AI assistance (Claude). The code has been tested locally with a live Clawdbot instance. I understand what all the code does.

[rmorse]

Some things worth discussing / checking:

New backend args

systemPromptArg: "--system-prompt",
usageFields: {
    input: ["input_tokens", "inputTokens"],
    output: ["output_tokens", "outputTokens"],
    cacheRead: ["cache_read_input_tokens", "cached_input_tokens", "cacheRead"],
    cacheWrite: ["cache_creation_input_tokens", "cache_write_input_tokens", "cacheWrite"],
    total: ["total_tokens", "total"],
  },
  streaming: true,
  streamingEventTypes: ["tool_use", "tool_result", "text", "result"],
  streamingFormat: {
    text: {
      eventTypes: ["assistant"],
      contentPath: "message.content",
      matchType: "text",
      textField: "text",
    },
    toolUse: {
      eventTypes: ["assistant"],
      contentPath: "message.content",
      matchType: "tool_use",
      idField: "id",
      nameField: "name",
      inputField: "input",
    },
    toolResult: {
      eventTypes: ["user"],
      contentPath: "message.content",
      matchType: "tool_result",
      idField: "tool_use_id",
      outputField: "content",
      isErrorField: "is_error",
    },
  },

• usageFields - defines what fields are used to extract context usage from the reponse json
• streaming + streamingEventTypes - enable streaming for the backend, and which entries are considered as events we want to capture
• streamingFormat - this might need some work - tried to figure out a cli agnostic way to define how to the parse the stream json result and extract the relevant data
• systemPromptArg - changed Claude's system prompt arg to "--system-prompt" , which should replace any existing prompts - but, on my max sub at least, --append-system-prompt and --system-prompt work the same - they only append.

Questions

1. If the cli backend wasn't fully implemented, do we need backwards compat?
2. I made a CliSessionManager class (unused) to roughly match the api surface of the pi SessionManager class - but its going to need updates across multiple files - should we go with it, or leave as is with our modifications in src\gateway\session-utils.fs.ts instead?
3. For streaming, we re-use parseReplyDirectives, output is a bit inconsistent, but better re-use what we already have?
4. Token usage for displaying to the user - having a bit of hard time getting this right, using the existing calculations everythinng is coming out way off, adding a custom calculation works (matches claude codes reported context usage) - is this what we want?
   • Are the original calculations for embedded functioning/accurate? Seems its not (or direct api usage works differently to CC cli with sub at least)

[rmorse]

Sorry for the tag @steipete but thought I better get your eyes on this before doing any more (see "questions" above).

[rmorse]

Token Calculation Challenges

While implementing CLI token display, we ran into some semantic ambiguity around what "tokens" should mean in different contexts.

The Problem

Claude CLI returns different token fields with different semantics:

{
  "input_tokens": 2,
  "cache_creation_input_tokens": 3538,
  "cache_read_input_tokens": 52381,
  "output_tokens": 5
}

Meanwhile, the Anthropic API (embedded flow) returns:

{
  "input_tokens": 1200,
  "output_tokens": 340,
  "cache_creation_input_tokens": 200,
  "cache_read_input_tokens": 50,
  "total_tokens": 1790
}

Semantic Differences

Metric	Formula	Purpose
API total_tokens	input + output + cacheRead + cacheWrite	Billing - all tokens consumed
"Context" for display	input + cacheRead + cacheWrite	What's in the context window (excludes output)

Current Implementation

For CLI providers, we now calculate context as cacheRead + cacheWrite + input (what's in the context window this turn).

For embedded/API, we use the same formula but the API also provides total_tokens which includes output.

Questions

1. What should the UI's "tokens" display represent?

   • Context tokens (what's in the window) = input + cacheRead + cacheWrite
   • Billing tokens (all consumed) = input + output + cacheRead + cacheWrite

2. Should we add a separate contextTokens field to NormalizedUsage to distinguish from billing total?

3. Are there other consumers of totalTokens that expect specific semantics?

The current fix makes CLI and embedded show similar context-based values, but wanted to flag this architectural question for review.

[sebslight]

Closing: This PR is marked as WIP/Draft and has been open without completion. Please reopen when the work is ready for review.

[steipete]

@sebslight do not close good Prs!