# Live-Attach Terminal: App vs PoC Demo — Debug Handoff

**Status:** ✅ **RESOLVED**, then **REARCHITECTED IN-BAND (2026-07-02)** — see below.
The PoC at `docs/refactor/ccdemo-poc/` remains the reference oracle.

## REARCHITECTURE (2026-07-02): in-band control-channel transport

The `d1ec56d6` fix (drain timer + out-of-band capture) still left residual races:
the 180 ms drain was a guess (attach repaint could land late → double paint), the
subscribe→capture-pane ordering replayed output that the snapshot already contained
(duplicate spinners on every (re)connect during streaming), and the frontend's manual
fit override fought FitAddon (double resize → repaint storms). Grey lines came from
`-J` (preserves trailing spaces → bg fills become full-width gray bars) plus the
unsanitized live path (Claude Code's neutral bg-237 canvas).

**New design (`terminal_live_attach.go` + `internal/liveattach`):** all transport
commands — `resize-window`, the capture-pane backfill, `#{history_size}`/cursor
queries — are written to the control client's OWN stdin and answered in-stream
between `%begin/%end` guards (`liveattach.Protocol` frames them; verified against
tmux 3.6a: replies are FIFO by command number, serialized with `%output`, ESC bytes
verbatim, `%error` keeps the queue in order). A viewer is spliced into the broadcast
**inside the scanner goroutine at its seed's `%end`**, so the seed and the live
stream can neither overlap nor gap — no drain timers, no replay window, no
out-of-band subprocesses on the hot path. A single writer-pump goroutine owns
control-stdin writes (never hold a lock across that write — it deadlocks against
`deliverReply`). `#{history_size}` is queried first because tmux clamps
`capture-pane -S … -E -1` into the visible screen when scrollback is empty
(otherwise row 0 seeds twice). History capture drops `-J`. Slow viewers are dropped
whole (WS close → reconnect → fresh seed) instead of losing mid-stream chunks.

**Frontend (`LiveAttachXtermPane`):** `fit.fit()` is the single sizing authority
(manual DOM-ruler override deleted — it guaranteed a double resize per layout tick);
the first frame of every (re)connect (the seed) gets `normalizeAnsiForEmbeddedXterm`
(gray-canvas strip) + selection clear, live bytes are written verbatim as raw
`Uint8Array` and never clear the selection; reconnect reschedules itself when the
pane is transiently hidden (used to strand the stream permanently); per-connection
`TextDecoder`.

**Regression net:** `TestLiveAttachRealTmuxEndToEnd` (cmd/server) drives the real
tmux + real WS handler against a continuously-printing pane and asserts the stream is
both duplicate-free AND contiguous (no gap); `TestLiveAttachSeedSpliceExcludesPreSeedOutput`
pins the splice semantics; parser tests pin `%begin/%end/%error` framing.

**Follow-up fix (commit `e19452b1`) — the seed SPLICE POINT.** The first cut spliced the
viewer on the CURSOR query's `%end`, with the screen captured one command earlier. tmux
emits `%output` *between* consecutive commands, so `%output` produced in the
screen→cursor window was lost: not in the snapshot (captured before it), not streamed
(spliced after it). A single dropped byte permanently desyncs xterm's grid → cursor-
addressed redraws land on wrong cells → characters from two logical lines interleave on
one row (the exact corruption reported). It only bites when the pane streams heavily
*during* the connect — e.g. a **scheduled/headless workflow the browser attaches to
mid-run** — so a quick interactive test looked clean. Fix: the current-screen
`capture-pane` is now the LAST seed command and the viewer splices on ITS `%end`, so
every byte is either in the snapshot or streamed, never both, never neither. Cursor is
queried just before the screen (benign momentary staleness, corrected by the first live
redraw). **Decisive diagnostic:** `tmux capture-pane -p -t <session>` showed the pane
content perfectly clean while the browser was garbled+frozen ⇒ transport/render bug, not
the session (the handoff's §8 test).

**Follow-up fix (commit `1fe27d1c`) — seed `2J` erased in-viewport history ("can't
scroll up").** The seed was `RIS → history → \x1b[H\x1b[2J → screen`, with a comment
claiming 2J "clears only the viewport, preserving scrollback". Wrong in the case that
matters: xterm.js ED(2) *erases viewport rows in place* — it only pushes to scrollback
if `scrollOnEraseInDisplay` is set, which we deliberately leave off (live TUI repaints
would stack stale frames into scrollback). Right after the RIS, the *tail* of the
just-written history is still inside the viewport (only lines scrolled past enter
scrollback), so the 2J destroyed up to one screenful of the most recent history on
every (re)connect. For in-place-repaint TUIs like Claude Code — whose tmux history
stays small because completed turns are redrawn, not scrolled — that was often *all*
of it: the pane could not scroll up past roughly the current screen even though
`capture-pane` showed tmux retaining the full render. Fix: drop the clear entirely.
`capture-pane -p` always returns the full pane height, so painting the screen directly
below the history scrolls the history up into xterm scrollback naturally and lands the
screen exactly on the viewport (cursor rows stay aligned); an SGR reset between the two
stops attribute bleed. Unit test now fails if any `2J` reappears in the seed.

---

**Original resolution:** commit `d1ec56d6` "Fix live attach initial repaint
duplication". Root cause + fix below; the rest of this doc is the original investigation
(kept for context).

## RESOLUTION (commit `d1ec56d6`)
**Root cause:** the app's control-mode attach starts **lazily** (on the first browser
subscribe), so tmux's initial full-screen repaint — emitted whenever a control client
attaches — reached the browser *right after* the capture-pane backfill → the same screen
drawn twice → duplicate lines even without resize. The PoC never hit this because its
attach runs at server startup, before any browser, so that repaint broadcasts to zero
subscribers and is dropped.
**Fix:** on first subscribe, warm the attach with no subscriber and **drain tmux's initial
repaint** (`liveAttachInitialDrainDelay` 180 ms / 750 ms cap; `drained` chan +
`waitInitialDrain`/`markInitialDrainComplete`) BEFORE adding the viewer's channel —
replicating the PoC's warm-up. Reordered `handleTerminalStream` to subscribe→backfill→
writer. Added `setSize` dedup (only `resize-window` when the grid actually changed). New
test `TestLiveAttachManagerDrainsInitialAttachBeforeSubscriber`. Builds + live-attach
tests pass. (This is exactly the "backfill + attach initial %output = double" lead flagged
in §6 below.)

---

(Original investigation, pre-fix:) The standalone PoC demo rendered tmux CLIs perfectly;
the real app, using the same approach, showed duplicated lines/stacked frames even without
resizing. This doc captured everything tried.

Branch: `terminal-live-attach-phase1`

---

## 1. What we're building

A live terminal transport that mirrors a **server-side tmux session** (detached,
`new-session -d`) into a browser **xterm.js** pane via **tmux control mode**
(`tmux -CC attach`) over a WebSocket. The backend parses `%output` and streams the raw
pane bytes; the browser writes them to xterm. This replaces an older snapshot/replay
(capture-pane polling) transport.

## 2. The two implementations

### A. PoC demo — WORKS PERFECTLY (the reference)
Location: `/private/tmp/claude-501/-Users-mipl-ai-work/a86ff7c1-184a-484a-90e1-e3bb2d10bdd7/scratchpad/ccdemo/`
- `main.go` — standalone Go server: `tmux -CC attach` under a creack/pty PTY, a
  `bufio.Scanner` reads control-mode lines, decodes `%output` octal escapes, broadcasts
  decoded pane bytes to WS viewers. Backfill = `capture-pane -e -S -1000` prefixed with
  `\x1b[H\x1b[2J` and `\n`→`\r\n`. Resize = `resize-window -x C -y R`.
- `static/index.html` — xterm.js page. `new Terminal({fontSize:13, scrollback:20000})`,
  FitAddon, writes WS bytes to xterm, resize via `term.onResize` + debounced
  `window 'resize'`.
- Runs as two instances:
  - `:8742` → session `ccdemo-live`, `pi --model google/gemini-3.5-flash` (exact app model)
  - `:8743` → session `ccdemo-claude`, `claude` (Claude Code)
- **Result: both CLIs stream smoothly, resize cleanly on the fly, render complex ASCII —
  flawless.**

### B. Real app — BROKEN (duplicate lines, stacking, even at rest)
- Backend: `agent_go/cmd/server/terminal_live_attach.go`
  - `liveAttachManager` / `liveAttachStream`: one `tmux -CC attach` per session
    (subscribe/unsubscribe; designed for 1 viewer/session).
  - `handleTerminalStream` (WS handler `GET /api/terminals/{id}/stream`): backfill via
    `liveAttachBackfill` (capture-pane, same shape as demo), then streams broadcast
    channel; reader loop: binary→send-keys, JSON→resize/input/key.
  - `setSize`: now `resize-window` only (pty.Setsize removed).
  - Parser: `agent_go/internal/liveattach/parser.go` (`DecodeOutput`, `ClassifyLine`).
- Frontend: `frontend/src/components/TerminalCenter.tsx`
  - `LiveAttachXtermPaneInner` (~line 2359): xterm `convertEol:false`,
    `disableStdin:true`, scrollback 20000, FitAddon, **manual** `term.write`,
    `ResizeObserver`-based fit, reconnect-on-close. `key={stableLiveAttachId}`.
  - Render branch (~line 4944): synthetic→`StructuredTerminalView`,
    else `stableLiveAttachId`→`LiveAttachXtermPane`, else empty placeholder.
  - `stableLiveAttachId`: a **debounced** mount id (~line 3886) so transient selection
    flicker doesn't unmount the pane.
- **Result: same content block renders 2+ times; "Working…" spinners stack; content
  mis-wraps. Happens DURING NORMAL STREAMING (no resize), and worse on resize.**

## 3. Symptoms (app)
- **Duplicate lines without resize**: e.g. a whole "STEP 4 — BACK UP FINAL STATE…" block
  appears twice, with different wrapping. ← strongest clue: the *live stream* is being
  rendered more than once, OR backfill overlaps the stream.
- On resize: multiple stacked spinners, mis-wrapped/garbled tables.
- Demo never shows any of this.

## 4. RULED OUT (with evidence)
| Hypothesis | Test | Result |
|---|---|---|
| Stale frontend build / cache | Deleted **all** `node_modules`, reinstalled, restarted `--only-frontend` | App still broken → **not stale** |
| xterm.js version | App is `@xterm/xterm@6.0.0`; switched **demo** to 6.0.0 too | Demo still perfect → **not the version** |
| Model / thinking level | Ran demo on `(google) gemini-3.5-flash • medium`, 1.0M ctx (exact app match) | Demo still perfect → **not the model** |
| Scrollback / content length | Generated 200-line output in demo, then resized | Demo still perfect → **not content length** |
| Write method (AttachAddon vs manual) | Switched demo to manual `term.write` | Demo still perfect → **not the write method** |
| Old capture-pane polling interfering | Gated then ripped out the legacy polls (selected-probe, rail-probe excl. selected, selected-detail, manual-refresh) | App still broken |
| Feature flag / dual-path | Ripped out `RUNLOOP_TERMINAL_LIVE_ATTACH` flag + `XtermTerminalPane`; app is always live-attach for tmux | App still broken |

## 5. NOT yet ruled out (prime suspects)
1. **CLI config/extensions**: the app launches CLIs with extra config the demo lacks —
   e.g. the live gemini workflow runs `gemini --model auto --admin-policy
   …/restrict-tools.toml`; `mcpagent/agent/agent.go:2226,3354` append a system prompt
   "running inside Pi CLI with built-in tools disabled. Use the MCP …" + MCP servers.
   The demo's CLIs are **bare**. *Plan: run the app's exact CLI through the demo's
   transport, OR replicate the config in the demo, to rule this out.*
2. **Backend transport code** (`terminal_live_attach.go`) differs subtly from the demo's
   `main.go`. Look hard at: the shared-attach manager (subscribe/unsubscribe vs demo's
   simpler hub), the broadcast buffer **drop-for-slow-viewer** path
   (`liveAttachSubBuffer`, ~line 271 "Slow viewer: drop"), the scanner buffer sizing,
   and whether **the backfill is being sent more than once** or **the WS reconnects**
   (each reconnect re-backfills → duplicate blocks).
3. **Frontend** (`LiveAttachXtermPane`) differs from `index.html`: React re-renders, the
   debounced remount, the flex/`[&_.xterm]:h-full` container, `disableStdin`.

## 6. STRONGEST lead for "duplicate WITHOUT resize"
A duplicated block during steady streaming almost always means one of:
- the WS **reconnects** and the backend **re-runs the backfill** (capture-pane snapshot
  appended on top of the live stream), or
- **two deliveries** of the same `%output` (double subscribe / two viewers / scanner
  re-emitting), or
- the **backfill cursor desync**: capture-pane backfill leaves the xterm cursor at the
  bottom, and the subsequent cursor-relative `%output` redraws append instead of
  overwrite (but the demo does the *same* backfill and is clean, so this alone
  shouldn't explain it — unless the app reconnects/re-backfills repeatedly).

**Recommended first check:** instrument how many times `handleTerminalStream` runs the
backfill and how many WS connects happen per terminal in the app vs the demo; and
count `term.write` calls per `%output` chunk. The demo connects **once** and stays.

## 7. Changes made this session (app, branch `terminal-live-attach-phase1`)
- Backend `terminal_live_attach.go`: `setSize` → `resize-window` only (removed
  `pty.Setsize`); `liveAttachEnabled()` → always true (flag removed);
  `newLiveAttachManagerIfEnabled` always builds (keeps tmux ≥2.9 guard).
- Frontend: capabilities **retry-with-backoff** + a `main.tsx` entry-point fetch
  (`initializeStores()` was dead code; capabilities never loaded reliably);
  **reverted** a bad cache-buster header that tripped CORS (`Cache-Control` not in
  `Access-Control-Allow-Headers`, 127.0.0.1 vs localhost); debounced `stableLiveAttachId`
  mount; ripped out legacy polling + `XtermTerminalPane` usage + the flag + debug
  scaffolding; experimented with removing then restoring `ResizeObserver` and
  `[&_.xterm]:h-full` (both restored — removing the observer **regressed** at-rest
  rendering because the app's React pane sizes after mount).
- `tsc` and `go build` both clean.

## 8. Suggested next steps for the reviewer
1. **Isolate transport vs CLI-config**: point the demo's `ccdemo` server at the app's
   *real* tmux session (`./ccdemo -s <mlp-… session> -addr …`) and view it — same CLI +
   config through the proven-clean transport. (Avoid two simultaneous `-CC` clients on
   one session; stop the app's attach first.) If it stacks → CLI/config; if clean →
   app transport code.
2. **Line-by-line diff** `terminal_live_attach.go` vs `ccdemo/main.go` (broadcast,
   backfill, subscribe/unsubscribe, scanner) and `LiveAttachXtermPane` vs
   `index.html`.
3. **Count duplicate writes**: log WS-connect count, backfill count, and per-chunk
   `term.write` count in the app; compare to the demo (which is 1 connect, 1 backfill).
4. Verify the app isn't opening **two** `/stream` WS for one terminal (e.g. rail +
   main), or re-subscribing.

## 9. How to run / reproduce the demo

The PoC source is committed at **`docs/refactor/ccdemo-poc/`** and has its own
**`README.md`**. **Keep it** — it is the proven-correct reference/oracle for this
transport. Key files: `main.go` + `listen.go` (the server, package main at root),
`static/index.html` (the xterm page), `launch.sh`/`stop.sh` (helpers).

### Build the demo server
```
cd docs/refactor/ccdemo-poc
go build -o ccdemo .          # main.go + listen.go are package main at the root
```
Deps (`go build` fetches them): `github.com/creack/pty`, `github.com/gorilla/websocket`.

### Run a CLI in a tmux session, then serve it
```
# pi on the app's exact model:
tmux new-session -d -s ccdemo-live -x 120 -y 36 "pi --model google/gemini-3.5-flash"
tmux set-option        -t ccdemo-live status off
tmux set-option        -t ccdemo-live remain-on-exit on
tmux set-window-option -t ccdemo-live window-size latest
./ccdemo -s ccdemo-live -addr 127.0.0.1:8742 &      # open http://127.0.0.1:8742/

# claude-code:
tmux new-session -d -s ccdemo-claude -x 120 -y 36 "claude"
tmux set-window-option -t ccdemo-claude window-size latest
./ccdemo -s ccdemo-claude -addr 127.0.0.1:8743 &    # open http://127.0.0.1:8743/
```
`./launch.sh [session]` is a helper that does the tmux setup for pi/codex automatically.

**GOTCHA:** ccdemo does NOT auto-reattach. If you kill/recreate the tmux session,
**restart the ccdemo server**, or it only shows the backfill on reload (no live stream).

### Config-vs-transport test — point the demo at the APP's real session
With the app backend OFF (so there's no competing `-CC` client on the session):
```
./ccdemo -s <mlp-… app session> -addr 127.0.0.1:8744 &   # open http://127.0.0.1:8744/
```
Clean here **and** duplicated in the app ⇒ app **transport** bug. Duplicated here too ⇒
CLI/**config**. (capture-pane reflects tmux's grid, which is never duplicated, so a clean
`:8744` means the app's render path is duplicating, not the session content.)

### The app
```
./run_server_with_logging.sh --only-frontend   # frontend (vite :51733) + Electron
```
Backend is separate on `:18743` and **must be running** for the live-attach WS. The
Electron renderer caches aggressively — for reliable comparison open a plain browser at
**http://localhost:51733/** instead.

### Servers running in this debug session
- `:8742` — pi `gemini-3.5-flash` (app's model)
- `:8743` — claude-code
- `:8744` — the app's real `mlp-claude-code-…` session through the demo transport