DanCode redesign: drop tmux, two-process backend with disk-persisted scrollback and layouts

[DanielGGordon]

Problem

DanCode currently uses tmux as the backend for persistent shell sessions, and that abstraction is the source of most of the day-to-day pain. The terminfo translation layer (screen-256color) causes formatting glitches; tmux's own scrollback buffer fights with xterm.js's scrollback causing scroll issues; the reconcile-on-startup logic races against tmux-resurrect and silently drops sessions; and the dependency on tmux + tmux-resurrect + tmux-continuum makes the system fragile and hard to reason about. tmux is also the wrong abstraction for the actual use case — what matters is files, Claude Code conversations, and a clean per-terminal experience, none of which need a multiplexer.

Solution

Replace the tmux backend with a two-process node-pty architecture: a long-lived dancode-shellhost daemon owns the PTYs and persists scrollback to disk, while the existing dancode-server handles HTTP, WebSockets, files, and projects. The two communicate over a UNIX socket. This makes ordinary server restarts (npm run dev reloads, crashes) fully invisible to the user, because the shellhost keeps running. Pi reboots still kill PTYs (unavoidable), but on the next open we re-spawn shells at the same cwd, replay the on-disk scrollback for context, and resume the matching Claude Code conversation by session id. A CodeMirror-based editor replaces the current text view. Per-project layout (which terminals existed, which files were open, splits) is serialized to disk and restored on project open.

Requirements

Core features

• A separate dancode-shellhost process owns all PTYs; the main DanCode server connects to it over a UNIX socket and proxies I/O to/from browsers.
• Shellhost runs as a systemd --user service with auto-restart, independent of the DanCode web server.
• Restarting the DanCode web server (e.g. npm run dev reload, crash) MUST NOT kill running shells. The web server reconnects to existing PTYs on startup.
• Restarting the shellhost (only) MUST recover all currently-known terminals by reading their persisted metadata.
• After a Pi reboot, opening a project re-spawns each known terminal at its saved cwd, with its saved startup command, and the prior scrollback visible.
• A "background" mode is available per-terminal that wraps the running command in systemd-run --user --scope so it truly survives a daemon restart (opt-in, for long jobs like builds/training).
• All current functionality preserved: project list, create/rename/delete project, file browser, view/edit files, split panes, tabs within a project, mobile UI, auth (bcrypt + TOTP + 30-day session).

Terminal behavior

• Each terminal is a node-pty process spawned with TERM=xterm-256color (no multiplexer in between).
• Per-terminal scrollback log appended to ~/.dancode/terminals/<id>/scrollback.log as the PTY produces output. File is rotated at 1MB; only the most recent rotation is kept (so disk usage per terminal is bounded ~2MB worst case during rotation).
• On socket reconnect (browser refresh, network blip), the last N KB of scrollback log is replayed to the client before live output resumes.
• After PTY death + respawn (e.g. post-reboot), the scrollback log from the prior session is replayed as a banner ("--- prior session ended at ---") and then live output starts.
• Resize events propagate to the PTY immediately.
• Paste of plain text fires exactly once. Paste of image data triggers upload to project (current image-paste behavior preserved).
• No bracketed-paste double-emit, no duplicated input, no terminfo translation glitches.

Project & layout persistence

• Each project stores a layout.json at ~/.dancode/projects/<slug>/layout.json describing:
  • Ordered list of terminals: id, cwd (relative to project root), startup command (if any), claude-session-id (if any), background-mode flag.
  • Ordered list of open files: path (relative to project root), pane index, scroll position.
  • Split/tab structure of the workspace.
• Layout is updated atomically on every meaningful change (terminal created/closed/moved, file opened/closed, split changed).
• Opening a project reads layout.json and reconstructs the workspace. Missing files are surfaced as a warning, not a crash.
• Renaming a project updates the slug AND moves the on-disk project directory (closes [[project_rename_bug]]).

Claude Code awareness

• DanCode detects when a terminal is running claude (by spawned-command introspection on the PTY's foreground process, or by scanning ~/.claude/projects/<project-slug>/ for an active session).
• When a Claude session is detected, the terminal's layout.json entry records the claude-session-id.
• On respawn after Pi reboot, the terminal's startup command is claude --resume <session-id> rather than a bare shell, so the prior conversation is restored.
• A "Resume last Claude session" UI affordance on the terminal pane offers manual resume.
• No native chat UI — Claude still renders inside the terminal.

File editor

• Replace the current view/edit with CodeMirror 6.
• Syntax highlighting for: JavaScript/TypeScript, JSX/TSX, Python, JSON, Markdown, YAML, Bash, HTML, CSS. (Extensible — add more languages via @codemirror/lang-*.)
• Standard editor affordances: find/replace, undo/redo, multi-cursor, line numbers.
• Save: explicit (Ctrl+S) and on-blur. Server validates path is inside the project root before writing.
• No LSP, no diagnostics, no completions (out of scope).

Migration from tmux

• A one-shot bin/dancode-migrate-from-tmux script that:
  • For each existing dancode-* tmux session, captures the current pane contents via tmux capture-pane -p -S - into the new scrollback log format.
  • Writes a meta.json per terminal with cwd extracted from tmux display -t <session> -p '#{pane_current_path}'.
  • Builds an initial layout.json per project from the captured terminals.
  • Kills the tmux sessions on success.
  • Removes the legacy ~/.dancode/terminals/*.json files.
• Migration is idempotent and safe to run multiple times.
• After migration, the codebase removes all tmux-related code (tmuxHasSession, session-name builders, reconcile retry logic, etc.) and removes tmux from documented system dependencies.

Wire protocol (shellhost ↔ server)

• UNIX socket at ~/.dancode/shellhost.sock.
• Length-prefixed JSON or msgpack frames. (Decide during implementation.)
• Commands: spawn, attach, detach, write, resize, kill, list, inspect.
• Events from shellhost: output, exit, respawn-needed.
• All commands carry a request id; responses correlate. Disconnect from the server side does not kill PTYs — only an explicit kill does.

AI integration / self-modification

• DanCode itself remains AI-editable via Claude Code running in a DanCode terminal (eat-your-own-dogfood). No new in-app AI surface is needed beyond what's in scope for Claude awareness above.

UX / interface

• No visible UI changes for the steady-state user beyond:
  • Fewer formatting bugs in terminals.
  • Smooth scrolling that doesn't fight tmux.
  • A small "Resume Claude" button on terminals where Claude was last running.
  • Layout actually restores on project open.
• Existing mobile UI, shortcut bar, image paste, and drag-and-drop file upload preserved.

Data / persistence

• ~/.dancode/projects/<slug>/layout.json — per-project layout state.
• ~/.dancode/terminals/<id>/meta.json — per-terminal metadata (project slug, cwd, command, claude session id, background flag, last-active timestamp).
• ~/.dancode/terminals/<id>/scrollback.log — append-only output stream, rotated at 1MB.
• ~/.dancode/credentials.json — unchanged (bcrypt hash + TOTP secret).
• ~/.dancode/sessions.json — unchanged (server-side 30-day session tokens).
• ~/.dancode/shellhost.sock — runtime socket; not data.

Testing

• Vitest workspaces (server, client) — already in place.
• New unit tests for:
  • Scrollback writer (append, rotate, replay correctness).
  • Layout serializer (round-trip, atomic write, partial-corruption recovery).
  • Shellhost wire-protocol framing.
  • Shellhost: spawn, attach, detach, kill, output streaming.
• New integration tests for:
  • Server-restart scenario: spawn terminal → kill+restart server process → attach → verify alive.
  • Shellhost-restart scenario: spawn → kill shellhost → restart → meta survives → respawn.
  • Migration script against a fixture set of tmux sessions (using a disposable tmux server in CI).
• Existing 230/232 server tests should pass at the same baseline (or better, with the reconcile race tests removed).
• Manual smoke test plan: paste single + double + image, large output scrollback, splits, tabs, project rename, Claude resume, simulated reboot via systemctl --user restart dancode-shellhost.

Technical decisions

• Stack: Node 20+, Express 5, Socket.IO, node-pty, CodeMirror 6, Vitest. Same as today minus tmux + tmux-resurrect + tmux-continuum. Add systemd --user unit file for the shellhost.
• Architecture: Two processes — dancode-server (web/HTTP/WS, file ops, auth, projects) and dancode-shellhost (PTY ownership, scrollback writers). UNIX-socket RPC between them. Browser only talks to dancode-server; never to shellhost directly.
• Data model:
  • Project: slug, name, root path, created/updated timestamps.
  • Terminal: id (uuid), project slug, cwd, startup command, claude-session-id, background flag, last-active.
  • Layout: project slug, terminals (ordered), open files (ordered), splits/tabs structure.
  • Session: token, user, created, expires (auth — unchanged).
• API surface (HTTP, unchanged shape, semantics shift):
  • GET /api/projects, POST /api/projects, PATCH /api/projects/:slug, DELETE /api/projects/:slug.
  • GET /api/projects/:slug/layout, PUT /api/projects/:slug/layout.
  • GET /api/projects/:slug/files, GET /api/projects/:slug/files/*, PUT /api/projects/:slug/files/*.
  • POST /api/projects/:slug/upload (image paste — unchanged).
  • GET /api/terminals?projectSlug=, POST /api/terminals, DELETE /api/terminals/:id.
  • WebSocket /terminal/:id proxies to shellhost.
  • Auth endpoints unchanged.
• API surface (UNIX socket, new): command/response with request ids, see Wire protocol above.
• Testing: Vitest unit + integration. Shellhost has its own test file. Migration script tested against fixture tmux sessions.
• Process model in dev: npm run dev starts dancode-server and dancode-shellhost concurrently (replacing the current concurrently config). On the Pi, shellhost lives in systemd --user; web server lives in either dev or systemd.

Out of scope

• LSP, IntelliSense, completions, diagnostics.
• Native Claude Code chat UI (Claude still runs in the terminal).
• Drag-to-rearrange layout, saved layout presets.
• Multi-user — DanCode remains single-user.
• True process survival across Pi power loss (impossible). Re-spawn-with-resume is the offered guarantee.
• Remote / cloud deployment. DanCode stays self-hosted on the Pi.
• Migration from anything other than the current tmux-based DanCode.

Open questions

• Wire-protocol encoding: length-prefixed JSON vs. msgpack vs. CBOR. Decide during planning based on size / Pi CPU profile.
• Scrollback log format: raw bytes vs. timestamped chunks. Decide during planning — timestamps cost ~10% disk but enable "scrollback at time X" UX later.
• Whether systemd-run --user --scope is sufficient for "background mode" or if we need a small supervisor inside shellhost.
• Handling of the existing 10 known terminals during migration — should the migration script run automatically on first server start with the new code, or be invoked manually? Probably manual, with a one-liner in the upgrade docs.