cockpit

cockpit is a Python supervisor that builds a tmux-based Linux operations cockpit for remote tmux sessions. It gives you one visible host window at a time, a live top-half view of remote tmux sessions on that host, and a real bottom-half Codex CLI paired to the selected remote session.

What It Does

• Polls remote tmux state every 2 seconds over ssh ... tmux ...
• Discovers new remote tmux sessions automatically
• Creates one persistent local Codex agent per remote tmux session
• Preserves agent context across restarts by relaunching in the same workdir and preferring codex resume --last
• Enforces human vs agent write ownership with a hard block on agent writes while the operator owns the remote pane
• Exposes a scoped MCP server for the paired session so Codex can inspect and control the remote session without relying on prompt injection
• Includes a fully local demo mode backed by separate tmux sockets

Local Requirements

• Linux
• Python 3.11+
• tmux
• ssh
• codex

Remote hosts only need:

• ssh
• tmux

Setup

1. Create a virtualenv and install the project:

   python3 -m venv .venv
   .venv/bin/pip install -e '.[dev]'

2. Install the Playwright browser runtime once for Google login automation:

   .venv/bin/python -m playwright install chromium

3. Check the environment:

   .venv/bin/python -m cockpit.cli doctor

First Run

Launch the app with one command:

.venv/bin/cockpit

On the first run, cockpit opens an interactive setup flow in the terminal instead of making you edit YAML by hand. It asks for:

• the SSH aliases to supervise, using an add/edit/remove loop
• the Google accounts to use for Codex login and rotation, also using an add/edit/remove loop
• whether to log into those accounts immediately

For normal hosts, the setup path only needs the alias you already use with ssh, such as ai. Cockpit fills in ssh_target=ai automatically and detects hostname, CPU count, RAM, and GPUs from the remote machine after it connects.

The runtime config is written to:

~/.local/share/cockpit/config.yaml

The file is written with mode 0600 when possible because it contains Google account passwords for automated login.

If you prefer to re-run setup explicitly:

.venv/bin/cockpit setup

config/hosts.example.yaml remains as an example only.

Main Commands

Start the cockpit:

.venv/bin/cockpit

If the outer tmux layout is partially damaged, panes have died, or the configured host list changes, rerunning cockpit up reconciles the visible host windows and restarts the hidden supervisor so config changes take effect immediately. While running, the supervisor also repairs outer host panes, recreates missing inner viewer or agent sessions, recreates the inner viewer or agent tmux servers if those local servers disappear, and restores inner viewer panes and per-session agent tmux panes if they drift into the wrong process.

If a remote host's tmux server disappears entirely, Cockpit treats that host as having no remote sessions, archives the affected viewers and agent panes, and reattaches automatically when remote tmux sessions come back.

Tear it down:

.venv/bin/cockpit down

Check runtime state:

.venv/bin/cockpit status

cockpit status reports the selected host, a concise per-host last_error, and per-session account/ownership details. When account rotation is degraded, pending retry attempts and backoff state are shown there too.

Run setup again:

.venv/bin/cockpit setup

Log into configured Google accounts:

.venv/bin/cockpit login-accounts
.venv/bin/cockpit login-accounts --headless

Run the local demo:

.venv/bin/cockpit demo
.venv/bin/cockpit demo-clean

Helper scripts are also provided:

• scripts/launch_cockpit.sh
• scripts/install_tmux_keybindings.sh
• scripts/viewer_wrapper.sh

Operator Keys

cockpit up installs these bindings into the dedicated outer tmux socket for the active data directory:

• F1: next host
• Shift-F1: previous host
• F2: next remote tmux session on current host
• Shift-F2: previous remote tmux session
• F3: toggle focus between remote viewer and Codex agent
• F5: toggle supervisor auto-wake on or off for the currently selected host
• F12: tear the whole cockpit down, equivalent to cockpit down

Changing hosts keeps the same top/bottom focus mode across hosts. If you switch hosts while the remote pane is focused, human write ownership is relinquished on the old host and transferred to the newly selected host's remote session instead of staying attached to a hidden background host. While focus is on the remote pane, changing sessions with F2 transfers human write ownership to the newly selected session on that same host. Session cycling wraps within the current host, so Shift-F2 on the first session moves to the last session and preserves remote focus ownership. If you switch to a host that currently has no remote sessions while remote focus is active, Cockpit keeps that empty host selected in remote focus and automatically claims the first returning session for the operator. F3 also works on an empty selected host: it toggles between the top and bottom placeholder panes and controls whether a later returning session comes back blocked for the operator or ready for the agent. F5 toggles the local supervisor wakeup loop for the currently selected host only. When it is off, idle nudges are suppressed for that host, the setting persists across restart, and the status line shows Wake off in red.

Equivalent CLI controls:

• cockpit ctl next-host
• cockpit ctl previous-host
• cockpit ctl next-session
• cockpit ctl previous-session
• cockpit ctl toggle-focus
• cockpit ctl toggle-wake
• cockpit down

Layout and Runtime

• Outer tmux server: one session named cockpit, on a dedicated socket derived from the configured data directory
• Inner viewer tmux server: one session per host, one pane per discovered remote tmux session
• Inner agent tmux server: one session per remote tmux session, running a real interactive Codex CLI

Persistent runtime data lives under the configured data directory, including:

• SQLite runtime state
• global config
• account homes and auth state
• per-agent workdirs
• snapshot and audit files

By default the data directory is:

~/.local/share/cockpit

Each agent workdir is:

~/.local/share/cockpit/agents/<host_alias>/<remote_session_id>/

Cockpit only manages remote tmux sessions whose names start with cockpit-. On first discovery of an empty host, Cockpit will create a detached cockpit-main session automatically and then pair it with a local Codex agent. For real SSH hosts, Cockpit will also recreate cockpit-main later if the host is empty again while it stays in agent focus.

Cockpit's SSH transport defaults to ForwardAgent=yes, so the visible remote tmux attach path carries your forwarded SSH agent into the remote host by default. If you reconnect to this machine with a different forwarded agent, rerun cockpit up from that shell to refresh SSH_AUTH_SOCK inside Cockpit's tmux runtime.

Codex and MCP

Each local supervisor agent launches Codex with model gpt-5.4-mini by default and injects the Cockpit MCP server definition at launch time. You can override that with top-level local_agent_model: in the config if you want an even cheaper or stronger local supervisor. The example config snippet in .codex/config.toml.example is useful if you want the same server configured explicitly for manual sessions.

Each remote session also gets:

• a stable agent HOME under ~/.local/share/cockpit/homes/<session_key>/
• a selected Google account key
• a linked ~/.codex/auth.json pointing at the current account home
• a stable logical identity keyed by host alias plus remote tmux session ID, so a tmux session rename updates labels without creating a new paired agent
• if a session dies and a later tmux session reuses the same name with a new tmux session ID, Cockpit treats it as a new logical session and creates a new paired agent
• if a host goes completely empty and a later tmux server reuses the same tmux session ID, Cockpit treats that as the same logical session because the tmux session ID is still the strongest available identity, so the paired agent mapping is reused

Configured Google accounts live in ordered priority. When the visible Codex pane reports the same token-exhaustion markers that multishell uses, Cockpit:

1. moves that exhausted account to the end of the configured list
2. tries replacement accounts in that rotated order until one is ready
3. re-links the affected agent to that account
4. restarts that one Codex session

Exhaustion deduping is tracked per (session, account, signature), so the same visible out_of_tokens text can still advance from account A to account B to account C without getting stuck on the first handled signature. If the next account is temporarily unavailable, Cockpit keeps retrying that session's account rotation quietly with backoff and resumes automatically once the replacement account is ready. Those pending rotation retries are persisted, so cockpit down / cockpit up does not lose them.

The automatic login flow is deterministic browser automation adapted from multishell, but it does not use any local hosted model for login decisions.

Agent-local files include:

• bootstrap.md
• metadata.json
• status.json
• snapshot.txt
• settled_delta.txt
• remote_codex_supervisor.md
• remote_login_workflow.md
• remote_codex_launch.md
• remote_login_accounts.json
• memory.md
• audit_tail.json

The local agent is intentionally framed as a cheap supervisor for the remote codex session, not a second engineer duplicating the task. By default it should trust the remote worker to be better at the main task, monitor it, nudge it, relaunch it, or re-authenticate it, and otherwise stay out of the way. When it relaunches remote codex, the written policy now tells it to use codex resume --last --model gpt-5.4 --dangerously-bypass-approvals-and-sandbox first, and to treat approval-gated remote launches as a misconfiguration to repair. The local Codex pane now launches with the same dangerous-bypass flag, so the local supervisor itself is not stalled on approval prompts either. If a local Codex supervisor has been idle at a prompt for 30 seconds, Cockpit nudges it to review the paired remote terminal and keep the session moving instead of waiting for user input by default. That idle timer resets on real local activity, while the local agent has not yet returned to a prompt, or while the remote Codex pane still advertises itself as actively working. The tmux status line shows this as Wake remote.

The MCP surface is session-scoped and includes:

• session_status()
  • now includes mode, prompt_text, prompt_block, approval_command, and approval_options
• capture_snapshot()
• capture_prompt()
  • returns a prompt-focused shell/codex/approval block instead of the whole pane history
• get_settled_delta()
• send_text()
  • submits by default, so send_text("ls") runs in one tool call
  • use submit=false only when you intentionally want to stage text without pressing Enter
• send_input()
  • chains mixed key and text actions in one tool call
  • useful for cases like CTRL_U followed by codex and Enter
• approve_prompt()
  • answers numbered or y/n approval prompts directly
• install_remote_auth()
  • copies the selected local account auth.json to remote ~/.codex/auth.json
  • this is the default remote re-auth path before any browser login fallback
• send_keys()
• interrupt()
• clear_scrollback()
  • trims old remote tmux scrollback so fresh state is easier to read
• list_remote_sessions()
• read_action_log()
• wait_for_change()

Safety Model

• Agent writes are blocked while the operator owns the remote pane
• No writes are queued while blocked
• A write epoch changes when focus ownership changes
• MCP read tools refresh the agent’s observed epoch
• MCP write tools require the observed epoch to match the current epoch
• Every remote action is audited in SQLite and mirrored into the agent workdir

Demo Mode

cockpit demo provisions 3 fake hosts locally using separate tmux sockets and starts changing terminal workloads. A background demo orchestrator also adds and removes sessions so discovery churn can be observed without any real remote machines.

The demo uses its own generated config and data directory, so it does not have to share runtime sockets with a real cockpit deployment.

If the demo churn does not look right, inspect:

.cockpit-demo/orchestrator.log

Limitations

• The structured control target for a remote session is the active pane of the active window in that tmux session.
• The visible layout is tmux-on-tmux by design: the outer cockpit view attaches to inner viewer and agent tmux sessions.
• Automatic prompt injection into visible Codex panes is not used for production control. MCP is the supported path.
• Automatic account rotation is per affected Codex session. Other sessions keep running until they are restarted or hit exhaustion themselves.
• cockpit down tears down the cockpit tmux servers only. Use cockpit demo-clean to remove the local demo host tmux servers as well.

Testing

Run the full suite:

.venv/bin/pytest -q

The suite covers:

• config parsing
• session identity mapping
• settled snapshot logic
• write-lock behavior
• reconnect backoff
• agent persistence mapping
• pane selection state
• MCP scoping
• audit logging
• a local tmux-backed integration test for demo discovery churn

Code Map

• Core models and config: cockpit/models.py, cockpit/config.py
• SQLite state and audit trail: cockpit/state.py
• Snapshot settling: cockpit/snapshot.py
• Write ownership: cockpit/locks.py
• Remote tmux control: cockpit/tmux_remote.py
• Local tmux orchestration: cockpit/tmux_local.py
• Viewer wrapper: cockpit/viewer.py
• Codex agent lifecycle: cockpit/agent_manager.py
• Supervisor loop: cockpit/supervisor.py
• MCP server: cockpit/mcp_server.py
• CLI and operator controls: cockpit/cli.py
• Demo harness: cockpit/demo.py

Docs

• Architecture and tradeoffs: DESIGN.md
• Example inventory: config/hosts.example.yaml
• Codex MCP example config: .codex/config.toml.example