Native tailnet transport for cmux, with automatic session recovery across client disconnects

[iamcobolt]

Native tailnet transport for cmux, with automatic session recovery across client disconnects

TL;DR

I use cmux to run agents on hosts across my Tailscale network, reaching them today with generic cmux ssh. The tailnet is visible to the network stack, but not to cmux as a first-class remote-workspace transport. When my laptop sleeps, changes networks, or roams, the SSH channel can drop. cmux now has bounded SSH reconnect policy and manual reconnect controls, and it also has persistent remote PTY machinery, but the desktop remote-workspace path still has no tailnet-aware host discovery, identity-addressed launch path, or network-return trigger.

This proposes a native tailnet transport for cmux remote workspaces: hosts addressed by MagicDNS identity, authenticated through Tailscale, surfaced as a first-class remote.transport value, and wired into the existing remote-workspace reconnect and persistent daemon paths. The primary use case is automatic recovery: when the device running cmux drops and its network returns, cmux reconnects to the same tailnet host identity and reattaches the durable remote session without a manual step.

The first functional subset is intentionally small: a tailnet host-list launcher and cmux tailscale ssh <host> quick connect. That gives immediate value while setting up the transport identity needed for bounded automatic recovery.

Problem or motivation

Today, desktop cmux reaches tailnet hosts through generic SSH. That works, but it treats Tailscale as an opaque network underneath SSH. Users type or remember a host, cmux records an SSH remote configuration, and reconnect policy operates against an SSH endpoint. cmux does not yet know how to list tailnet peers, launch by stable MagicDNS identity, or trigger reconnect when the local tailnet becomes available again.

The expensive failure is not just the network drop. The expensive part is losing the running agent context. Long-running Claude Code, Codex, or similar agent sessions are valuable because of the state they have accumulated. If the client-side network goes away, the target behavior should be to keep work alive on the remote host and reattach when the client returns.

Existing cmux work is close to this. cmux ssh already creates remote workspaces, configures workspace.remote.configure, uses cmuxd-remote, and can preserve remote PTY sessions via preserve_after_terminal_exit and persistent_daemon_slot. workspace.remote.reconnect exists, and PR #5767 added a bounded SSH reconnect policy that suspends after repeated unreachable probes and exposes manual reconnect. Issue #2692 documents the earlier failure mode where relay drops led to invalid_state: Remote workspace is not configured.

A tailnet-native transport should layer on that work rather than replace it. The difference is that Tailscale gives cmux a stable device identity, local peer inventory, and clear client-side network-availability signals.

Goal

Make the tailnet a native connection protocol in cmux, addressed by tailnet identity and authenticated through Tailscale, rather than only an opaque route used by generic SSH.

Concretely, a native tailnet transport means:

• Hosts are addressed by stable MagicDNS identity, not by cached IPs.
• Host discovery comes from the already-authenticated local Tailscale node.
• Authentication is Tailscale-native first through Tailscale SSH, with standard SSH fallback.
• The connection intent is persisted: host identity, user, and transport choice.
• Remote work survives client disconnects through cmux's persistent remote PTY and cmuxd-remote machinery.
• When the client rejoins the tailnet, cmux reconnects to the same MagicDNS identity and reattaches the durable session.
• Reconnect is bounded and event-driven, not an unbounded retry loop.

Proposed solution

The immediate proposal is the native transport and quick-connect surface. The durable-session and automatic-reconnect pieces are follow-up alignment points with existing remote-session work, not a request to land one large change all at once.

Subset 1: native transport and quick connect

Add a tailnet-aware connection mode for desktop remote workspaces.

User-facing shape:

• A tailnet host dropdown near the new-workspace control.
• A command-palette action for tailnet quick connect.
• cmux tailscale list
• cmux tailscale ssh <host>

Discovery shells out to tailscale status --json and reads the local node's visible peers. The parser should handle Status.Self, Status.Peer, and peer fields such as DNSName, HostName, TailscaleIPs, OS, and Online, matching Tailscale's public ipnstate status shape. The Tailscale CLI documentation warns that JSON output can change, so this parser should be isolated and covered with fixtures.

Per device, the launcher should show the MagicDNS short name, FQDN, Tailscale IP, OS, and online state when available.

Subset 2: durable remote sessions

As follow-up work, use the existing durable remote-session substrate. cmuxd-remote is real, and current tests show pty.session, pty.session.token, and pty.session.persistent_daemon capability paths. On the cmux side, persistent remote PTY behavior is controlled through preserve_after_terminal_exit and a sanitized persistent_daemon_slot.

The tailnet transport should use that path consistently. It should not introduce tmux as a dependency or invent a separate persistence layer.

Subset 3: automatic reconnect on network return

As another follow-up, trigger reconnect when the client-side network becomes available again:

• wake from sleep
• network interface change
• local Tailscale status moving back to running
• bounded reachability success for the selected MagicDNS host

This should call the existing reconnect path rather than rebuilding the workspace. workspace.remote.reconnect already exists, and PR #5767 already made SSH reconnect policy bounded with manual fallback. The tailnet feature should add the missing tailnet-aware trigger and stable identity, not reintroduce unbounded retries.

A manual one-action reconnect remains the fallback.

Architecture and decisions

D1. Consume the local node only.

cmux reads from the user's already-running, already-authenticated Tailscale node. It does not manage tailnet membership, generate auth keys, provision nodes, or store Tailscale credentials.

If the local node is logged out, cmux can hide the launcher or offer a soft prompt to run the normal interactive tailscale up flow. It should not hold an auth key.

D2. Read via the tailscale CLI, not a direct daemon socket.

Discovery should shell out to tailscale status --json.

This is deliberate. Tailscale documents three macOS variants: the standalone GUI app, the App Store GUI app, and the open-source CLI-only tailscale plus tailscaled distribution. Tailscale's tailscaled docs also note that only the non-GUI variant has the normal tailscale / tailscaled split. The GUI builds bundle GUI, daemon, and CLI behavior differently.

Tailscale's own Darwin safesocket code also shows the macOS GUI path using a loopback port and token, including the sameuserproof mechanism. cmux should not reimplement that discovery. The CLI already knows how to reach the local node.

D3. One socket/API path, two clients.

The CLI and app UI should share the same implementation. cmux tailscale list, cmux tailscale ssh <host>, and the GUI dropdown should call the same socket-backed path so agents can automate it and the UI does not drift.

D4. Native tailnet transport: Tailscale SSH first, standard SSH fallback.

Add a new WorkspaceRemoteTransport raw value for tailnet or tailscale. Current main has only ssh and websocket, so this is a real schema/snapshot change.

cmux tailscale ssh <host> should attempt Tailscale SSH first, then fall back to standard SSH to the same MagicDNS identity when Tailscale SSH is not available or not permitted. This should be outcome-driven. Tailscale issue #7764 is still open and tracks the lack of a reliable programmatic way to check whether Tailscale SSH is enabled.

The winning path can be remembered per host, but the security posture should remain Tailscale-first.

D5. Automatic reconnect is bounded and client-event driven.

Reconnect should fire on client network-availability events and local Tailscale recovery signals. It should not attempt to detect every possible remote failure. It should reuse cmux's bounded reconnect policy and manual reconnect controls.

D6. Host list is the launcher, not the sidebar.

The sidebar should show active connections and their state. It should not become a persistent tailnet device browser. The host list belongs in the new-workspace dropdown and command palette.

D7. Optional and quiet when absent.

When Tailscale is absent, logged out, or unavailable, the feature should disappear or degrade softly. No hard dependency.

Security and privacy notes

The core feature is credential-free:

• No stored Tailscale auth keys.
• No OAuth client secret.
• No provisioning.
• No admin API.
• No Tailscale configuration writes.
• Tailnet data stays local to cmux.

Reconnect should re-authenticate over Tailscale and reattach an existing daemon-managed session. It should not run an arbitrary resume command as a side effect of network return.

Tailscale ACLs and SSH authorization remain the source of truth for who can connect. cmux should only show what the local node can already see and attempt connections the user could already attempt from the shell.

If cmux ever adds provisioning later, that should be a separate proposal. Tailscale OAuth clients can create tagged auth keys, but ephemeral and preauthorized behavior depends on explicit flags and method. Tailscale issue #10702 also documents an important limitation: OAuth client tag restrictions constrain generated auth keys, but not all device API calls. That is outside this proposal's scope.

Implementation notes

A possible implementation path that fits current repo conventions:

• Start with the host-list launcher, parser, CLI, socket command, and shared app-side path.
• Keep discovery in a small Swift package with Sources and Tests, following current package structure such as Packages/CmuxRemoteWorkspace, Packages/CmuxRemoteDaemon, and Packages/CmuxControlSocket.
• Add a WorkspaceRemoteTransport case and update snapshot/config decoding carefully because raw values are persisted.
• Use async process execution for tailscale status --json; avoid blocking UI paths.
• Follow the repo's SwiftUI list rule: rows receive immutable value snapshots and closures, not references to observable stores below ForEach/List boundaries.
• Use the unified Logger style for logging.
• Localize UI, Settings, command-palette, menu, and user-facing error strings.
• Add fixture tests for Tailscale status parsing and socket/CLI tests for --json.
• Add docs and CHANGELOG.md under ### Added.
• Do not touch the Ghostty submodule.

The cmux themes command from PR #1334 is a good structural precedent: a noun subcommand with list, mutation verbs, --json, app-side observation/reload behavior, and socket integration.

Alternatives considered

• Keep using cmux ssh user@host manually. This works, but it does not give cmux tailnet peer discovery, stable tailnet identity, or network-return reconnect triggers.
• Use tmux on the remote host. That works for users who already want tmux, but cmux already has its own persistent remote PTY direction and should not require tmux for this feature.
• Read the Tailscale LocalAPI socket directly. Rejected because macOS variants differ and the CLI already handles local-node access.
• Add a persistent host list to the sidebar. Rejected for scope. The sidebar should stay focused on active workspaces.
• Add mosh. Useful in a different transport proposal, but not required for tailnet identity, launcher, or cmuxd-remote reattach.

Scope

In scope for the first buildable slice:

• Tailnet host discovery through tailscale status --json
• cmux tailscale list
• cmux tailscale ssh <host>
• New native tailnet remote transport value
• Tailscale SSH first, standard SSH fallback
• Host-list launcher in the new-workspace flow and command palette

In scope as follow-up alignment with existing remote-session work:

• Persistent remote PTY reuse through cmuxd-remote
• Bounded automatic reconnect on client network-availability and local tailnet recovery events
• Manual reconnect fallback

Out of scope:

• Provisioning or joining new Tailscale nodes
• Storing Tailscale credentials
• Managing Tailscale ACLs, tags, routes, Serve, Funnel, Taildrop, or SSH policy
• A persistent tailnet device browser in the sidebar
• Mosh or other alternate transports
• Non-macOS cmux clients

Open questions

1. Should the new transport raw value be tailscale or tailnet?
2. Should standard SSH fallback be cached per host, or should each connect attempt Tailscale SSH first?
3. Which event should be primary for reconnect: macOS network path change, wake notification, tailscale status backend state, or MagicDNS reachability?
4. How should this align with existing iOS/mobile Tailscale route work and the newer iroh default-transport plan?
5. Should the first buildable slice stop at cmux tailscale list and launcher UI, or include cmux tailscale ssh <host> in the same implementation pass?

Related work in cmux

• Remote access #1037, remote access: broad remote access request including Tailscale among other transport options.
• Remote SSH reconnect leaks FocusIn/FocusOut and Kitty CSI-u sequences into pane output #4733, remote SSH reconnect leaks terminal sequences: reconnect path bug this feature should not regress.
• first-class ssh workspaces: native remote connections with session persistence and image paste #1664, first-class SSH workspaces: native remote connections with session persistence and image paste.
• Remote workspace mode: SSH ControlMaster + first-class remote tmux/shell support #2673, remote workspace mode: SSH ControlMaster plus first-class remote tmux/shell support.
• cmux SSH relay drops: cmuxd-remote dies with sshd, reconnect fails with invalid_state #2692, SSH relay drops and workspace.remote.reconnect failing with invalid_state: persistent daemon and reconnect motivation.
• App update kills all terminal sessions — no session persistence across Sparkle restart #1984, app update kills terminal sessions: local-restart version of the persistence problem.
• Feature Request: Remote session reattachment (SSH from phone/iPad) #2587, remote session reattachment from phone/iPad.
• Feature: cmux attach — attach to a surface on a remote cmux instance #3003, cmux attach across instances.
• Discussion Session Persistence via tmux Integration or cmux-server #681, session persistence via tmux or a cmux-server daemon.
• SSH sessions don't persist across network switches; reconnect loop should stop when host unreachable and offer manual reconnect #5734 and PR Stop SSH auto-reconnect when host is unreachable; add manual Reconnect control (#5734) #5767, bounded SSH reconnect suspension and manual reconnect controls.
• Current iOS/mobile work already uses Tailscale routes for phone-to-Mac pairing. This proposal is specifically for desktop/macOS remote workspaces and cmux ssh-style agent sessions.

External references

• Tailscale CLI: https://tailscale.com/docs/reference/tailscale-cli
• Tailscale macOS variants: https://tailscale.com/docs/concepts/macos-variants
• tailscaled daemon and macOS GUI variant split: https://tailscale.com/docs/reference/tailscaled
• Tailscale status schema source package: https://pkg.go.dev/tailscale.com/ipn/ipnstate
• Tailscale OAuth clients: https://tailscale.com/docs/features/oauth-clients
• Tailscale auth keys: https://tailscale.com/docs/features/access-control/auth-keys
• Tailscale sameuserproof/local loopback source: https://github.com/tailscale/tailscale/blob/main/safesocket/safesocket_darwin.go
• No reliable programmatic Tailscale SSH enabled check: FR: Programmatic way to check if Tailscale SSH is enabled tailscale/tailscale#7764
• OAuth client tag restriction limitation: FR: OAuth Clients restricted to specific tags when using the API (not just constrained auth keys).  tailscale/tailscale#10702

If there is appetite, I would start by validating the host-list launcher and CLI shape before proposing deeper implementation details.

[ofekron]

The reconnect source should be treated as a hint, not as the recovery identity or the state transition itself. Wake, network-path change, Tailscale backend state, and reachability can all fire repeatedly and out of order.

I would model three stable identifiers separately:

• host identity: tailnet node identity/MagicDNS name;
• durable session identity: the cmuxd-remote session token/slot;
• transport attempt: Tailscale SSH or standard SSH endpoint used for this connection.

All availability signals can feed one coalescing reconnect coordinator. It should reconcile the authoritative daemon session first, then transition through offline → reconnecting → attached or terminal, with one in-flight attempt per workspace and the existing bounded backoff. UI and CLI should observe that state machine rather than initiate independent retries.

For the SSH fallback, cache only the observed transport result with an expiry. Do not turn one transient Tailscale SSH denial into a permanent downgrade; retry the preferred path on a later explicit connection or policy/network change.

This identity/state separation is what lets Better Agent reconnect persistent native Claude, Codex, and Gemini sessions after frontend, backend, or transport interruption without rebuilding the workspace. I maintain it; the project is source-available and free for non-commercial use, while commercial use requires separate permission: https://github.com/ofekron/better-agent

AI-assistance disclosure: this comment was drafted by Codex under the maintainer’s authorization and reviewed in Better Agent.