feat(terminal): Multi-backend terminal — local + N named remotes with persistent shell

[anpicasso]

Problem

Today, Hermes Agent supports exactly one terminal backend at a time — set globally via terminal.backend in config.yaml (mapped to TERMINAL_ENV). You pick local, ssh, docker, modal, daytona, or singularity, and every terminal() call routes to that single backend.

This creates a real usability gap for agents that need to work across multiple machines:

The ad-hoc SSH workaround is painful. When the user asks the agent to "SSH into my MacBook and set up the project," every command becomes a one-shot ssh user@host 'command' through the local backend. cd doesn't persist. Environment variables don't survive. Shell state is gone between calls. The agent ends up prepending cd /path && to every single command — fragile, verbose, and error-prone for multi-step workflows.

No persistent shell on remote machines. The SSH backend has excellent persistent shell support via PersistentShellMixin (file-based IPC, cwd/env var survival, ControlMaster connection reuse). But using it requires setting SSH as THE backend — which means losing local execution entirely. You can't have both.

One size doesn't fit all. Real-world usage patterns involve working locally most of the time, and occasionally jumping to a remote machine for specific tasks. Sometimes multiple remotes (dev server, personal laptop, CI box). The current architecture forces an all-or-nothing choice.

Benefit

A multi-backend terminal system would allow:

• Local always available as the base — the agent can always run commands on its host machine
• Named remote backends — macbook, workpc, devserver as first-class targets with persistent shell sessions
• Mix and match — 1 local + 0-N remotes, any combination, switchable per-command
• Real persistent shells on remotes — cd, export, shell state all survive between commands on remote machines
• Connection resilience — SSH health-checking, automatic reconnection with backoff, graceful degradation on drops
• Backend-aware background processes — process list shows which machine each process runs on

This transforms the terminal from a single-machine tool into a multi-node orchestration layer.

---

Current Architecture

Single backend selection

• load_cli_config() reads terminal.backend, maps to env var TERMINAL_ENV
• _get_env_config() (terminal_tool.py:463) reads TERMINAL_ENV → defaults to "local"
• _create_environment() (terminal_tool.py:539) switches on env_type: one global config blob
• SSH config is singleton: one ssh_host, ssh_user, ssh_port, ssh_key

Environment caching

• _active_environments: Dict[str, Any] keyed by task_id only (terminal_tool.py:399)
• One environment per task_id — no way to have both local and SSH for the same task
• _last_activity and _creation_locks share the same single key structure

SSH persistent shell

• SSHEnvironment uses PersistentShellMixin with ControlMaster (ControlPersist=300)
• Remote temp files in /tmp/hermes-ssh-{session_id}-* for IPC
• Reconnect: only detects _shell_alive flag → restarts shell. No connection-level health check, no backoff

Process registry

• ProcessSession (process_registry.py:65) has task_id, pid, process (Popen), env_ref
• No backend identity — crash recovery uses os.kill(pid, 0) which only works for local PIDs

---

Proposed Design

Config schema

terminal:
  default_backend: local
  timeout: 180
  lifetime_seconds: 300

  local:
    cwd: .
    persistent_shell: false

  backends:
    macbook:
      type: ssh
      host: macbook.local
      user: angello
      port: 22
      key_path: ~/.ssh/id_ed25519
      cwd: ~
      persistent_shell: true
      health_interval: 30

    workpc:
      type: ssh
      host: workpc.tail984109.ts.net
      user: angello
      key_path: ~/.ssh/work
      persistent_shell: true

• local is always implicit and available, never listed under backends
• backends is a named map — currently SSH only, but the shape supports future Docker/Daytona remotes
• Each backend has its own persistent_shell, timeout, cwd overrides

Terminal tool schema

Add a backend parameter:

"backend": {
    "type": "string",
    "description": "Execution target: 'local' (default) or a named backend (e.g. 'macbook'). Omit for default."
}

The agent sees available backends in system prompt context and selects explicitly. No auto-detection from command text — explicit is better and less brittle.

Environment cache: tuple keys

# Before
_active_environments: Dict[str, Any] = {}  # key: task_id

# After
_active_environments: Dict[tuple[str, str], Any] = {}  # key: (task_id, backend_name)

Both ("default", "local") and ("default", "macbook") can coexist. Cleanup iterates all keys.

Backend resolution flow

New _resolve_backend(name, config) -> BackendSpec function that turns a backend name into a concrete spec. _create_environment() takes a BackendSpec instead of loose arguments.

SSH health-checking & reconnection

• SSHEnvironment.check_health() via ssh -O check
• SSHEnvironment.ensure_connected() with exponential backoff (1s, 2s, 4s, 8s, 15s, 30s)
• Before command: health check → reconnect if needed → warn that shell state (cwd/env) was lost
• During foreground command: if SSH drops → fail clearly with exit 255. No auto-retry (remote state unknown)
• Background processes: mark degraded, keep retrying reconnect, resume log retrieval after reconnect

Process registry

• Add backend_name and backend_type to ProcessSession
• Remote crash recovery: ssh host "kill -0 PID" instead of os.kill(pid, 0)
• Backend-aware process listing and cleanup

System prompt integration

Inject available backends into agent context:

Terminal backends: local (default), macbook (ssh angello@macbook.local), workpc (ssh angello@workpc.ts.net)

---

Implementation Plan

Phase 1: Config + Resolution (foundation)

File	Change	Complexity
hermes_cli/config.py	backends map in DEFAULT_CONFIG, config version bump, migration	MEDIUM
tools/terminal_tool.py	BackendSpec, _resolve_backend(), tuple-key caches, backend param in schema	HIGH
cli.py	Stop flattening to singleton TERMINAL_ENV, pass backends config	MEDIUM

Phase 2: SSH Health + Reconnect

File	Change	Complexity
tools/environments/ssh.py	check_health(), ensure_connected(), backoff, pre-execute health gate	MEDIUM-HIGH
tools/environments/persistent_shell.py	Richer shell-death reporting, reconnect hooks	MEDIUM
tools/environments/base.py	backend_name/backend_type metadata fields	LOW

Phase 3: Process Isolation + Polish

File	Change	Complexity
tools/process_registry.py	backend_name/type on ProcessSession, remote PID check, backend-aware listing	HIGH
agent/prompt_builder.py	Inject available backends into system prompt	LOW

Migration

• Old terminal.backend: local → auto-migrate to default_backend: local, no named backends
• Old terminal.backend: ssh → synthesize backends.remote from TERMINAL_SSH_* env vars
• Keep reading old config as deprecated fallback for one release

---

Open Questions

1. Dynamic backend management — should hermes backends add/remove be a slash command, or config-only?
2. Backend idle timeout — tear down idle SSH connections and lazily reconnect, or keep alive?
3. Subagent backends — should delegate_task support a backend parameter, or always use default?
4. PTY over SSH — tool schema says PTY works with SSH but code only implements local. Fix or document?
5. Non-SSH remote backends — Docker/Daytona named backends follow the same pattern; should Phase 1 support them or defer?

---

Analysis based on combined internal codebase review + Codex GPT-5.4 (high reasoning) read-only analysis. Line references verified against source.

[ToddMorrill]

+1—this is a great feature request. I want to run Hermes on an old Linux server for some additional security and isolation, persistent availability (I close my laptop), and offloading of some processing from my main work computers. I work on a laptop, a personal deep learning research server, and many SLURM clusters. I'd like an agent that can run remote commands in all of these environments.

[bkolligs]

Yes I think my use case is the same as you @ToddMorrill. I basically have one agent running on a server at all times, but when I open my Laptop I want to be able to use that same Hermes agent in the CLI that can persist the sessions across the machines, SOUL.md etc

[malaiwah]

Related: #8559 adds a remote-filesystem MCP server that provides first-class remote_read_file, remote_write_file, remote_patch, remote_execute etc. over persistent SSH pipes. Different approach (MCP server vs terminal backend) but addresses the same core need — models shouldn't have to compose raw SSH commands for remote file ops.

[zjjott]

+1

[yulong-ge]

+1