feat(remote): add route bridge status/repair commands and docs

roodboi · roodboi · commit e53edfdd8f37 · 2026-02-27T20:47:39.000-05:00
diff --git a/.codex/skills/hack-cli/SKILL.md b/.codex/skills/hack-cli/SKILL.md
@@ -77,6 +77,14 @@ Use `hack` as the primary interface for local development.
 - Start/stop/status: `hack global up`, `hack global down`, `hack global status`
 - Global logs: `hack global logs <service> --no-follow --tail 200`
 
+## Remote Nodes + Dispatch
+
+- Pair/register a node: `hack node pair ...`, then verify via `hack node list` and `hack node status --watch`.
+- Repair SSH access for remote Git/mutagen: `hack node ssh setup --node <id>`.
+- Inspect node workspace map on the node host: `ssh <user@host> 'hack node workspace list --json'`.
+- Inspect/repair controller route bridge: `hack node routes status`, `hack node routes repair`.
+- Dispatch command to remote workspace: `hack dispatch run --project <name|id> --node default --branch <branch> --runner generic -- "pwd"`.
+
 ## Lifecycle + Startup
 
 - Put host setup in `.hack/hack.config.json` under `startup` / `lifecycle`.
diff --git a/AGENTS.md b/AGENTS.md
@@ -287,6 +287,13 @@ Global infra:
 - Start/stop/status: `hack global up`, `hack global down`, `hack global status`
 - Use `hack global up` before Loki/Grafana queries if global logging is offline.
 
+Remote nodes + dispatch:
+- Pair/register nodes: `hack node pair ...`, then verify with `hack node list` and `hack node status --watch`.
+- Repair SSH for remote Git/mutagen: `hack node ssh setup --node <id>`.
+- On node host, inspect workspace map via `hack node workspace list|resolve|attach|remove`.
+- Inspect/repair controller-side route bridge with `hack node routes status` and `hack node routes repair`.
+- Dispatch remote commands: `hack dispatch run --project <name|id> --node default --branch <branch> --runner generic -- "<command>"`.
+
 When to use a branch instance:
 - You need two versions running at once (PR review, experiments, migrations).
 - You want to keep a stable environment while testing another branch.
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -182,6 +182,13 @@ Global infra:
 - Start/stop/status: `hack global up`, `hack global down`, `hack global status`
 - Use `hack global up` before Loki/Grafana queries if global logging is offline.
 
+Remote nodes + dispatch:
+- Pair/register nodes: `hack node pair ...`, then verify with `hack node list` and `hack node status --watch`.
+- Repair SSH for remote Git/mutagen: `hack node ssh setup --node <id>`.
+- On node host, inspect workspace map via `hack node workspace list|resolve|attach|remove`.
+- Inspect/repair controller-side route bridge with `hack node routes status` and `hack node routes repair`.
+- Dispatch remote commands: `hack dispatch run --project <name|id> --node default --branch <branch> --runner generic -- "<command>"`.
+
 When to use a branch instance:
 - You need two versions running at once (PR review, experiments, migrations).
 - You want to keep a stable environment while testing another branch.
diff --git a/docs/README.md b/docs/README.md
@@ -22,6 +22,8 @@ Monorepo note:
 Quick diagnostics:
 - `hack usage` (resource usage across running projects)
 - `hack usage --watch` (live resource trends)
+- `hack node routes status` (controller-side remote route bridge state)
+- `hack node routes repair` (re-apply persisted remote route bridge routes)
 
 ## Guides
 
diff --git a/docs/cli.md b/docs/cli.md
@@ -757,6 +757,7 @@ Subcommands:
 | `use` | Set default node |
 | `remove` | Remove node registration |
 | `workspace` | Inspect and repair node workspace map entries |
+| `routes` | Inspect and repair controller-side remote route bridge |
 | `provider` | Manage provider-specific node bootstrap workflows |
 | `devcontainer` | Manage remote node devcontainer lifecycle and attach hints |
 
@@ -804,6 +805,11 @@ ssh <user@node-host> 'hack node workspace resolve --project <name|id> --json'
 ssh <user@node-host> 'hack node workspace attach --project <name|id> --path <absolute-path> --json'
 ssh <user@node-host> 'hack node workspace remove --project <name|id> --json'
 
+# inspect/repair controller-side remote route bridge
+hack node routes status
+hack node routes status --json
+hack node routes repair
+
 # Railway provider bootstrap (existing service + auto domain endpoint)
 hack node provider railway bootstrap \
   --railway-project "<project-id-or-name>" \
@@ -874,6 +880,25 @@ hack node workspace attach --project my-project --path "$HOME/.hack/projects/my-
 hack node workspace remove --project my-project --json
 ```
 
+### hack node routes
+
+Usage: `hack node routes <subcommand>`
+
+Subcommands:
+
+| Subcommand | Summary |
+| --- | --- |
+| `status` | Show controller-side remote route bridge state |
+| `repair` | Re-render and re-apply persisted route bridge stack |
+
+Examples:
+
+```bash
+hack node routes status
+hack node routes status --json
+hack node routes repair
+```
+
 ### hack dispatch
 
 Usage: `hack dispatch <subcommand>`
diff --git a/docs/guides/remote-node-laptop-e2e.md b/docs/guides/remote-node-laptop-e2e.md
@@ -124,6 +124,8 @@ ssh <user@node-host> 'hack node workspace list --json'
 Expected:
 1. Mapping entry exists for controller project id/name.
 2. Workspace root resolves under `~/.hack/projects/<project-slug>/` unless explicitly attached elsewhere.
+3. Controller writes/updates the local remote-route bridge so `https://<project>.hack` is proxied through local Caddy to the remote node host.
+4. Route bridge diagnostics/repair are available on controller via `hack node routes status` and `hack node routes repair`.
 
 ### Private repo bootstrap behavior (default, no manual copy)
 
@@ -208,6 +210,10 @@ cat ~/.hack/registry/runs/<run-id>/summary.md
 9. `Mutagen binary was not found on this machine`
    - Cause: controller is missing mutagen and auto-install failed (permissions/network/tooling).
    - Fix: run `hack doctor --fix` (or `hack global install`) and retry.
+10. `hack up --target auto` succeeds remotely but local browser on `https://<project>.hack` fails TLS/connection
+   - Cause: local global proxy stack is down, so remote bridge routes were saved but not applied.
+   - Fix: run `hack global up`; remote routes are reconciled automatically on startup.
+   - Optional: run `hack node routes status` and `hack node routes repair` to inspect/re-apply bridge routes explicitly.
 
 ## Evidence capture
 
@@ -216,6 +222,7 @@ Record these outputs in your run notes:
 ```bash
 hack node list
 hack node status --json
+hack node routes status --json
 hack config get "controlPlane.execution.mode"
 hack config get "controlPlane.execution.nodeId"
 hack dispatch status <run-id>
diff --git a/docs/guides/remote-node-quickstart.md b/docs/guides/remote-node-quickstart.md
@@ -46,6 +46,7 @@ Validate:
 ```bash
 hack node list
 hack node status --node <node-id>
+hack node routes status
 ```
 
 ## Step 3: route a project to the remote node
@@ -75,6 +76,12 @@ Optional remote check:
 ssh <user@remote-host> 'hack node workspace list --json'
 ```
 
+Controller route bridge check:
+
+```bash
+hack node routes status --json
+```
+
 ## macOS app flow
 
 You can do the same flow in Hack Desktop:
@@ -92,3 +99,6 @@ You can do the same flow in Hack Desktop:
    - install `hack` on remote or pass `--remote-hack` with the full path.
 3. `Connection closed by ... port 22`:
    - confirm username and shell access with `ssh <user@host> "echo ok"`.
+4. Local project URL fails after remote dispatch (`https://<project>.hack`):
+   - run `hack node routes status` then `hack node routes repair`.
+   - if global proxy is down, run `hack global up` and retry.
diff --git a/src/agents/codex-skill.ts b/src/agents/codex-skill.ts
@@ -193,6 +193,14 @@ export function renderCodexSkill(): string {
     "- Start/stop/status: `hack global up`, `hack global down`, `hack global status`",
     "- Global logs: `hack global logs <service> --no-follow --tail 200`",
     "",
+    "## Remote Nodes + Dispatch",
+    "",
+    "- Pair/register a node: `hack node pair ...`, then verify via `hack node list` and `hack node status --watch`.",
+    "- Repair SSH access for remote Git/mutagen: `hack node ssh setup --node <id>`.",
+    "- Inspect node workspace map on the node host: `ssh <user@host> 'hack node workspace list --json'`.",
+    "- Inspect/repair controller route bridge: `hack node routes status`, `hack node routes repair`.",
+    '- Dispatch command to remote workspace: `hack dispatch run --project <name|id> --node default --branch <branch> --runner generic -- "pwd"`.',
+    "",
     "## Lifecycle + Startup",
     "",
     "- Put host setup in `.hack/hack.config.json` under `startup` / `lifecycle`.",
diff --git a/src/agents/cursor.ts b/src/agents/cursor.ts
@@ -159,6 +159,12 @@ export function renderCursorRules(): string {
     "hack global status",
     "hack session",
     "hack tickets list",
+    "hack node list",
+    "hack node status --watch",
+    "hack node ssh setup --node <id>",
+    "hack node routes status",
+    "hack node routes repair",
+    'hack dispatch run --project <name|id> --node default --branch <branch> --runner generic -- "pwd"',
     "hack internal extra-hosts list",
     "hack internal extra-hosts set <hostname> <target>",
     "hack internal extra-hosts unset <hostname>",
diff --git a/src/agents/init-patterns.ts b/src/agents/init-patterns.ts
@@ -59,6 +59,9 @@ export function renderAgentInitPatterns(): string {
     "- If host-side setup is required (auth proxy, tunnel, credential bootstrap), encode it in `.hack/hack.config.json` under `startup`/`lifecycle`.",
     "- For service-to-host proxy/tunnel routing, configure static `internal.extra_hosts` or dynamic mappings via `hack internal extra-hosts set`.",
     "- For externally reachable local HTTP routes, ensure services carry Caddy labels (`caddy`, `caddy.reverse_proxy`, `caddy.tls=internal`) and attach to `hack-dev`.",
+    "- For remote-run projects, set `controlPlane.execution.mode=local_edit_remote_run` and pin `controlPlane.execution.nodeId` to a paired node.",
+    "- When remote project URLs fail locally, inspect/re-apply controller route bridge with `hack node routes status` and `hack node routes repair`.",
+    "- Keep remote node workspaces healthy with `hack node workspace list|resolve|attach|remove` (run on node host over SSH).",
     "- Use `.hack.gy` alias hosts for OAuth callback testing when provider rules reject `.hack`.",
     "- Treat `.hack/.internal/*` and `.hack/.branch/*` overrides as generated artifacts; keep edits in source config/compose files.",
     "- Use `hack run <service> <cmd...>` for one-off tasks.",
diff --git a/src/agents/primer.ts b/src/agents/primer.ts
@@ -57,6 +57,13 @@ export function renderAgentPrimer(): string {
     "- Sessions: `hack session`",
     "- Tickets: `hack tickets list`",
     "",
+    "Remote execution + node ops:",
+    "- Pair/register nodes: `hack node pair ...`, `hack node list`, `hack node status --watch`.",
+    "- Repair SSH for remote Git/mutagen: `hack node ssh setup --node <id>`.",
+    "- Inspect node workspace map on remote host: `ssh <user@host> 'hack node workspace list --json'`.",
+    "- Inspect/repair controller route bridge: `hack node routes status`, `hack node routes repair`.",
+    '- Dispatch remote command: `hack dispatch run --project <name|id> --node default --branch <branch> --runner generic -- "pwd"`.',
+    "",
     "Branch instances (parallel envs):",
     "- `hack up --branch <name> --detach`",
     "- `hack open --branch <name>`",
diff --git a/src/commands/dispatch.ts b/src/commands/dispatch.ts
@@ -60,6 +60,10 @@ import {
   resolveRegisteredProjectByName,
   upsertProjectRegistration,
 } from "../lib/projects-registry.ts";
+import {
+  type RemoteCaddyRouteBridgeResult,
+  reconcileRemoteCaddyRoutesForProject,
+} from "../lib/remote-caddy-routes.ts";
 import { exec } from "../lib/shell.ts";
 import { display } from "../ui/display.ts";
 import { logger } from "../ui/logger.ts";
@@ -507,6 +511,34 @@ async function handleDispatchRun({
     ...(bootstrapProbe ? { probe: bootstrapProbe } : {}),
     ...(bootstrapAuthEnsured ? { ensured: bootstrapAuthEnsured } : {}),
   };
+  const remoteRouteBridge = await reconcileDispatchRemoteRouteBridge({
+    project,
+    workspace: workspace.data.workspace,
+    node: selectedNode.node,
+  });
+  if (remoteRouteBridge) {
+    await appendDispatchRunEvent({
+      runId,
+      event: {
+        type: "run.route.bridge",
+        status: remoteRouteBridge.status,
+        reason: remoteRouteBridge.reason,
+        hosts: remoteRouteBridge.hosts,
+        upstream: remoteRouteBridge.upstream,
+        composePath: remoteRouteBridge.composePath,
+        ...(remoteRouteBridge.error ? { error: remoteRouteBridge.error } : {}),
+      },
+    });
+    if (remoteRouteBridge.status === "failed") {
+      logger.warn({
+        message: `Remote route bridge failed: ${remoteRouteBridge.error ?? remoteRouteBridge.reason}`,
+      });
+    } else if (remoteRouteBridge.status === "applied") {
+      logger.info({
+        message: `Remote route bridge applied (${remoteRouteBridge.hosts.length} host${remoteRouteBridge.hosts.length === 1 ? "" : "s"})`,
+      });
+    }
+  }
 
   let syncMetadata: DispatchSyncMetadata | null = null;
   const preparedSync = await prepareDispatchSync({
@@ -690,6 +722,7 @@ async function handleDispatchRun({
       route: routeMetadata,
       sync: syncMetadata,
       bootstrapAuth,
+      remoteRouteBridge,
     });
     await writeDispatchRunArtifacts({
       runId,
@@ -718,6 +751,7 @@ async function handleDispatchRun({
         riskReasons: policy.reasons,
         route: routeMetadata,
         ...(Object.keys(bootstrapAuth).length > 0 ? { bootstrapAuth } : {}),
+        ...(remoteRouteBridge ? { remoteRouteBridge } : {}),
         ...(syncMetadata ? { sync: syncMetadata } : {}),
         ...(ticketId ? { ticketId } : {}),
         ...(prOutcome
@@ -1423,6 +1457,26 @@ type DispatchWorkspaceBootstrapAuth = {
   readonly ensured?: GatewayNodeBootstrapAuthSource;
 };
 
+async function reconcileDispatchRemoteRouteBridge(input: {
+  readonly project: DispatchProjectResolution;
+  readonly workspace: GatewayNodeWorkspace;
+  readonly node: NodeRecord;
+}): Promise<RemoteCaddyRouteBridgeResult | null> {
+  const projectKey =
+    input.project.controllerProjectId ??
+    normalizeOptionalString(input.workspace.projectId) ??
+    normalizeOptionalString(input.workspace.projectName);
+  if (!projectKey) {
+    return null;
+  }
+  return await reconcileRemoteCaddyRoutesForProject({
+    projectKey,
+    projectDir: input.project.projectDir,
+    fallbackProjectHost: input.workspace.projectName,
+    node: input.node,
+  });
+}
+
 function formatWorkspaceEnsureError(input: {
   readonly workspace: {
     readonly status: number;
@@ -2760,6 +2814,7 @@ function buildSummaryMarkdown(input: {
   readonly route: DispatchRouteMetadata;
   readonly sync: DispatchSyncMetadata | null;
   readonly bootstrapAuth: DispatchWorkspaceBootstrapAuth;
+  readonly remoteRouteBridge: RemoteCaddyRouteBridgeResult | null;
 }): string {
   let prLines: string[] = ["- status: not requested"];
   if (input.prOutcome?.ok) {
@@ -2844,6 +2899,24 @@ function buildSummaryMarkdown(input: {
       ? [`- workspace_ensure_source: ${input.bootstrapAuth.ensured}`]
       : ["- workspace_ensure_source: unknown_or_not_bootstrapped"]),
     "",
+    "## Local route bridge",
+    ...(input.remoteRouteBridge
+      ? [
+          `- status: ${input.remoteRouteBridge.status}`,
+          `- reason: ${input.remoteRouteBridge.reason}`,
+          `- compose_path: ${input.remoteRouteBridge.composePath}`,
+          ...(input.remoteRouteBridge.upstream
+            ? [`- upstream: ${input.remoteRouteBridge.upstream}`]
+            : []),
+          ...(input.remoteRouteBridge.hosts.length > 0
+            ? input.remoteRouteBridge.hosts.map((host) => `- host: ${host}`)
+            : ["- host: none"]),
+          ...(input.remoteRouteBridge.error
+            ? [`- error: ${input.remoteRouteBridge.error}`]
+            : []),
+        ]
+      : ["- status: not attempted"]),
+    "",
     "## Sync",
     ...(input.sync
       ? [
diff --git a/src/commands/global.ts b/src/commands/global.ts
@@ -63,6 +63,10 @@ import {
   getMutagenPath,
 } from "../lib/mutagen.ts";
 import { isMac } from "../lib/os.ts";
+import {
+  reconcileRemoteCaddyRoutesStack,
+  stopRemoteCaddyRoutesStack,
+} from "../lib/remote-caddy-routes.ts";
 import { exec, execOrThrow, findExecutableInPath, run } from "../lib/shell.ts";
 import { resolveSessionsMuxMode } from "../mux/mux-config.ts";
 import {
@@ -803,6 +807,16 @@ export async function globalUp(): Promise<number> {
   if (caddyExit !== 0) {
     return caddyExit;
   }
+  const remoteRoutes = await reconcileRemoteCaddyRoutesStack();
+  if (remoteRoutes.status === "failed") {
+    logger.warn({
+      message: `Remote route bridge sync failed: ${remoteRoutes.error}`,
+    });
+  } else if (remoteRoutes.status === "applied") {
+    logger.step({
+      message: `Remote route bridge active (${remoteRoutes.routeCount} host${remoteRoutes.routeCount === 1 ? "" : "s"})`,
+    });
+  }
 
   logger.step({ message: "Starting logging stack…" });
   const logExit = await run(
@@ -1213,6 +1227,12 @@ async function globalDown(): Promise<number> {
       cwd: dirname(paths.caddyCompose),
     });
   }
+  const remoteRoutes = await stopRemoteCaddyRoutesStack();
+  if (remoteRoutes.status === "failed") {
+    logger.warn({
+      message: `Unable to stop remote route bridge during shutdown: ${remoteRoutes.error}`,
+    });
+  }
 
   if (isMac()) {
     const ok = await confirm({
diff --git a/src/commands/node.ts b/src/commands/node.ts
diff --git a/src/constants.ts b/src/constants.ts
diff --git a/src/lib/remote-caddy-routes.ts b/src/lib/remote-caddy-routes.ts
diff --git a/src/mcp/agent-docs.ts b/src/mcp/agent-docs.ts
diff --git a/tests/remote-caddy-routes.test.ts b/tests/remote-caddy-routes.test.ts
