[Bug]: Docker agent commands unavailable (command_agent_disconnected) despite agent reporting Healthy in Agent Doctor (and --enable-commands added)

[peterzolczer]

Describe the bug

Docker/Podman container actions (e.g. "update") fail with action_execution_unavailable / command_agent_disconnected, even though the Agent Doctor report shows the agent connection as Healthy and actively reporting metrics. Reproduced independently on two separate Docker hosts.

Environment:

Pulse server: v6.1.1 (stable channel, deploymentType proxmoxve, containerized)
Agent: v6.1.1, unified agent, Docker/Podman module

Two independent Docker hosts affected, both installed via install.sh with --enable-commands and fresh exec-scoped tokens generated after upgrading (both pulse and pulse-agent) to v6.1.1. (I checked #1586 , uninstalled agents completely, then installed again with new token generated, no change in result, container update still shows failed status if button clicked on UI.)

To reproduce

Steps to reproduce

1. Install the unified agent on a Docker host with --enable-commands and a token generated via "Add Pulse Agent" with "Enable Pulse command execution" ticked.
2. Confirm agent registers and reports metrics normally; Agent Doctor shows the connection as Healthy.
3. Attempt a container action (e.g. "Update") from the web UI on any container on that host.
4. Action fails immediately.

Expected behavior

Docker action commands should not be rejected if --enable-commands is added.

Pulse version

v6.1.1

Agent version

v6.1.1

Image tag or digest

v6.1.1

Installation type

ProxmoxVE LXC

Relevant logs or diagnostics

error: "action_execution_unavailable"
message: "Action execution is unavailable"
details: {
  capabilityName: "update",
  reason: "Docker / Podman command agent is not connected.",
  reasonCode: "command_agent_disconnected",
  resourceId: "app-container-7311efa327faa70f"
}

Confirmations

• I verified the exact Pulse version and image reference from the running instance.
• I searched existing issues for duplicates.

[rcourtman]

Implemented on main in c41edb6.

Root cause

Two independent contracts were being conflated:

1. In v6.1.1, PULSE_AGENT_INGEST_PORT admitted /api/agents/* but rejected the command WebSocket at /api/agent/ws. The documented PULSE_AGENT_CONNECT_URL split-port setup directed the unified agent to that listener for both traffic classes. Docker and Podman reports therefore remained fresh while the command client received a 404 and reconnected forever. Rotating an agent:exec token, reinstalling, or restarting could not repair the missing route.
2. The command registry was keyed only by agent ID and retained only a boolean token decision. It did not preserve the canonical organization/token/agent/hostname admission, did not revalidate a live socket after token rotation or revocation, and allowed report identity fallbacks to stand in for command-session ownership.

That is why telemetry health and Agent Doctor could look healthy while actions failed with action_execution_unavailable / command_agent_disconnected.

Fix

• The dedicated agent listener now exposes an exact, bounded unified-agent control-plane allowlist: report/config routes, /api/agent/ws, version/server-info checks, and installer/binary bootstrap routes. UI and management routes still return 404, including normalized-path and traversal lookalikes.
• A command session now carries an immutable organization ID, token ID, agent ID, and hostname admission. First-use install tokens persist that binding; pre-v6.1.1 hostname-bound identities receive a one-time persisted migration; later identity replay fails closed.
• Session lookup, dispatch, pending request correlation, durable receipt queries, and deployment progress are tenant-scoped. Same IDs may exist in different tenants, while duplicate IDs or hostnames inside one tenant are rejected. Same-identity reconnect replaces only the old socket and cannot delete the replacement session or its progress subscription.
• Revoked, expired, rotated, or rebound tokens invalidate stale sockets before visibility or dispatch. A persistence failure denies first-use admission and restores the prior in-memory metadata.
• Docker/Podman action routing resolves the live command session from the report's immutable token ID first. If that token has no admitted session, it does not fall back to a stale agent ID or hostname.
• The Connections ledger and Agent Doctor now distinguish current telemetry from command-channel admission. A reporting agent with commands enabled but no admitted WebSocket is shown as remoteControl: disconnected with critical command_channel_disconnected, rather than healthy command readiness.

The existing action envelope remains fail-closed: reviewed action/attempt/operation/digest/agent binding, replay denial, approval and audit receipts, bounded timeout/cancellation, typed update/start/stop/restart operations, and durable reconciliation are preserved.

Verification

• go test ./internal/agentexec ./pkg/server ./internal/api
• Focused go test -race coverage for reconnect replacement, tenant isolation, duplicate identity/hostname rejection, stale-session revocation, token binding/migration, Docker token routing, and telemetry-vs-command health
• go vet ./internal/agentexec ./internal/api ./pkg/server
• 88 focused frontend Connections/Doctor/settings tests
• Frontend type-check, lint, and production build
• Canonical completion, status, control-plane, registry, contract, mutation-registry, repository-governance, readiness, and release-control unit suites

Coverage includes the shared Docker/Podman path, fresh first-use binding, legacy upgrade migration, reconnect/session replacement, rotation/revocation, duplicate identities, multi-tenant isolation, restart/recovery correlation, stale sockets, command policy presentation, and update/start/stop/restart routing.

Release applicability and v6.1.1 workaround

This affects v6.1.1 deployments that use the dedicated agent port, or any reverse proxy/firewall that forwards /api/agents/* but not /api/agent/ws. The fix is on main and is applicable to the next patch release.

Until that patch is installed, use one of these bounded workarounds:

• On a trusted/VPN-only network, point the agent at the normal FRONTEND_PORT listener, which already serves both reports and the WebSocket.
• Otherwise, proxy the exact agent-owned routes to the normal Pulse listener: /api/agents/*, /api/agent/ws with WebSocket Upgrade support, /api/agent/version, /api/server/info, /install.sh, /install.ps1, and /download/pulse-agent. Keep /, login, and management APIs blocked.

After changing PULSE_AGENT_CONNECT_URL or the agent service environment, restart the agent. Do not work around this by repeatedly minting tokens or weakening command policy. Confirm that Agent Doctor reports the command channel/remote control as connected; a fresh report by itself is not sufficient.

Leaving this issue open pending confirmation on an affected installation and availability in a patch release.

[peterzolczer]

Thanks for the quick reaction. For the workaround: I'm not using PULSE_AGENT_INGEST_PORT (single-port setup, agent points at the normal FRONTEND_PORT on 7655) and there's no reverse proxy in between — the agent connects directly to the Pulse server's IP on my LAN. So this looks like it's the second bug you described (the command registry/session-binding issue), not the split-port routing issue, since neither documented workaround applies to my setup.

[rcourtman]

Thanks — that clarification confirms this is the session-binding/registry half of the defect. Because your agent already connects directly to the normal FRONTEND_PORT, the split-port workaround does not apply to your setup.

The fix in c41edb65a is on main, but it is not included in any published release yet.

[erikrt]

I got the start and stop buttons to work with Docker after updating to the latest version and redoing the API tokens, however I cannot get the Update button to work. It appears to work but if you watch the button in the background you'll see it say failed. Also I am unable to use custom API tokens to try to scope more permissions. The script will just fail to install the token if I don't use the automatically generated one from the UI flow.

[rcourtman]

Closing the reported command_agent_disconnected defect as fixed by c41edb65a on main. The latest field report confirms start/stop commands now reach the Docker agent after updating and rotating the enrollment token, which verifies command-channel admission. A container Update action that is admitted and later reports Failed is a separate execution/verification outcome and needs its failure reason or action-history details to diagnose; likewise, an arbitrary custom API token is not a substitute for the single-use enrollment token because enrollment also establishes the immutable agent/host/token binding. Please open a focused Update-action report with the action result/reason code and agent log around the command so that path can be tracked without conflating it with the resolved disconnected-channel failure.