feat(dashboard): build the real /api/v1/swarm/nodes/* fleet-control API

[defenwycke]

PROBLEM: 8 live dashboard fleet-control routes return fabricated success (Restart command sent) while doing nothing.
EVIDENCE: crates/ghost-verification/src/routes.rs:10783-10895.
TEST: integration test asserting each route performs a real action or errors, never a silent no-op.
ACCEPTANCE: restart/config/update/refresh wired to real authenticated node RPC; fake-success stubs removed.

[defenwycke]

The fake-success violation is already fixed — verified, not read

Checked against the code and the test suite tonight, because three issue descriptions have turned out
stale in the last day.

All six fleet-control handlers now route to swarm_not_implemented (routes.rs:11155, 11194, 11214, 11222, 11231, 11296), which answers 501 with the action named in the body. The guard test passes:

test routes::tests::swarm_control_routes_report_not_implemented_rather_than_fake_success ... ok

and it asserts the response SHAPE, not just the status — a bare 501 with no body would satisfy a status
check while still telling the operator nothing.

So {"message": "Restart command sent"} with nothing sent is gone. That was the zero-tolerance part:
an operator pressing Restart was told it worked and nothing happened, which hides a missing capability
rather than reporting it. It now reports it.

What remains is a feature, and it needs a security decision first

Making these routes actually work needs two things that do not exist:

• a swarm-node store (grep swarm_node crates/ghost-storage finds nothing)
• an authenticated node-to-node control RPC

One node being able to restart another, or rewrite its config, is a security-sensitive capability.
The existing comment in routes.rs says the auth design was deliberately left out rather than
improvised, and that is the right call — inventing one unattended is how the rest of that file grew
stubs in the first place.

Operator decision needed on the trust model before any code:

1. Operator-authenticated only — the dashboard holds a credential and signs the request; nodes do
   not command each other. Simplest, and matches how the fleet is actually run today.
2. Node-to-node with the existing identity keys — reuses the mesh identity that already signs
   consensus messages, but means a compromised node can restart its peers, which is a new and
   significant blast radius.
3. Threshold — a control action needs signatures from a quorum of nodes. Safest, most work, and
   overlaps with Decentralise the SV2 pool identity: threshold-Schnorr authority certificates (no secret holder, no distributor) #520's threshold-Schnorr work.

I would take (1). It delivers the dashboard capability that was wanted, and it does not give any node
authority over another — which matters more once the fleet is genuinely multi-operator, since option (2)
would hand a new operator a restart button pointed at everyone else.

Not building any of them tonight: the violation is closed, and the remainder is a design question, not
a coding task.

[defenwycke]

Status check against the code and the live fleet, because this issue reads as "three routes to build" and
that is no longer the shape of it.

restart is done

fleet_auth.rs + the wiring in 72dea2406 (#574) implement operator-signed control end to end:
signature over ghost-fleet-v1\n<target_node_id>\n<action>\n<params_hash>\n<nonce>\n<expires_at>,
path/target agreement checked, action checked, then the real local restart. Operator→node only; no
node-to-node path, and nothing consults the elder set or a quorum.

It is fail-closed and currently disabled: GHOST_OPERATOR_PUBKEY is set on none of the 8 nodes, so
every request gets NoOperatorKey → 501. Probed live on vm5 with a well-formed unsigned command: 401 from
the internal HMAC tier that sits in front of it, and the service stayed up with an unchanged start
timestamp. Two layers, both working.

So restart needs an operator keypair, not code. That part is on the operator, and the private board has
the detail.

The other two should not be built the same way — or at all

update (remote version change): recommend this stays 501 permanently.

A remote binary swap over HTTP would bypass every protection the actual deploy path has.
scripts/deploy-node.sh backs up the prior binary, captures a share-throughput baseline, swaps, waits
180s, re-measures, and rolls back automatically if throughput regressed. That rollback exists because
of the H-13 outage, where every check passed — unit active, port listening, /health answering, no errors
— while every share was being rejected. A POST .../update that just fetches and restarts would
reintroduce exactly the failure mode that cost 30 minutes of fleet-wide rejection.

The one-node-per-command rule came from the same place: batching three hit a 10-minute timeout mid
verification and left a node unsupervised. An API endpoint invites the batch.

If remote deploys are wanted later, the right shape is an endpoint that triggers the existing script and
streams its verdict, not one that swaps a binary itself. Different feature, different issue.

config (arbitrary settings write): recommend scoping it to an allow-list, not generalising.

There is already a real, validated, authenticated config write —
POST /api/v1/config/maxconnections — which parses ghostd's config, refuses an ambiguous file rather
than guessing which duplicate is in force, rewrites in place preserving every other line, and bounds the
value. That is what a config endpoint should look like, and it took that much care for one integer.

A generic PUT .../config taking arbitrary keys cannot have any of those properties. It would be a
foot-gun with an operator signature on it. The useful version is per-setting routes reusing the restart
route's signature check with action: "config:<setting>", added one setting at a time as each earns its
validation.

Worth noting #614 here: the maxconnections ceiling is currently derived from instantaneous
MemAvailable, so even that well-built endpoint can refuse a valid value on a dip. Any config surface
exposed to the dashboard inherits that.

Also outstanding

• dashboard/src/lib/api/swarm.ts sends no signed command, so the swarm page's buttons cannot work even
  once a key is deployed. The dashboard needs a signing path first — and a decision on whether the
  private key lives on the dashboard host at all, or whether a local CLI signs and the operator pastes the
  blob.
• /api/v1/node/restart restarts ghost-pool only, not ghostd. The UI should say which, or offer
  both. ghost-restart-watch.service also re-starts ghost-pool within ~5 minutes regardless, so "stop"
  has never been a durable operation here.

Suggested reframing

This issue's original complaint — routes reporting success without acting — is fixed. What is left is
three separate things with different owners: an operator keypair, a dashboard signing path, and a
per-setting config surface. Happy to split them out if that is preferred; leaving it as one issue makes it
look like a block of code that nobody has started, which is not the case.

[defenwycke]

Fixed — verified against c49ed365b on 2026-08-05.

The fabricated success is gone. The three fleet-control routes now call
swarm_not_implemented(...) instead of reporting work they never did, and the old behaviour is
recorded in the past tense at crates/ghost-verification/src/routes.rs:

/// Fleet-control routes that claim to act on a REMOTE node but have nothing behind them.
///
/// These returned fabricated success — `{"message": "Restart command sent"}` with no command
/// sent, `204 No Content` for an update that updated nothing. An operator pressing Restart in
/// the dashboard was told it worked, and nothing happened. That is worse than an error: it
/// hides the missing capability instead of reporting it (#403).

and on the add path:

// Returned a fully-formed node object — id, name, online, shares — none of which was
// stored anywhere. The caller had every reason to believe the node had been added.
swarm_not_implemented(&format!("{:08x}", fxhash(&body.address)), "add node")

So the zero-tolerance violation in #608 §5 (fake success must not ship) is resolved: the routes
now report the missing capability rather than concealing it.

Note this closes the HONESTY defect, not the feature — the swarm fleet-control API still does not
exist. If a working /api/v1/swarm/nodes/* is wanted for v1 it needs its own issue, scoped as a
build rather than a fix.

[defenwycke]

Reopening — I closed this a moment ago and that was wrong.

What is fixed is the fake success (#608 §5's zero-tolerance blocker): the routes now call
swarm_not_implemented(...) instead of reporting work they never did. That is the part that
blocked v1.

What this issue actually asks for — build the real /api/v1/swarm/nodes/* fleet-control API —
has NOT been done. The routes are honest about being unimplemented; they are still unimplemented.

So: no longer a v1 blocker, still an open feature. Leaving it open under its own title rather
than closing it on a partial result.