fix(ops): the fleet check sees a coordinator nobody can reach - #884
Merged
Conversation
All eight nodes opted in as Wraith coordinators on 2026-09-10, and 9100 was open on two. The installer already makes the port follow the opt-in — `reconcile-mining-firewall.sh`, re-run by `ghost-mining-firewall.path` whenever pool.toml changes — but nothing deploys it after install. Five nodes ran a copy older than its 9100 block and two had no reconcile unit at all, so the opt-in fired, reconciled the Stratum ports, and left the coordinator unreachable. Measured: the elected coordinator answered 200 on 127.0.0.1:9100 and timed out from outside. This check reported the fleet uniform throughout. It now asserts: - ghost-mining-firewall.path is enabled (it is what makes a port follow its opt-in), - reconcile-mining-firewall.sh matches the repo, not just the other nodes, - coordinator_role_enabled = true means ufw allows 9100 AND something listens there, and an open 9100 with the role off is a failure too. Run against the live fleet it fails on exactly the measured drift: 9100 closed on vm1,2,3,5,6,7; the unit missing on vm5,6; the script stale on the other six.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was measured (2026-09-11)
All eight nodes opted in as Wraith coordinators on 2026-09-10 (#708). Port 9100 was open on two of them (vm4, vm8).
The elected coordinator (vm3) answered
127.0.0.1:9100/healthwith 200, but from outside the request timed out. ufw isdefault deny (incoming)and vm3 has no 9100 rule. So every node electing vm3 was sending wallets to a coordinator none of them could reach.check-fleet-uniformity.shreported "fleet uniform, all invariants hold" the whole time.Why the automation didn't fire
It exists:
reconcile-mining-firewall.sh, re-run byghost-mining-firewall.pathwheneverpool.tomlchanges, opens 9100 whencoordinator_role_enabled = true. But nothing deploys it after install, sincedeploy-node.shswaps binaries only.journalctl -t ghost-mining-firewall:public mining ON -> Stratum 3333+34255 OPEN).coordinator role ON -> Wraith 9100 OPEN.What the check now asserts
ghost-mining-firewall.pathis enabled, since it's what makes a port follow its opt-in.reconcile-mining-firewall.shmatches the repo, not just the other nodes. Uniform staleness is how the restart watchdog hid for months.coordinator_role_enabled = truemeans ufw allows 9100 and something listens on it. An open 9100 with the role off fails too.Run against the live fleet
It fails on exactly the measured drift: 9100 closed on vm1, 2, 3, 5, 6 and 7; the unit missing on vm5 and 6; the script stale on the other six.
⚠
coordinator opted in but NOTHING listens on :9100also fires on seven nodes. That's correct for the deployed binary, which runs a coordinator only on the elected seat. It clears once every opted-in node runs one (the tier-leader change that follows #883).Not in this PR
The fleet remediation (copy the current script, install the units on vm5/vm6, trigger it) is a production change and belongs to the roll's config-first step.