Cluster

Cluster

Run secwatch on several boxes and join them into a peer-to-peer cluster — there is no central hub to stand up or defend. Every node stays fully autonomous (it detects and bans itself); nodes gossip bans so a hit on one hardens all, and any peer can view the whole fleet from its own dashboard.

How it works

• Explicit join, shared secret — like a Proxmox cluster. A node only joins if it has the secret; there's no auto-discovery, so a random box on the LAN can't join, read, or inject.
• HMAC-signed peer traffic — the secret never crosses the wire; requests are timestamped to bound replay. Use TLS or a WireGuard tunnel if the inter-node path isn't trusted.
• Ban gossip, two ways — a new ban is pushed to reachable peers immediately and every node periodically pulls each peer's blocklist, so convergence is robust and a firewalled node still hardens itself by pulling. Received bans are applied locally and are poison-safe: secwatch refuses to ban a trusted or shared-edge (e.g. Cloudflare) IP, so a bad entry can't take out a legit address. A leaf can't be pulled from, so its own bans reach the cluster only by push — to keep that reliable it also re-pushes its full blocklist on boot and every cluster.leaf_reshare_every cycles (idempotent), so a leaf-originated ban still converges even if a one-shot push was missed.
• Federated view — open any peer's Cluster tab and it assembles the fleet by querying its peers; no data is centralized.

Roles

Set cluster.role per node — nothing is hard-coded:

Role	For	Behavior
standalone	not clustered (default)	no cluster activity
peer	internal / trusted boxes	full member: shares bans, is queryable, reads peers
leaf	exposed / less-trusted boxes	push-only: contributes its bans + events and pulls the blocklist, but is not queryable and can't read peers

Why a leaf? Your internet-facing box is your riskiest one. As a leaf it still contributes to fleet hardening and stays visible, but a compromise of it can't be used to read your internal fleet's telemetry or poison the shared blocklist. A leaf also only makes outbound connections, so its port can stay firewalled — ideal for a box on an isolated VLAN.

Setup

The easy way — the dashboard. Open the Cluster tab on any node and:

• Create cluster — pick a role, click Create, and copy the shared secret it shows you.
• Join cluster — paste a peer's URL and that secret.

No CLI and no restart — the node starts participating immediately. Manage peers, reveal the secret to add more nodes, or leave the cluster, all from that tab.

The CLI below does the same thing if you prefer the shell or are scripting it.

On the first node:

# secwatch.yaml:  cluster: { role: peer, url: http://THIS-NODE:8931 }   → restart
python -m secwatch.cluster init          # prints the shared secret — copy it

On each other internal node (peer):

# secwatch.yaml:  cluster: { role: peer, url: http://THIS-NODE:8931 }   → restart
python -m secwatch.cluster join http://FIRST-NODE:8931 '<secret>'

On an exposed node (leaf):

# secwatch.yaml:  cluster: { role: leaf }   (no url — a leaf isn't queried)   → restart
python -m secwatch.cluster join http://A-PEER:8931 '<secret>'

CLI: python -m secwatch.cluster {init | join <url> <secret> | list | leave <name> | ping <url>}.

Add a device (one-liner enrollment)

Once a node is in the cluster, its Cluster tab has an Add a device button. Pick the new box's role, click generate, and you get an RMM-style one-liner:

curl -fsSL "http://SEEDER:8931/install.sh?token=…" | sudo sh

Run it (as root) on a fresh Linux box — it installs secwatch, writes the cluster role, and auto-joins. The token is single-use and short-lived.

Security: the installer runs as root and the script embeds the cluster secret so the box can auto-join — so the one-liner is a credential. Mint it just before you use it, and run it only over a network you trust (plain HTTP means the script + secret are visible on-path; prefer HTTPS or a WireGuard/SSH path on untrusted segments). You can disable enrollment by not minting tokens; the /install.sh endpoint does nothing without a valid one.

Networking

• Peers must reach each other on their secwatch port (default 8931) — that's how they gossip bans and serve the fleet view. Each peer must both listen on a LAN-reachable address and allow the port from the other peers.
• A leaf needs only outbound reach to a peer's port — nothing inbound — so an exposed box can keep its port firewalled from everything, including the cluster.
• Keep secwatch's port off the public internet — restrict it to your LAN/subnet.

"Peer shows unreachable / offline"

If a node is in the roster but the dashboard shows it unreachable (and the warning banner lists it), this node can't open :8931 to it. It does not mean the cluster is broken — the roster persists and pushed updates can still queue — but that node's live stats and real-time ban gossip won't flow until it's reachable.

Diagnose on the unreachable node, in order:

1. Is secwatch running and listening on all interfaces?

   systemctl is-active secwatch
   sudo ss -ltn 'sport = :8931'

   If it shows 127.0.0.1:8931 (loopback only), it's not reachable from other hosts. Bind the LAN by adding Environment=SECWATCH_HOST=0.0.0.0 to the unit (e.g. a drop-in at /etc/systemd/system/secwatch.service.d/bind.conf) and restart. (A dashboard password must be set — secwatch refuses to serve 0.0.0.0 without one.)

2. Is a host firewall blocking it? Allow the port from the cluster subnet only, never the whole internet:

   sudo ufw allow from 10.0.0.0/24 to any port 8931 proto tcp   # adjust the subnet

3. Is it a leaf? A leaf is push-only and is never reachable by design — it's expected to show as a leaf, not a warning. Only peers need inbound reach.

Connection refused usually means a firewall REJECT or nothing listening on that address; timed out usually means packets are dropped (firewall DROP or host down).

Verify

Open a peer's dashboard → Cluster tab (it appears only when clustered) → all nodes are listed with liveness, role, event and ban counts. Ban an IP on one box and watch it appear on the others within a gossip cycle (cluster.gossip_secs, default 60s), attributed as cluster:<origin-node>.