Cluster

Cluster

One warden looks after one machine. Point several at a common one and you get a fleet: every node still runs its own show, and the hub knows who exists, what each hands out, and who has stopped answering.

The shape

   a service                a service                 a service
       |                        |                         |
       v                        v                         v
  warden on               warden on                 warden on
  build-01                web-02                    db-03
  pool 9000-9099          pool 9000-9099            pool 9000-9099
       |                        |                         |
       |  POST /v1/nodes, three times per lease            |
       +------------------------+-------------------------+
                                |
                                v
                          warden on hub
                          knows all three
                          warden nodes

A service always talks to the warden on its own machine. It never needs to know the hub exists, and nothing it does stops working when the hub does.

Why the hub is a directory and never the owner

This is the decision everything else follows from, so it is worth being plain about.

warden does not only track what it handed out. Before giving a port away it tries to bind it, and skips it if something else got there first — a container, a service someone started by hand, a stray process from last week. That check only has an answer on the machine itself. From the hub, 10.0.0.7:9000 either refuses a connection or does not, which says nothing about whether the port can be bound.

Move allocation into the hub and warden loses the one thing that makes it more than a shared spreadsheet. It also makes the hub load-bearing: nothing could start anywhere while it is down.

So allocation stays where the machine is. The hub aggregates.

Setting it up

On the machine that will be the hub — nothing special, it is just a warden that nobody points upward:

WARDEN_CLUSTER_TOKEN=pick-a-secret warden serve

On every other machine:

WARDEN_CLUSTER_TOKEN=pick-a-secret \
WARDEN_NODE=build-01 \
WARDEN_UPSTREAM=http://hub:7010 \
WARDEN_ADVERTISE=http://build-01:7010 \
  warden serve

That is the whole configuration. Within seconds:

$ warden nodes --url http://hub:7010
NODE      URL                   POOL       VERSION  STATUS  LAST SEEN
build-01  http://build-01:7010  9000-9099  0.1.0    online  4s ago
web-02    http://web-02:7010    9000-9099  0.1.0    online  11s ago
db-03     http://db-03:7010     9000-9099  0.1.0    stale   6m ago

--json gives the same as data.

Trying it on one machine first

Two wardens side by side, before spreading anything over two servers:

# hub
WARDEN_DATABASE=/tmp/hub.db WARDEN_CLUSTER_TOKEN=secret WARDEN_NODE=hub \
  warden serve --port 7010 --pool 8000-8099

# node
WARDEN_DATABASE=/tmp/edge.db WARDEN_CLUSTER_TOKEN=secret WARDEN_NODE=build-01 \
WARDEN_UPSTREAM=http://127.0.0.1:7010 WARDEN_NODE_TTL=30 \
  warden serve --port 7020 --pool 9000-9099

Give them separate databases, separate ports and separate pools, then:

warden nodes --url http://127.0.0.1:7010
warden register shop-api --kind backend --url http://127.0.0.1:7020

Stop the hub and register something else on the node — it still works. Start the hub again and the node reports back on its own.

Both may use loopback here, because a hub on the same machine reaches it perfectly well. See Addresses for when that is refused.

The life of a node

Announcing. On start, a node with WARDEN_UPSTREAM set posts its name, address, pool and version to the hub. The first report creates the entry, every later one refreshes it. first_seen survives; nothing else does, so a node that moved simply reports its new address.

Renewing. Three times per WARDEN_NODE_TTL, in the background. Reporting that often means two lost messages still leave a margin before the entry lapses.

Going stale. A node that stops reporting keeps its row and is shown as stale. It is not deleted, and that is deliberate: a server that has stopped answering is a fact worth seeing. Dropping it silently makes it look like it was never there, which is exactly the wrong impression when a machine has fallen over.

Being forgotten. For a machine that is gone for good:

warden nodes --forget build-01

A deliberate act, by a person.

What survives what

Situation	What happens
Hub is down	Nodes hand out ports as usual, log a warning on each attempt, and re-announce by themselves once it is back
Hub is down when a node starts	The node starts anyway; reporting runs in the background and never blocks the boot
A node is down	The hub keeps serving and shows that node as stale once its lease lapses
Node has the wrong cluster token	The hub answers 401; the node logs it and keeps working locally
Node advertises an unreachable address	Refused at startup, not left to fail quietly later

The pattern is the same throughout: a node's own work never depends on anything across the network.

Two tokens, on purpose

Variable	Who carries it	What it opens
WARDEN_CLUSTER_TOKEN	wardens, to each other	POST /v1/nodes
WARDEN_TOKEN	a person, or a tool acting for one	everything else, including reading the fleet

A node has to be able to report without being handed the token that lets a person read and delete everything. And someone reading the node list is doing something different from a machine checking in. Neither token works in place of the other.

Both are empty by default, which is fine while warden listens on loopback. Set them before binding to anything else.

Addresses

WARDEN_ADVERTISE is the address the hub should use to reach the node. It defaults to the node's own listening address, which is right when that is already reachable and wrong when the node listens on 0.0.0.0 or 127.0.0.1.

A node pointing at a hub elsewhere while advertising a loopback address is refused at startup:

the warden at http://hub:7010 cannot reach this one at http://127.0.0.1:7020;
set WARDEN_ADVERTISE to an address it can use

The alternative is worse than a startup failure: the hub records an address it can never open, everything looks healthy, and the mistake only surfaces the first time somebody tries to use the entry.

Two wardens on the same machine may both use loopback, since a hub there reaches it perfectly well.

What the hub keeps

One row per node, in the same database as its own registrations:

Field	Meaning
name	What the node calls itself (WARDEN_NODE)
url	The address the node says to use (WARDEN_ADVERTISE)
pool_start, pool_end	The range that node hands out
version	Which warden it is running
first_seen	When it first reported, never overwritten
last_seen	Its most recent report
expires_at	When the entry goes stale

status is not stored. It is online while expires_at is in the future and stale afterwards, so it is true at the moment you ask rather than at the moment something last wrote it.

Telling one warden from another

$ curl -s http://hub:7010/health
{"status":"ok","version":"0.1.0","node":"hub","role":"hub","services":0,"nodes":3}

$ curl -s http://build-01:7010/health
{"status":"ok","version":"0.1.0","node":"build-01","role":"edge","services":2,"nodes":0}

role is edge when the warden reports to another and hub when it does not. It is the quickest way to tell a misconfigured node from a hub nobody is reporting to.

What this does not do yet

The hub knows the fleet; it does not yet act on it. Reading across all nodes, registering through the hub onto a named node, and a fleet view in the dashboard are tracked in #2, #3, #4, #5 and #6.

Today, warden ls and warden ports speak to whichever single warden --url points at.