Cluster

Cluster

One warden looks after one machine. Point several at a common one and you get a fleet: every node still runs its own show, and the hub knows who exists, what each hands out, and who has stopped answering.

The shape

   a service                a service                 a service
       |                        |                         |
       v                        v                         v
  warden on               warden on                 warden on
  build-01                web-02                    db-03
  pool 9000-9099          pool 9000-9099            pool 9000-9099
       |                        |                         |
       |  POST /v1/nodes, three times per lease            |
       +------------------------+-------------------------+
                                |
                                v
                          warden on hub
                          knows all three
                          warden nodes

A service always talks to the warden on its own machine. It never needs to know the hub exists, and nothing it does stops working when the hub does.

Why the hub is a directory and never the owner

This is the decision everything else follows from, so it is worth being plain about.

warden does not only track what it handed out. Before giving a port away it tries to bind it, and skips it if something else got there first — a container, a service someone started by hand, a stray process from last week. That check only has an answer on the machine itself. From the hub, 10.0.0.7:9000 either refuses a connection or does not, which says nothing about whether the port can be bound.

Move allocation into the hub and warden loses the one thing that makes it more than a shared spreadsheet. It also makes the hub load-bearing: nothing could start anywhere while it is down.

So allocation stays where the machine is. The hub aggregates.

Setting it up

On the machine that will be the hub — nothing special, it is just a warden that nobody points upward:

WARDEN_CLUSTER_TOKEN=pick-a-secret warden serve

On every other machine:

WARDEN_CLUSTER_TOKEN=pick-a-secret \
WARDEN_NODE=build-01 \
WARDEN_UPSTREAM=http://hub:7010 \
WARDEN_ADVERTISE=http://build-01:7010 \
  warden serve

That is the whole configuration. Within seconds:

$ warden nodes --url http://hub:7010
NODE      URL                   POOL       VERSION  STATUS  LAST SEEN
build-01  http://build-01:7010  9000-9099  0.1.0    online  4s ago
web-02    http://web-02:7010    9000-9099  0.1.0    online  11s ago
db-03     http://db-03:7010     9000-9099  0.1.0    stale   6m ago

--json gives the same as data.

Trying it on one machine first

Two wardens side by side, before spreading anything over two servers:

# hub
WARDEN_DATABASE=/tmp/hub.db WARDEN_CLUSTER_TOKEN=secret WARDEN_NODE=hub \
  warden serve --port 7010 --pool 8000-8099

# node
WARDEN_DATABASE=/tmp/edge.db WARDEN_CLUSTER_TOKEN=secret WARDEN_NODE=build-01 \
WARDEN_UPSTREAM=http://127.0.0.1:7010 WARDEN_NODE_TTL=30 \
  warden serve --port 7020 --pool 9000-9099

Give them separate databases, separate ports and separate pools, then:

warden nodes --url http://127.0.0.1:7010
warden register shop-api --kind backend --url http://127.0.0.1:7020

Stop the hub and register something else on the node — it still works. Start the hub again and the node reports back on its own.

Both may use loopback here, because a hub on the same machine reaches it perfectly well. See Addresses for when that is refused.

The life of a node

Announcing. On start, a node with WARDEN_UPSTREAM set posts its name, address, pool and version to the hub. The first report creates the entry, every later one refreshes it. first_seen survives; nothing else does, so a node that moved simply reports its new address.

Renewing. Three times per WARDEN_NODE_TTL, in the background. Reporting that often means two lost messages still leave a margin before the entry lapses.

Going stale. A node that stops reporting keeps its row and is shown as stale. It is not deleted, and that is deliberate: a server that has stopped answering is a fact worth seeing. Dropping it silently makes it look like it was never there, which is exactly the wrong impression when a machine has fallen over.

Being forgotten. For a machine that is gone for good:

warden nodes --forget build-01

A deliberate act, by a person.

What survives what

Situation	What happens
Hub is down	Nodes hand out ports as usual, log a warning on each attempt, and re-announce by themselves once it is back
Hub is down when a node starts	The node starts anyway; reporting runs in the background and never blocks the boot
A node is down	The hub keeps serving and shows that node as stale once its lease lapses
Node has the wrong cluster token	The hub answers 401; the node logs it and keeps working locally
Node advertises an unreachable address	Refused at startup, not left to fail quietly later

The pattern is the same throughout: a node's own work never depends on anything across the network.

Two tokens, on purpose

Variable	Who carries it	What it opens
WARDEN_CLUSTER_TOKEN	wardens, to each other	announcing, and reading
WARDEN_TOKEN	a person, or a tool acting for one	everything, including anything that changes state

A node has to be able to report without being handed the token that lets a person delete things. The cluster token also opens reads, because a hub asking its nodes what they hold is exactly that; it opens nothing that changes state, and it can never open a door WARDEN_TOKEN has closed.

Both are empty by default, which is fine while warden listens on loopback. Set them before binding to anything else.

Addresses

WARDEN_ADVERTISE is the address the hub should use to reach the node. It defaults to the node's own listening address, which is right when that is already reachable and wrong when the node listens on 0.0.0.0 or 127.0.0.1.

A node pointing at a hub elsewhere while advertising a loopback address is refused at startup:

the warden at http://hub:7010 cannot reach this one at http://127.0.0.1:7020;
set WARDEN_ADVERTISE to an address it can use

The alternative is worse than a startup failure: the hub records an address it can never open, everything looks healthy, and the mistake only surfaces the first time somebody tries to use the entry.

Two wardens on the same machine may both use loopback, since a hub there reaches it perfectly well.

What the hub keeps

One row per node, in the same database as its own registrations:

Field	Meaning
name	What the node calls itself (WARDEN_NODE)
url	The address the node says to use (WARDEN_ADVERTISE)
pool_start, pool_end	The range that node hands out
version	Which warden it is running
first_seen	When it first reported, never overwritten
last_seen	Its most recent report
expires_at	When the entry goes stale

status is not stored. It is online while expires_at is in the future and stale afterwards, so it is true at the moment you ask rather than at the moment something last wrote it.

Telling one warden from another

$ curl -s http://hub:7010/health
{"status":"ok","version":"0.1.0","node":"hub","role":"hub","services":0,"nodes":3}

$ curl -s http://build-01:7010/health
{"status":"ok","version":"0.1.0","node":"build-01","role":"edge","services":2,"nodes":0}

role is edge when the warden reports to another and hub when it does not. It is the quickest way to tell a misconfigured node from a hub nobody is reporting to.

One view across the fleet

The hub can answer for every node at once:

$ warden ls --all --url http://hub:7010
NODE      SERVICE       KIND     PROJECT  ADDRESS         PID
build-01  build-runner  worker   ci       127.0.0.1:9000  -
build-01  build-cache   cache    ci       127.0.0.1:9001  -
hub       hub-api       backend  shop     127.0.0.1:8000  -

--project and --kind filter the whole fleet, not just the warden you asked.

A node that does not answer is named, never dropped. It goes to standard error, so a pipe still gets a clean table:

$ warden ls --all --url http://hub:7010
NODE  SERVICE  KIND     PROJECT  ADDRESS         PID
hub   hub-api  backend  shop     127.0.0.1:8000  -
build-01 (http://build-01:7010) could not be reached

That distinction matters more than it looks. A shorter list because a machine was down reads exactly like a shorter list because nothing is registered there, and those are not the same thing at all. --json keeps them in separate fields for the same reason.

Every node is asked, including stale ones: a node the hub lost sight of may be perfectly well and simply unable to report, and skipping it would hide real services. They are asked in parallel, so a rack of dead nodes costs one timeout in total rather than one each.

To ask one named node about one service:

$ warden get build-01/build-runner --url http://hub:7010
127.0.0.1:9000

A name two machines both hold

A service name is unique per node and never across the fleet, so the aggregated view is the only place a clash can be seen at all:

two nodes hold shop-api: build-01, web-02

It is nearly always two projects that drifted apart rather than anybody's plan. Only names inside whatever --project or --kind you asked for are reported, so the warning can never point at a row that is not in the table above it.

Ports across the fleet

warden pool --all      # how much every node has left, and the totals
warden ports --all     # every socket bound anywhere, with a NODE column
warden tui --all       # both of those, live

A port number means nothing on its own across machines: 3000 on two nodes is two unrelated processes, and the node is what tells them apart.

Writing to one node through the hub

warden register runner --kind worker --node build-01 --url http://hub:7010
warden heartbeat runner --ttl 600 --node build-01 --url http://hub:7010
warden release runner --node build-01 --url http://hub:7010

The node still decides. Only the machine itself can try to bind a port, so the hub passes the request along and hands back exactly what came out — a refusal arrives in that node's own words, with its own status code.

Forwarding takes WARDEN_TOKEN, not the cluster token, and carries the caller's own authorization to the node. The cluster token reads and announces; letting it write through the hub would make the hub the one door it can change things by. Each node also keeps its own WARDEN_ALLOW_KILL: the hub opens nothing that was shut.

Where that token is allowed to go

A name is pinned to the address it first announced. A second announcement claiming a different one is refused:

build-01 is already at http://build-01:7010 and now claims http://elsewhere:7010.
If it really moved, `warden nodes --forget build-01` first

The reason is worth stating. Anyone holding the cluster token can announce a node, and the hub sends a person's token to whatever address that node gave. Without the pin, re-announcing an existing node at a machine of your own would collect the next token forwarded through the hub. With it, a move is a deliberate act by a person rather than a side effect of holding a shared secret.

WARDEN_REQUIRE_HTTPS refuses to register a plain HTTP node at all, and refuses to forward a token to one that was registered before it was turned on. It is off by default, because turning it on would break every setup on this page; until then each plain-HTTP node is named in the log the first time a token goes there:

sending a token to http://build-01:7010 over plain HTTP; set WARDEN_REQUIRE_HTTPS
once the fleet can speak it