Events and webhooks

Events and webhooks

warden history answers afterwards, and only if somebody thinks to look. The interesting moments are the ones nobody is watching for: a port taken while a deploy runs, a lease that quietly expired overnight, a holder that died and left its port booked.

Every one of those already lands in warden's own book. This carries it further.

Watching it happen

$ warden events
09:41:02  registered   shop-api  127.0.0.1:8600
09:41:11  renewed      shop-api  127.0.0.1:8600
09:41:44  released     shop-api  127.0.0.1:8600

warden events --json writes one event per line and flushes each as it arrives, so it pipes into anything:

warden events --json | jq -r 'select(.action == "expired") | .name'

GET /v1/events is the same stream as server-sent events, behind the same token as every other read. See HTTP API.

What warden will tell you about

Thirteen things, in three scopes. warden events --known prints this list on any machine, so nobody has to guess what to write in webhook_events.

Event	Means	Posted by default
port.registered	a name got a port it did not have	yes
port.renewed	a heartbeat, or a re-register on the same port	ask
port.moved	a name came back on a different port	yes
port.released	given back, by its holder or by hand	yes
port.expired	a lease ran out and warden took the port back	yes
node.joined	a warden reported in for the first time	yes
node.returned	one that had gone quiet is answering again	yes
node.stale	one stopped reporting, and is past its lease	yes
node.forgotten	one was removed by hand	ask
firewall.applied	a ruleset was made true on this machine	yes
firewall.confirmed	somebody kept it, and the rollback was called off	yes
firewall.rolled_back	nobody confirmed, so the machine went back	yes
firewall.restored	a snapshot was put back by hand	ask

A node going quiet is said once, not on every listing. The same node still being quiet an hour later is not news.

Choosing which ones

webhook_events takes three ways of naming them, and mixes them freely:

warden settings set webhook_events "firewall.*, node.stale, port.registered"

Written	Means
firewall.*	every event in that scope, including ones added later
node.stale	that one
registered	the same as port.registered — the name 0.2.0 used, still understood

Posting it somewhere

warden setup asks for all of this and posts a test event before saving anything. One setting at a time works too:

warden settings set webhook https://discord.com/api/webhooks/...
warden settings set webhook_format discord
warden settings set webhook_events registered,released
warden webhook --test

Restart the warden afterwards. It reads its settings once, at startup.

Where the address comes from

Discord	Channel → Edit → Integrations → Webhooks → New Webhook → copy the URL
Slack	api.slack.com/apps → your app → Incoming Webhooks → Add New Webhook
Teams	Channel → ⋯ → Workflows → "Post to a channel when a webhook request is received"
Your own	Anything that accepts a POST

The Teams route is a Power Automate flow because the old Office 365 connector cards are being retired; warden posts an adaptive card in a message, which is what that flow accepts.

The four shapes

webhook_format	What arrives
json	The event as it is, signed. Build anything custom on this
discord	An embed, coloured by action
slack	Blocks, plus a text fallback for phone notifications
teams	An adaptive card in a message attachment

What arrives, exactly

With webhook_format = json, this is the whole body. GET /v1/events sends the same shape as the data: of each server-sent event.

Every event has these:

Field
at	when, in UTC
scope	port, node or firewall
action	what happened, within that scope
subject	what it happened to: a service name, a node name, a backend
node	which warden is telling you

A port event carries the registration's own fields at the top level, in the order 0.2.0 sent them. A reader written against that version does not notice that anything widened:

{
  "at": "2026-03-01T09:30:00Z",
  "scope": "port",
  "action": "registered",
  "subject": "shop-api",
  "name": "shop-api",
  "kind": "backend",
  "project": "shop",
  "host": "0.0.0.0",
  "port": 8000,
  "pid": 4242,
  "node": "hub"
}

Field
name	the service, the same as subject
kind	what it said it was
project	its project, or null
host port	where it is bound
pid	the process, when it gave one

Anything else puts what it has in body, and leaves the port fields out rather than sending them empty:

{
  "at": "2026-03-01T09:30:00Z",
  "scope": "node",
  "action": "stale",
  "subject": "build-01",
  "body": {
    "url": "http://build-01:7010",
    "last_seen": "2026-03-01T09:28:00Z"
  },
  "node": "hub"
}

{
  "at": "2026-03-01T09:30:00Z",
  "scope": "firewall",
  "action": "rolled_back",
  "subject": "nftables",
  "body": { "snapshot": 3 },
  "node": "hub"
}

Scope	subject	body holds
node	the node's name	url, and pool version on joining, last_seen when it goes quiet
firewall	the backend	rules, rollback, reason on applying; snapshot on going back

body is open on purpose: an event added later can carry what it needs without the shape changing again.

Headers

Header
X-Warden-Event	the bare action — registered, stale
X-Warden-Scope	port, node, firewall
X-Warden-Node	which warden sent it
X-Warden-Signature	sha256=…, when a secret is set

X-Warden-Event keeps the bare action rather than the full name so that a receiver routing on it since 0.2.0 keeps working. The scope is beside it.

Colours, one event at a time

warden settings set webhook_colours "node.stale=#e5544b, port.moved=#c8892a"

Discord and Teams take a colour per message, and each event comes with one that suits it - green for a node reporting in, red for one going quiet. Naming one changes that one and leaves the rest alone. Slack has no colour to set; its messages carry the same words.

Checking it really came from you

With json and a webhook_secret, every post carries a signature:

X-Warden-Signature: sha256=b1646dcf...

That is an HMAC over exactly the bytes that were sent, keyed with the secret. warden serialises the body itself rather than leaving it to the HTTP client, because a signature over a body somebody else re-serialises signs something else.

expected = hmac.new(secret.encode(), request.body, hashlib.sha256).hexdigest()
hmac.compare_digest(request.headers["X-Warden-Signature"], f"sha256={expected}")

Every post also carries X-Warden-Node and X-Warden-Event, whatever the shape.

Two things that are not negotiable

Handing out a port never waits on a webhook. Delivery happens after the change is committed and off the request path, retried three times and then given up on. A chat server having a bad afternoon cannot make a port take longer to hand out. A reader that falls behind loses its oldest event, never its newest, and never the write.

A webhook that fails is not allowed to fail quietly. From the inside, one failing all day looks exactly like a quiet day. warden webhook and warden doctor both report the target, the shape and the last failure:

$ warden doctor
...
warn  events to https://discord.com/... are not arriving - Server error '502 Bad Gateway'

The address is always cut back to its host. A webhook URL is a credential, and its path is the half worth stealing.

In a fleet

Every warden reports its own events, so every node you want to hear from needs the setting. A hub does not relay its nodes' events — a node that cannot reach the hub still hands out ports, and the same independence applies here.