Events and webhooks

Events and webhooks

warden history answers afterwards, and only if somebody thinks to look. The interesting moments are the ones nobody is watching for: a port taken while a deploy runs, a lease that quietly expired overnight, a holder that died and left its port booked.

Every one of those already lands in warden's own book. This carries it further.

Watching it happen

$ warden events
09:41:02  registered   shop-api  127.0.0.1:8600
09:41:11  renewed      shop-api  127.0.0.1:8600
09:41:44  released     shop-api  127.0.0.1:8600

warden events --json writes one event per line and flushes each as it arrives, so it pipes into anything:

warden events --json | jq -r 'select(.action == "expired") | .name'

GET /v1/events is the same stream as server-sent events, behind the same token as every other read. See HTTP API.

What warden will tell you about

Thirteen things, in three scopes. warden events --known prints this list on any machine, so nobody has to guess what to write in webhook_events.

Event	Means	Posted by default
port.registered	a name got a port it did not have	yes
port.renewed	a heartbeat, or a re-register on the same port	ask
port.moved	a name came back on a different port	yes
port.released	given back, by its holder or by hand	yes
port.expired	a lease ran out and warden took the port back	yes
node.joined	a warden reported in for the first time	yes
node.returned	one that had gone quiet is answering again	yes
node.stale	one stopped reporting, and is past its lease	yes
node.forgotten	one was removed by hand	ask
firewall.applied	a ruleset was made true on this machine	yes
firewall.confirmed	somebody kept it, and the rollback was called off	yes
firewall.rolled_back	nobody confirmed, so the machine went back	yes
firewall.restored	a snapshot was put back by hand	ask

A node going quiet is said once, not on every listing. The same node still being quiet an hour later is not news.

Choosing which ones

webhook_events takes three ways of naming them, and mixes them freely:

warden settings set webhook_events "firewall.*, node.stale, port.registered"

Written	Means
firewall.*	every event in that scope, including ones added later
node.stale	that one
registered	the same as port.registered — the name 0.2.0 used, still understood

Posting it somewhere

warden setup asks for all of this and posts a test event before saving anything. One setting at a time works too:

warden settings set webhook https://discord.com/api/webhooks/...
warden settings set webhook_format discord
warden settings set webhook_events registered,released
warden webhook --test

Restart the warden afterwards. It reads its settings once, at startup.

Where the address comes from

Discord	Channel → Edit → Integrations → Webhooks → New Webhook → copy the URL
Slack	api.slack.com/apps → your app → Incoming Webhooks → Add New Webhook
Teams	Channel → ⋯ → Workflows → "Post to a channel when a webhook request is received"
Your own	Anything that accepts a POST

The Teams route is a Power Automate flow because the old Office 365 connector cards are being retired; warden posts an adaptive card in a message, which is what that flow accepts.

The four shapes

webhook_format	What arrives
json	The event as it is, signed. Build anything custom on this
discord	An embed: the event above, the subject as the title, the node in the footer
slack	A coloured attachment with the facts as fields, plus a text fallback for phone notifications
teams	An adaptive card with a toned header band, an icon and a fact set

What arrives, exactly

With webhook_format = json, this is the whole body. GET /v1/events sends the same shape as the data: of each server-sent event.

Every event has these:

Field
at	when, in UTC
scope	port, node or firewall
action	what happened, within that scope
subject	what it happened to: a service name, a node name, a backend
node	which warden is telling you

A port event carries the registration's own fields at the top level, in the order 0.2.0 sent them. A reader written against that version does not notice that anything widened:

{
  "at": "2026-03-01T09:30:00Z",
  "scope": "port",
  "action": "registered",
  "subject": "shop-api",
  "name": "shop-api",
  "kind": "backend",
  "project": "shop",
  "host": "0.0.0.0",
  "port": 8000,
  "pid": 4242,
  "node": "hub"
}

Field
name	the service, the same as subject
kind	what it said it was
project	its project, or null
host port	where it is bound
pid	the process, when it gave one

Anything else puts what it has in body, and leaves the port fields out rather than sending them empty:

{
  "at": "2026-03-01T09:30:00Z",
  "scope": "node",
  "action": "stale",
  "subject": "build-01",
  "body": {
    "url": "http://build-01:7010",
    "last_seen": "2026-03-01T09:28:00Z"
  },
  "node": "hub"
}

{
  "at": "2026-03-01T09:30:00Z",
  "scope": "firewall",
  "action": "rolled_back",
  "subject": "nftables",
  "body": { "snapshot": 3 },
  "node": "hub"
}

Scope	subject	body holds
node	the node's name	url, and pool version on joining, last_seen when it goes quiet
firewall	the backend	rules, rollback, reason on applying; snapshot on going back

body is open on purpose: an event added later can carry what it needs without the shape changing again.

Headers

Header
X-Warden-Event	the bare action — registered, stale
X-Warden-Scope	port, node, firewall
X-Warden-Node	which warden sent it
X-Warden-Signature	sha256=…, when a secret is set

X-Warden-Event keeps the bare action rather than the full name so that a receiver routing on it since 0.2.0 keeps working. The scope is beside it.

How a message looks, one event at a time

Each event comes with a colour, an icon and the words that follow the subject — green and an inbox tray for a port being taken, red for a node going quiet. Naming one changes that one and leaves the other twelve alone.

The Embed tab of warden settings picks an event, takes the three, and draws the message it would send underneath as you type:

warden settings embed

It writes three settings, which can be set by hand instead:

warden settings set webhook_colours "node.stale=#e5544b, port.moved=#c8892a"
warden settings set webhook_titles "node.stale=has stopped answering"
warden settings set webhook_icons "node.stale=!, port.moved=-"

! build-01
has stopped answering

Setting	What it changes
webhook_colours	The stripe on a Discord embed, the bar on a Slack attachment. #4c9a5b or 4c9a5b
webhook_titles	The words after the subject, in every shape that shows words
webhook_icons	What stands in front of the subject. A single - means none at all

In warden.toml all three are inline tables, one entry per event:

webhook_colours = { "node.stale" = "#e5544b", "port.moved" = "#c8892a" }
webhook_titles = { "node.stale" = "has stopped answering" }
webhook_icons = { "node.stale" = "!" }

A colour that is not one keeps the colour it came with, rather than posting something the far end will refuse: losing the stripe beats losing the message.

json carries none of it. It sends the event, and whatever reads it decides how that should look — which is the point of having it.

There is no template to write. A shape that can be edited freely is a shape that can be edited into invalid JSON, and a broken embed is not noticed until the moment it matters.

What each shape does with them

Discord puts the event name and the mascot in the author line, the icon and the subject in the title, the sentence under it, the facts as inline fields, and the node in the footer — the node is the same on every message in the channel and does not deserve a field of its own. Addresses are set as code, so they are not something to hunt for in the middle of a line.

Slack wraps everything in one attachment, which is the only way Slack will draw a coloured bar down the side of a message. The mascot rides along as the section accessory, the facts go in a two-column field block, and the time is sent as <!date^…> so it renders in whichever timezone the person reading it is sitting in. The plain text is still there for phone notifications and for clients too old for blocks.

Teams cannot be given a hex colour, so the header band takes the nearest tone the format has a name for — good, warning, attention — which comes from what the event means rather than from the colour you chose for it. The icon sits beside the subject, and the facts go in a fact set below. An adaptive card has no code spans, so an address there is written plainly.

None of the three repeats what its own sentence already said: an address in the line is not an address in a field underneath it as well.

Seeing all thirteen without waiting for them

warden webhook --all                 # one made-up event of each kind, in order
warden webhook --event node.stale    # just that one
warden webhook --test                # one port.registered, as before

Posted one at a time and in order, because a chat window shows them in the order they land. Nothing is registered, nothing changes: the events are made up on the spot, and only the message is real.

port.registered       posted
port.renewed          posted
...
firewall.restored     posted
to https://discord.com/api/webhooks/...

Checking it really came from you

With json and a webhook_secret, every post carries a signature:

X-Warden-Signature: sha256=b1646dcf...

That is an HMAC over exactly the bytes that were sent, keyed with the secret. warden serialises the body itself rather than leaving it to the HTTP client, because a signature over a body somebody else re-serialises signs something else.

expected = hmac.new(secret.encode(), request.body, hashlib.sha256).hexdigest()
hmac.compare_digest(request.headers["X-Warden-Signature"], f"sha256={expected}")

Every post also carries X-Warden-Node and X-Warden-Event, whatever the shape.

Two things that are not negotiable

Handing out a port never waits on a webhook. Delivery happens after the change is committed and off the request path, retried three times and then given up on. A chat server having a bad afternoon cannot make a port take longer to hand out. A reader that falls behind loses its oldest event, never its newest, and never the write.

A webhook that fails is not allowed to fail quietly. From the inside, one failing all day looks exactly like a quiet day. warden webhook and warden doctor both report the target, the shape and the last failure:

$ warden doctor
...
warn  events to https://discord.com/... are not arriving - Server error '502 Bad Gateway'

The address is always cut back to its host. A webhook URL is a credential, and its path is the half worth stealing.

In a fleet

Every warden reports its own events, so every node you want to hear from needs the setting. A hub does not relay its nodes' events — a node that cannot reach the hub still hands out ports, and the same independence applies here.