-
Notifications
You must be signed in to change notification settings - Fork 0
Troubleshooting
Every message below is one warden actually prints. Find yours and read across.
Before any of it, though:
warden doctorIt checks the things this page describes one at a time — whether a warden is answering, where its settings came from, whether it is open without a token, how much pool is left, whether registrations are held by processes that are gone, whether the nodes are reporting — and names the ones that are wrong.
Nothing is listening there. Either it is not running, or it is running somewhere else.
Note that warden ports, warden kill and the dashboard's ports view do not
need one at all — if those work and nothing else does, the registry simply is not
running.
warden ports --port 7010 # is anything on it at all?
curl localhost:7010/health # is it answering?If your warden listens elsewhere, point the client at it:
warden ls --url http://hub:7010
# or, once
export WARDEN_URL=http://hub:7010Note that warden ports and warden kill never need a server. If those work and
everything else does not, the registry simply is not running.
Something already holds port 7010. Find it and decide:
warden ports --port 7010
warden serve --port 7011 # or move out of its way--pool 8999-8000 the wrong way round, or a WARDEN_POOL_START above
WARDEN_POOL_END.
A node pointing at a hub on another machine while advertising a loopback address. The hub would record an address it can never open, so warden refuses at startup rather than letting it fail silently weeks later.
WARDEN_ADVERTISE=http://build-01:7010 # a name or address the hub can useTwo wardens on the same machine may both use loopback; the check only applies when the hub is elsewhere.
You asked for that exact port with --require-port and another registration has
it. Either take what you are given:
warden register shop-web --kind frontend --preferred-port 8000or free it first, if the holder is finished with it:
warden release shop-apiNothing in the registry holds it, but something on the machine is bound to it. See what:
warden ports --port 8000It is in WARDEN_RESERVED, or it is the registry's own port. Reserved ports are
never handed out, whatever is asked.
The pool is genuinely full. Look for registrations nothing is using any more:
warden ls
warden poolOld entries stay until released, which is what makes ports stick across restarts. Release the dead ones, or widen the pool:
warden serve --pool 8000-9999For things that will not clean up after themselves, register with --ttl so the
entry expires by itself.
They mean opposite things. --preferred-port is a wish, --require-port is a
demand.
The request body did not hold up. Common causes:
- a name with a capital letter or a space — names are lowercase letters, digits,
._and-, starting with a letter or digit - a misspelled field. Unknown fields are rejected rather than ignored, so
prefered_portfails loudly instead of quietly giving you a different port
That socket belongs to another user. The operating system does not hand over another account's process details, which is normal rather than a fault. warden counts them:
3 of 42 belong to another user - run warden as administrator to see them
Run elevated — administrator on Windows, sudo on Linux — if you need those
rows. On macOS it is not a matter of missing rows: it refuses the whole listing
without root. See Ports and processes.
warden asked politely and waited five seconds. Some processes trap that signal.
warden kill 3000 --forceRefused on purpose. Pids 0 to 4 belong to the system, and warden will not end its own process.
Your account may not touch it. Run elevated, or stop it however its owner normally would.
DELETE /v1/listeners/{pid} is off unless you ask for it:
WARDEN_ALLOW_KILL=true warden serveIt is off by default because a warden reachable from the network would otherwise
let anyone holding the token end processes on that machine. warden kill on the
command line is unaffected — it acts locally and never asks the API.
Work down this list on the node:
-
Is
WARDEN_UPSTREAMset?curl localhost:7010/health—rolemust sayedge. If it sayshub, the node does not know it should report to anyone. -
Look at the node's log. It says so on every attempt:
could not reach the warden at http://hub:7010: All connection attempts failed -
Can it reach the hub at all?
curl http://hub:7010/healthfrom the node. -
Do the cluster tokens match? A mismatch shows as a
401in the node's log.WARDEN_CLUSTER_TOKENmust be identical on both, and it is not the same setting asWARDEN_TOKEN. -
Give it a moment. A node reports three times per
WARDEN_NODE_TTL, so up to 30 seconds with the default of 90.
The node is sending the wrong secret, or none. Announcing takes
WARDEN_CLUSTER_TOKEN; the human WARDEN_TOKEN does not work in its place, and
that is deliberate.
Neither token matched. Reading accepts either WARDEN_TOKEN or
WARDEN_CLUSTER_TOKEN; anything that changes something takes WARDEN_TOKEN.
warden nodes --url http://hub:7010 --token "$WARDEN_TOKEN"It has not reported within its lease. Either it is down, or it cannot reach the hub. Its row is deliberately kept rather than deleted — a server that stopped answering is worth seeing. When it is gone for good:
warden nodes --forget build-01Nothing. They keep handing out ports, warn in their logs on each attempt, and report back on their own once it returns. That is the whole point of the arrangement: a node's own work never depends on the network.
warden ls asks one warden. Add --all to ask the whole fleet:
warden ls --all --url http://hub:7010A service name is unique per node, never across the fleet, so two machines can both answer to it. Nearly always two projects that drifted apart. Rename one, or leave it if both are meant to exist — nothing breaks either way, the fleet view just says so instead of hiding it.
The --node you named has never announced itself to the warden you asked.
warden nodes lists the ones it knows; check WARDEN_NODE on the machine you
meant, since that is the name it reports under.
Forwarding a change takes WARDEN_TOKEN, not the cluster token, and the token
you send has to be one the target node accepts as well — the hub carries your
authorization on rather than vouching for you with its own.
It does not leave it out silently. Look at standard error:
build-01 (http://build-01:7010) could not be reached
The reason says which of these it is:
| Reason | Means |
|---|---|
could not be reached |
Nothing answered at the address the node advertised. Check WARDEN_ADVERTISE on the node — the hub uses that address, not the one you typed |
refused the token - check WARDEN_CLUSTER_TOKEN matches |
The hub's cluster token is not the node's |
did not answer within 3s |
The node is reachable but too slow |
answered 500 |
The node is up but its own registry failed |
warden webhook says what the running warden is configured with;
warden webhook --test posts from this machine using the file on disk. If the
two disagree, the warden has not been restarted since the setting was written.
It reads its settings once, at startup.
webhook_format is still json. Set it to discord, slack or teams and
restart.
By design. webhook_events leaves renewed out unless you ask for it — a
channel told about every heartbeat is a channel people mute within the week.
Only json is signed, and only when webhook_secret is set. The HMAC covers
exactly the bytes that were sent, so a receiver that re-serialises the body
before checking is signing something else.
TERM is unset or dumb, which is what a cron job, a serial console and some
build runners get. warden setup falls back to asking one question at a time;
warden ls and warden ports read the same things as the dashboard in plain
text.
Under 84 columns it puts each label above its field and drops the mascot. It is meant to work at 80 by 24; below about 60 columns a terminal is smaller than the questions are.
On Linux, a systemd user unit belongs to your session. Without lingering it
stops when your last one ends, which on a server is when you log out of ssh.
warden service install looks and says so:
this account does not linger, so the unit stops when its last session ends -
on a server, when you log out. `sudo loginctl enable-linger you` keeps it running.
warden says it rather than doing it: enabling lingering needs root.
The console is running a legacy code page that cannot encode the block character. warden falls back rather than aborting the command over a decoration. To get the blocks on Windows:
chcp 65001
The terminal is under 30 rows tall, so the banner gives way to the table. Make the window taller.
No node has announced itself to this warden. If you expected some, work through the fleet section above.
| System | Path |
|---|---|
| Windows | %LOCALAPPDATA%\warden\registry.db |
| Linux | ~/.local/share/warden/registry.db |
| macOS | ~/Library/Application Support/warden/registry.db |
Override with WARDEN_DATABASE or --database. Two wardens on one machine need
separate files.
Stop warden, delete the database, start it again. Every registration and every known node goes with it; nothing else is affected.
No. A schema change adds what it needs when the database is next opened. A database written by an older warden keeps its registrations.
curl -s localhost:7010/health is the fastest single check — it tells you the
version, the node name, whether this warden reports to another, and how much it
is holding, without a token.
If something looks like a defect, the repository is at https://github.com/vxnsin/warden/issues.
0.1.0 is the newest there is
You are on it. The check compares against GitHub releases, not tags or commits. If it has no answer at all it says why instead — the machine is offline, or GitHub rate limited it.
The machine being asked has not been told it may update itself:
WARDEN_ALLOW_REMOTE_UPDATE=trueOff by default on purpose. See Updates.
It is allowed to update but does not know how. The command lives on that machine, never in the request:
WARDEN_UPDATE_COMMAND="/usr/local/bin/update-warden.sh"The program named is not on that machine, or not on its PATH. Use a full path.
The command runs without a shell, so cd x && y will not work — put it in a
script.
Your command failed and warden passed its output back. Read that output: usually the machine cannot reach wherever it pulls from, or the working tree is dirty.
Given up on. Make the command finish faster, or have it start the real work in the background and return.
Exactly as designed: every node is asked, every answer reported, and one machine being away neither stops the others nor fails the command.
Your update command has to restart the service; warden does not restart itself,
because it cannot report the result of a command that ends the process running
it. End the script with systemctl --user restart warden or the equivalent.
A node's name is pinned to the address it first announced. This is a second announcement asking for a different one.
If the machine really did move:
warden nodes --forget build-01Then let it announce itself again. If it did not move, something else is using that name — worth finding out what, since the hub sends a person's token to whatever address a node gives it.
WARDEN_REQUIRE_HTTPS is on and the node is at an http:// address. Either put
the node behind HTTPS and let it re-announce, or turn the requirement off while
the fleet is on a network you trust.
Not an error. The hub is forwarding a person's token to a node over an unencrypted connection, which is fine on a trusted network and worth fixing before it is not one. Said once per node, not once per request.
warden — nothing binds a port without asking ·
uv tool install warden-ports
Repository · Issues · Releases · PyPI · MIT
Getting started
While it runs
Several machines
Reference