Troubleshooting

Troubleshooting

Every message below is one warden actually prints. Find yours and read across.

Starting up

no warden reachable at http://127.0.0.1:7010 - start one with 'warden serve'

Nothing is listening there. Either it is not running, or it is running somewhere else.

warden ports --port 7010     # is anything on it at all?
curl localhost:7010/health   # is it answering?

If your warden listens elsewhere, point the client at it:

warden ls --url http://hub:7010
# or, once
export WARDEN_URL=http://hub:7010

Note that warden ports and warden kill never need a server. If those work and everything else does not, the registry simply is not running.

[Errno 10048] error while attempting to bind on address

Something already holds port 7010. Find it and decide:

warden ports --port 7010
warden serve --port 7011      # or move out of its way

pool_start must not be greater than pool_end

--pool 8999-8000 the wrong way round, or a WARDEN_POOL_START above WARDEN_POOL_END.

the warden at http://hub:7010 cannot reach this one at http://127.0.0.1:7020

A node pointing at a hub on another machine while advertising a loopback address. The hub would record an address it can never open, so warden refuses at startup rather than letting it fail silently weeks later.

WARDEN_ADVERTISE=http://build-01:7010    # a name or address the hub can use

Two wardens on the same machine may both use loopback; the check only applies when the hub is elsewhere.

Registering

port 8000 is held by 'shop-api'

You asked for that exact port with --require-port and another registration has it. Either take what you are given:

warden register shop-web --kind frontend --preferred-port 8000

or free it first, if the holder is finished with it:

warden release shop-api

port 8000 is already in use on 127.0.0.1

Nothing in the registry holds it, but something on the machine is bound to it. See what:

warden ports --port 8000

port 8080 is reserved

It is in WARDEN_RESERVED, or it is the registry's own port. Reserved ports are never handed out, whatever is asked.

no free port left in 8000-8999 on 127.0.0.1

The pool is genuinely full. Look for registrations nothing is using any more:

warden ls
warden pool

Old entries stay until released, which is what makes ports stick across restarts. Release the dead ones, or widen the pool:

warden serve --pool 8000-9999

For things that will not clean up after themselves, register with --ttl so the entry expires by itself.

set either preferred_port or require_port, not both

They mean opposite things. --preferred-port is a wish, --require-port is a demand.

A 422 with no obvious cause

The request body did not hold up. Common causes:

• a name with a capital letter or a space — names are lowercase letters, digits, . _ and -, starting with a letter or digit
• a misspelled field. Unknown fields are rejected rather than ignored, so prefered_port fails loudly instead of quietly giving you a different port

Ports and processes

unknown in the PROCESS column, and no PID

That socket belongs to another user. The operating system does not hand over another account's process details, which is normal rather than a fault. warden counts them:

3 of 42 belong to another user - run warden as administrator to see them

Run elevated — administrator on Windows, sudo on Linux and macOS — if you need those rows.

node.exe (25084) ignored the request to stop - pass --force to kill it

warden asked politely and waited five seconds. Some processes trap that signal.

warden kill 3000 --force

process 4 belongs to the operating system / that process is warden itself

Refused on purpose. Pids 0 to 4 belong to the system, and warden will not end its own process.

not allowed to stop X - it belongs to another user

Your account may not touch it. Run elevated, or stop it however its owner normally would.

stopping processes over the API is switched off

DELETE /v1/listeners/{pid} is off unless you ask for it:

WARDEN_ALLOW_KILL=true warden serve

It is off by default because a warden reachable from the network would otherwise let anyone holding the token end processes on that machine. warden kill on the command line is unaffected — it acts locally and never asks the API.

The fleet

The hub does not list a node

Work down this list on the node:

1. Is WARDEN_UPSTREAM set? curl localhost:7010/health — role must say edge. If it says hub, the node does not know it should report to anyone.
2. Look at the node's log. It says so on every attempt:

   could not reach the warden at http://hub:7010: All connection attempts failed
   

3. Can it reach the hub at all? curl http://hub:7010/health from the node.
4. Do the cluster tokens match? A mismatch shows as a 401 in the node's log. WARDEN_CLUSTER_TOKEN must be identical on both, and it is not the same setting as WARDEN_TOKEN.
5. Give it a moment. A node reports three times per WARDEN_NODE_TTL, so up to 30 seconds with the default of 90.

invalid or missing cluster token

The node is sending the wrong secret, or none. Announcing takes WARDEN_CLUSTER_TOKEN; the human WARDEN_TOKEN does not work in its place, and that is deliberate.

invalid or missing token

Neither token matched. Reading accepts either WARDEN_TOKEN or WARDEN_CLUSTER_TOKEN; anything that changes something takes WARDEN_TOKEN.

warden nodes --url http://hub:7010 --token "$WARDEN_TOKEN"

A node shows as stale

It has not reported within its lease. Either it is down, or it cannot reach the hub. Its row is deliberately kept rather than deleted — a server that stopped answering is worth seeing. When it is gone for good:

warden nodes --forget build-01

The hub went down. What happens to the nodes?

Nothing. They keep handing out ports, warn in their logs on each attempt, and report back on their own once it returns. That is the whole point of the arrangement: a node's own work never depends on the network.

warden ls against the hub does not show a node's services

warden ls asks one warden. Add --all to ask the whole fleet:

warden ls --all --url http://hub:7010

warden ls --all leaves a node out

It does not leave it out silently. Look at standard error:

build-01 (http://build-01:7010) could not be reached

The reason says which of these it is:

Reason	Means
could not be reached	Nothing answered at the address the node advertised. Check WARDEN_ADVERTISE on the node — the hub uses that address, not the one you typed
refused the token - check WARDEN_CLUSTER_TOKEN matches	The hub's cluster token is not the node's
did not answer within 3s	The node is reachable but too slow
answered 500	The node is up but its own registry failed

Display

The banner is drawn with # instead of blocks

The console is running a legacy code page that cannot encode the block character. warden falls back rather than aborting the command over a decoration. To get the blocks on Windows:

chcp 65001

The dashboard shows no banner

The terminal is under 30 rows tall, so the banner gives way to the table. Make the window taller.

warden nodes prints no other warden has reported in

No node has announced itself to this warden. If you expected some, work through the fleet section above.

Data

Where is the database?

System	Path
Windows	%LOCALAPPDATA%\warden\registry.db
Linux	~/.local/share/warden/registry.db
macOS	~/Library/Application Support/warden/registry.db

Override with WARDEN_DATABASE or --database. Two wardens on one machine need separate files.

Starting over

Stop warden, delete the database, start it again. Every registration and every known node goes with it; nothing else is affected.

Does an upgrade lose the registry?

No. A schema change adds what it needs when the database is next opened. A database written by an older warden keeps its registrations.

Still stuck

curl -s localhost:7010/health is the fastest single check — it tells you the version, the node name, whether this warden reports to another, and how much it is holding, without a token.

If something looks like a defect, the repository is at https://github.com/vxnsin/warden/issues.