Troubleshooting

Troubleshooting

Every message below is one warden actually prints. Find yours and read across.

Before any of it, though:

warden doctor

It checks the things this page describes one at a time — whether a warden is answering, where its settings came from, whether it is open without a token, how much pool is left, whether registrations are held by processes that are gone, whether the nodes are reporting — and names the ones that are wrong.

Starting up

no warden reachable at http://127.0.0.1:7010 - start one with .warden serve.

Nothing is listening there. Either it is not running, or it is running somewhere else.

Note that warden ports, warden kill and the dashboard's ports view do not need one at all — if those work and nothing else does, the registry simply is not running.

warden ports --port 7010     # is anything on it at all?
curl localhost:7010/health   # is it answering?

If your warden listens elsewhere, point the client at it:

warden ls --url http://hub:7010
# or, once
export WARDEN_URL=http://hub:7010

Note that warden ports and warden kill never need a server. If those work and everything else does not, the registry simply is not running.

[Errno 10048] error while attempting to bind on address

Something already holds port 7010. Find it and decide:

warden ports --port 7010
warden serve --port 7011      # or move out of its way

pool_start must not be greater than pool_end

--pool 8999-8000 the wrong way round, or a WARDEN_POOL_START above WARDEN_POOL_END.

the warden at http://hub:7010 cannot reach this one at http://127.0.0.1:7020

A node pointing at a hub on another machine while advertising a loopback address. The hub would record an address it can never open, so warden refuses at startup rather than letting it fail silently weeks later.

WARDEN_ADVERTISE=http://build-01:7010    # a name or address the hub can use

Two wardens on the same machine may both use loopback; the check only applies when the hub is elsewhere.

Registering

port 8000 is held by 'shop-api'

You asked for that exact port with --require-port and another registration has it. Either take what you are given:

warden register shop-web --kind frontend --preferred-port 8000

or free it first, if the holder is finished with it:

warden release shop-api

port 8000 is already in use on 127.0.0.1

Nothing in the registry holds it, but something on the machine is bound to it. See what:

warden ports --port 8000

port 8080 is reserved

It is in WARDEN_RESERVED, or it is the registry's own port. Reserved ports are never handed out, whatever is asked.

no free port left in 8000-8999 on 127.0.0.1

The pool is genuinely full. Look for registrations nothing is using any more:

warden ls
warden pool

Old entries stay until released, which is what makes ports stick across restarts. Release the dead ones, or widen the pool:

warden serve --pool 8000-9999

For things that will not clean up after themselves, register with --ttl so the entry expires by itself.

set either preferred_port or require_port, not both

They mean opposite things. --preferred-port is a wish, --require-port is a demand.

A 422 with no obvious cause

The request body did not hold up. Common causes:

• a name with a capital letter or a space — names are lowercase letters, digits, . _ and -, starting with a letter or digit
• a misspelled field. Unknown fields are rejected rather than ignored, so prefered_port fails loudly instead of quietly giving you a different port

Ports and processes

unknown in the PROCESS column, and no PID

That socket belongs to another user. The operating system does not hand over another account's process details, which is normal rather than a fault. warden counts them:

3 of 42 belong to another user - run warden as administrator to see them

Run elevated — administrator on Windows, sudo on Linux and macOS — if you need those rows.

node.exe (25084) ignored the request to stop - pass --force to kill it

warden asked politely and waited five seconds. Some processes trap that signal.

warden kill 3000 --force

process 4 belongs to the operating system / that process is warden itself

Refused on purpose. Pids 0 to 4 belong to the system, and warden will not end its own process.

not allowed to stop X - it belongs to another user

Your account may not touch it. Run elevated, or stop it however its owner normally would.

stopping processes over the API is switched off

DELETE /v1/listeners/{pid} is off unless you ask for it:

WARDEN_ALLOW_KILL=true warden serve

It is off by default because a warden reachable from the network would otherwise let anyone holding the token end processes on that machine. warden kill on the command line is unaffected — it acts locally and never asks the API.

The fleet

The hub does not list a node

Work down this list on the node:

1. Is WARDEN_UPSTREAM set? curl localhost:7010/health — role must say edge. If it says hub, the node does not know it should report to anyone.
2. Look at the node's log. It says so on every attempt:

   could not reach the warden at http://hub:7010: All connection attempts failed
   

3. Can it reach the hub at all? curl http://hub:7010/health from the node.
4. Do the cluster tokens match? A mismatch shows as a 401 in the node's log. WARDEN_CLUSTER_TOKEN must be identical on both, and it is not the same setting as WARDEN_TOKEN.
5. Give it a moment. A node reports three times per WARDEN_NODE_TTL, so up to 30 seconds with the default of 90.

invalid or missing cluster token

The node is sending the wrong secret, or none. Announcing takes WARDEN_CLUSTER_TOKEN; the human WARDEN_TOKEN does not work in its place, and that is deliberate.

invalid or missing token

Neither token matched. Reading accepts either WARDEN_TOKEN or WARDEN_CLUSTER_TOKEN; anything that changes something takes WARDEN_TOKEN.

warden nodes --url http://hub:7010 --token "$WARDEN_TOKEN"

A node shows as stale

It has not reported within its lease. Either it is down, or it cannot reach the hub. Its row is deliberately kept rather than deleted — a server that stopped answering is worth seeing. When it is gone for good:

warden nodes --forget build-01

The hub went down. What happens to the nodes?

Nothing. They keep handing out ports, warn in their logs on each attempt, and report back on their own once it returns. That is the whole point of the arrangement: a node's own work never depends on the network.

warden ls against the hub does not show a node's services

warden ls asks one warden. Add --all to ask the whole fleet:

warden ls --all --url http://hub:7010

two nodes hold shop-api

A service name is unique per node, never across the fleet, so two machines can both answer to it. Nearly always two projects that drifted apart. Rename one, or leave it if both are meant to exist — nothing breaks either way, the fleet view just says so instead of hiding it.

no node registered as .build-01.

The --node you named has never announced itself to the warden you asked. warden nodes lists the ones it knows; check WARDEN_NODE on the machine you meant, since that is the name it reports under.

--node gives invalid or missing API token

Forwarding a change takes WARDEN_TOKEN, not the cluster token, and the token you send has to be one the target node accepts as well — the hub carries your authorization on rather than vouching for you with its own.

warden ls --all leaves a node out

It does not leave it out silently. Look at standard error:

build-01 (http://build-01:7010) could not be reached

The reason says which of these it is:

Reason	Means
could not be reached	Nothing answered at the address the node advertised. Check WARDEN_ADVERTISE on the node — the hub uses that address, not the one you typed
refused the token - check WARDEN_CLUSTER_TOKEN matches	The hub's cluster token is not the node's
did not answer within 3s	The node is reachable but too slow
answered 500	The node is up but its own registry failed

Display

The banner is drawn with # instead of blocks

The console is running a legacy code page that cannot encode the block character. warden falls back rather than aborting the command over a decoration. To get the blocks on Windows:

chcp 65001

The dashboard shows no banner

The terminal is under 30 rows tall, so the banner gives way to the table. Make the window taller.

warden nodes prints no other warden has reported in

No node has announced itself to this warden. If you expected some, work through the fleet section above.

Data

Where is the database?

System	Path
Windows	%LOCALAPPDATA%\warden\registry.db
Linux	~/.local/share/warden/registry.db
macOS	~/Library/Application Support/warden/registry.db

Override with WARDEN_DATABASE or --database. Two wardens on one machine need separate files.

Starting over

Stop warden, delete the database, start it again. Every registration and every known node goes with it; nothing else is affected.

Does an upgrade lose the registry?

No. A schema change adds what it needs when the database is next opened. A database written by an older warden keeps its registrations.

Still stuck

curl -s localhost:7010/health is the fastest single check — it tells you the version, the node name, whether this warden reports to another, and how much it is holding, without a token.

If something looks like a defect, the repository is at https://github.com/vxnsin/warden/issues.

Updates

warden update says nothing about a new version

0.1.0 is the newest there is

You are on it. The check compares against GitHub releases, not tags or commits. If it has no answer at all it says why instead — the machine is offline, or GitHub rate limited it.

updating over the API is switched off

The machine being asked has not been told it may update itself:

WARDEN_ALLOW_REMOTE_UPDATE=true

Off by default on purpose. See Updates.

this warden has no WARDEN_UPDATE_COMMAND

It is allowed to update but does not know how. The command lives on that machine, never in the request:

WARDEN_UPDATE_COMMAND="/usr/local/bin/update-warden.sh"

cannot run the update command

The program named is not on that machine, or not on its PATH. Use a full path. The command runs without a shell, so cd x && y will not work — put it in a script.

the update command exited 1: ...

Your command failed and warden passed its output back. Read that output: usually the machine cannot reach wherever it pulls from, or the working tree is dirty.

the update command ran longer than 300s

Given up on. Make the command finish faster, or have it start the real work in the background and return.

One node updated, another says could not be reached

Exactly as designed: every node is asked, every answer reported, and one machine being away neither stops the others nor fails the command.

warden updated but is still running the old version

Your update command has to restart the service; warden does not restart itself, because it cannot report the result of a command that ends the process running it. End the script with systemctl --user restart warden or the equivalent.

build-01 is already at http://build-01:7010 and now claims ...

A node's name is pinned to the address it first announced. This is a second announcement asking for a different one.

If the machine really did move:

warden nodes --forget build-01

Then let it announce itself again. If it did not move, something else is using that name — worth finding out what, since the hub sends a person's token to whatever address a node gives it.

... is plain HTTP and this warden requires HTTPS

WARDEN_REQUIRE_HTTPS is on and the node is at an http:// address. Either put the node behind HTTPS and let it re-announce, or turn the requirement off while the fleet is on a network you trust.

sending a token to http://... over plain HTTP in the log

Not an error. The hub is forwarding a person's token to a node over an unencrypted connection, which is fine on a trusted network and worth fixing before it is not one. Said once per node, not once per request.