Server Troubleshooting

Server Troubleshooting

For people running a server. Player-side problems are on Troubleshooting.

Where the logs are

Setup	Log
Linux installer	journalctl -u 5th-echelon -f (server), journalctl -u 5th-echelon-coordinator -f (coordinator), journalctl -u caddy -f
Windows, from the launcher	Host › Manage a server › Log, or server.log.json next to dedicated_server.exe
Windows, on its own	server.log.json next to dedicated_server.exe
Docker	docker logs -f 5th-echelon

sudo bash install-server.sh --status sums up a Linux install: what's running, whether it has joined its coordinator, the updater's last result, and whether its programs match the signed release (Programs).

The installer

What you see	What to do
"doesn't resolve yet", or "points at … not at this server"	Fix the A record. On Cloudflare, switch it to DNS only
"Caddy has no certificate for …"	The A record must point here, and TCP 80 and 443 must be open in your provider's firewall. Run the installer again once they are
--status says "not joined yet"	The coordinator must answer at its https:// address, and the token must be current. The line under it shows the last error
"ports already in use"	Another program holds a port (often a web server on 80). Stop it, or put the server behind a reverse proxy; --force overrides the check
"this release isn't signed"	An older release; --allow-unsigned installs it on its checksum alone
The installer refuses the domain name	A game server's name can be at most 27 characters
--status shows the server delisted, or the updater failed	See Updates in the admin UI, or cat /var/lib/5th-echelon-update/update-status.json. The previous release is kept in /opt/5th-echelon/previous/
--status says Programs: not checked yet after updating to 0.4.2	Run the installer once more (sudo bash install-server.sh): it turns the check on, and keeps settings, accounts and keys
The program check says a program doesn't match the release	The program isn't the one the release key signed. A build of your own (--binary) shows as unverified instead. Otherwise, find out what changed it, and run the installer again to put the signed release back
SSH stopped answering after harden-host.sh	Wait 10 minutes (the change undoes itself unless confirmed), or use your provider's console. After moving SSH, connect with -p and the new port

Players can't connect

1. From outside your network, check the server answers:

   curl http://play.example.com/api/info

   It should answer with the server's info, including its ports.
2. Every port open? TCP 80, 443 (with Caddy), 8000, 50051 (without Caddy), and UDP 21126–21129, in your provider's firewall and any firewall on the machine (ufw, firewalld, Windows Firewall).
3. Public address set? Without it, the server tells players to connect to 127.0.0.1. Check --public-address / FE_PUBLIC_ADDRESS, or [public] behind a proxy or with remapped ports.
4. Port 80 really reaches the server? The game always uses port 80 for its configuration. A router or ISP that blocks inbound 80 at home stops it.
5. Cloudflare: the game server's records must be DNS only. The proxy can't carry the game's UDP or its plain HTTP.
6. Ask a player to run the connection test (Settings › Network): it names the part that fails.

Players see "Unencrypted connection"

The launcher's API isn't on HTTPS. With the Linux installer, give the server a domain name and make sure Caddy got its certificate (--status). Players' sign-ins then go over HTTPS.

Joins fail between players

• Relayed players can't be joined? Check UDP 21128–21129 are open, and [nat] isn't off. The overlay's Server tab on each player's PC shows how they're reached.
• Everyone plays over one VPN (Radmin, say) and joins fail? Set trusted_subnet to the VPN's subnet, or ask players to set Internet play to LAN or VPN only.
• Docker Desktop hides players' real addresses from the server, which breaks internet play. Run the server directly, or with host networking.
• Players' reports (admin UI › Reports) and Sessions show refused sign-ins, failed joins with the game's error, and players who dropped.

Players see "Update the launcher"

The server refuses launchers older than its own release by default ([clients] minimum_version). Players' launchers update themselves; minimum_version = "off" accepts any, e.g. while a release is fresh.

The server stops or restarts

The server exits if one of its services stops, so its supervisor brings it straight back: systemd's Restart=always (the installer sets it), Docker's --restart unless-stopped, or your scheduled task. Look at the log just before the restart, and open an issue with the Running your own server form.

Failover

Problem	What to do
Not sure what the standby would do	sudo /opt/5th-echelon/coordinator standby --status: where the record points, what Cloudflare answers for the coordinator, how long this server would wait, and whether the others reach it. journalctl -u 5th-echelon-standby has what it did
Nothing takes over in a drill	Stop both the standby and the coordinator on the coordinator's machine: the standby there otherwise restarts it. Takeover waits 3 minutes for the first in line, 6 for the second
The installer stops: the name doesn't answer through Cloudflare	Proxy the coordinator's record (orange cloud) before running the installer with --standby
Servers or launchers fail to reach the coordinator after proxying it	Turn off Bot Fight Mode, I'm Under Attack and any WAF rule that challenges its name: servers and launchers can't answer a challenge. Cloudflare often blocks data centres' addresses outright ("Sorry, you have been blocked"): from each server, curl -sI https://play.example.com/v1/info should answer 200
A TLS error for the coordinator's name once it's proxied	The name is two levels deep (play.group.example.com), which Cloudflare's free certificate doesn't cover. Set the group up under a one-level name (play.example.com)
The coordinator came back on its old machine but doesn't run there	By design: the data moved on. Move it back with standby --take-over on that machine, when it's quiet

More: docs/failover.md.

The admin UI

Problem	What to do
Every admin locked out by the restrictions	sudo bash install-server.sh --admin-open-access
An admin lost their second factor	Another admin uses Reset sign-in, or sudo bash install-server.sh --reset-admin NAME
Cloudflare error 525	Caddy has no certificate for the name: install the Origin CA certificate
A TLS error before Cloudflare answers	The name is two levels deep; use a one-level name
ERR_HTTP2_PROTOCOL_ERROR right after switching the record to proxied	Your computer still has the server's own address cached: flush your DNS cache
The admin UI takes many seconds to load from far away, or Chrome shows ERR_QUIC_PROTOCOL_ERROR	Run the installer again on the coordinator's machine (0.4.2 or newer): it turns on BBR and compresses the admin UI

More: docs/deploying.md › When something's wrong and docs/operations.md › When something's wrong.