Running a Network

Running a Network

A network is a group of servers that share friends through a coordinator: a small service one person runs for the group. The community network (scbl.jdevwebb.net) is one; you can run your own for your community, completely independent of it.

Note: The community network takes no outside servers: it's run by the project's maintainers only. Run a network of your own instead; it gets everything below.

What a coordinator gives you

• Shared friends and blocks: friends on one server are friends on every server of the network.
• One name per player: each name belongs to one player across all your servers.
• A server directory: players type your network's address (e.g. play.example.com), the launcher pings every server and sets them up on the best, and its server menu switches between them in one click.
• Friends on other servers: the launcher and overlay show where friends are playing.
• Global stats and leaderboards across the network.
• Automatic updates: each signed release is rolled out to your servers, one first, then the rest.
• An admin UI: players and where they are, what's being played, load, bandwidth, matches, players' reports, alerts to Discord or Slack, and the update rollout.
• Maintenance windows: book downtime for a server or the whole network, and players' launchers and games warn them ahead of it.
• Failover (optional): a standby coordinator on each server, which takes over when the coordinator's machine goes down.

The full reference: docs/friends.md (what the coordinator knows and trusts) and docs/operations.md (updates, metrics, the admin UI).

Shapes

Shape	Machines	For
Server and coordinator together	1	The first server of a group; others can join later
Join a group	1 per server	Each further server
Coordinator on its own	1, plus the servers	A group with no "main" server

Give the group one address. Hand out the coordinator's name (play.example.com) and give each server a regional one (eu.example.com, oceania.example.com). The community network does exactly this.

Setting it up

All with the Linux installer (Hosting a Server).

The first server, with the coordinator:

sudo bash install-server.sh --yes \
  --domain eu.example.com \
  --coordinator-domain play.example.com \
  --server-name "Example Europe" --region Frankfurt

Or a coordinator on its own:

sudo bash install-server.sh --yes --coordinator-only --coordinator-domain play.example.com

Each further server needs the join token. On the coordinator's machine:

sudo bash install-server.sh --show-join-token

Copy it to the new server as a file, privately, then:

sudo bash install-server.sh --yes \
  --domain oceania.example.com \
  --coordinator https://play.example.com \
  --join-token-file token.txt \
  --server-name "Example Oceania" --region Sydney
shred -u token.txt

--status on each server shows whether it has joined. Within a minute or two it's in the directory: curl https://play.example.com/v1/servers.

Moving the coordinator to a new name later works the same way: each server runs the installer with --coordinator https://<new name> and the join token again, and keeps its membership (docs/failover.md › Moving a coordinator to a new name). Without the token it stays unjoined, and its log says join_token is empty.

The join token

Anyone with the join token can add a server to your network, and a member server can change friendships and blocks between players on it. Give it only to people you trust, over a private channel, never in a public issue or chat.

sudo bash install-server.sh --rotate-join-token   # a new one; servers that already joined keep working

To remove a server, run remove-server with its id (in the directory and in that server's server-id.txt), then rotate the token. See docs/deploying.md › The join token.

Updates

Servers in a network update themselves:

1. The coordinator checks GitHub every 10 minutes for a release signed with the release key. Unsigned releases are never rolled out.
2. One server goes first (the quietest): it must run the release and keep reporting for 10 minutes, or the rollout halts. From 0.4.5 it's never the server on the coordinator's own machine: that one (and with it the coordinator) updates last, once every other server runs the release, or after two hours at most.
3. The rest follow when they have no players on, and by two hours into the stage at the latest.
4. Each machine's updater checks the signature again, keeps the previous release, and puts it back if the new one doesn't come back healthy.

Players' launchers show when their server updates, in their own time zone. Admins can pause, pin, halt or roll back in the admin UI. A server that doesn't install updates (--no-auto-update), or still runs an old release a day after a rollout, leaves the directory. No update starts on a server during its maintenance window.

Each server checks its own programs against the release's signed checksums before it starts. The result is in the admin UI's Updates page (the Program column) and in install-server.sh --status, and a program that doesn't match the signed release raises an alert. It never stops a server starting.

Note: After each server has updated to 0.4.2, run the installer on it once more (sudo bash install-server.sh; it keeps settings, accounts and keys). The update only replaces the programs; the installer turns on the program check, larger uploads for players' reports (8 MB, 3 minutes), BBR for TCP and the admin UI's compression, and backups if /etc/5th-echelon/backup.env exists.

Backups and failover

Back up the databases off the machine: write /etc/5th-echelon/backup.env with the bucket keys and run the installer again. From 0.4.6 each database goes to Cloudflare R2 as a snapshot every 2 hours (unchanged ones skipped, a week of them kept), with a daily archive to Backblaze B2. Before, Litestream streamed every change, which ran past R2's free requests. On a server set up before, sudo bash install-server.sh --backups-only switches it. Backups need 0.4.2 or newer, whose databases use SQLite's WAL mode; on an older release, run the installer again once the server has updated. Guide: docs/backups.md.

Failover keeps the network going when the coordinator's machine is down. Each game server runs a standby coordinator; when the coordinator has been unreachable for 3 minutes, the next server in line checks the others can't reach it either, restores it from its latest snapshot (up to 2 hours old; run backup.sh snapshot on the coordinator's machine before a planned takeover), and takes over its name. Matches and sign-ins on each server carry on throughout.

It needs:

• every server on 0.4.2 or newer, with the API over HTTPS;
• backups (above) on every server, to the same R2 bucket;
• the coordinator's name proxied through Cloudflare, with an Origin certificate on every server, and a Cloudflare API token that can edit the zone's DNS (/etc/5th-echelon/standby.env);
• the coordinator's name one level below your domain (play.example.com, not play.group.example.com): Cloudflare's free certificate covers only *.example.com, and a deeper name fails with a TLS error for every launcher and server;
• nothing in Cloudflare that blocks or challenges the servers: Bot Fight Mode off, no I'm Under Attack, WAF rule or security level on the coordinator's name. Servers run in data centres, which Cloudflare often blocks outright ("Sorry, you have been blocked"). From each server, curl -sI https://play.example.com/v1/info should answer 200;
• --standby NAME=HOST,... on each server, listing the group in the order they take over.

sudo /opt/5th-echelon/coordinator standby --status      # where the coordinator runs, and who'd take over
sudo /opt/5th-echelon/coordinator standby --take-over   # move the coordinator to this server

A replaced server stands down when it's back; moving the coordinator home is your choice. Guide, with a drill to test it: docs/failover.md.

The admin UI

In the launcher's look: an icon rail on the left, and each page opens with its key numbers and actions. Bandwidth and Activity are tabs under Network, the audit log is under Security. Light or dark: the button at the foot of the rail switches between the system's choice, light and dark. Live shows every game open now on every server: the map and mode, public or private, the host, and everyone in it with their ping and whether the relay carries them.

Turned on with --metrics-domain, served only through Cloudflare, with an Origin CA certificate:

sudo bash install-server.sh --metrics-domain admin.example.com \
  --metrics-cert origin.pem --metrics-key origin.key
sudo bash install-server.sh --add-admin yourname    # prints a one-time setup link

• Use a name one level below your domain (admin.example.com, not admin.play.example.com): Cloudflare's free certificate doesn't cover deeper names.
• Sign-in takes a password and a passkey or authenticator app (or a passkey alone), with recovery codes. Sessions end after 30 minutes idle.
• Restrictions by address range and country, under Security. Locked yourself out? sudo bash install-server.sh --admin-open-access.
• Lost a second factor? sudo bash install-server.sh --reset-admin NAME gives a new setup link.
• Your own account: change your password, make new recovery codes, add passkeys, and end your other sessions.

Step by step: docs/operations.md › The admin UI.

Maintenance windows

On the Updates page, book a time a server, several, or the whole network (the coordinator itself) will be down: the start in your own time (with UTC beside it), how long, and a note for players, e.g. "Moving to a faster machine". A preview shows what players will read. Booking only warns players; the work is yours to do.

• Launchers show it under Play on the day, or from 12 hours before, in each player's own time, and tag the server in the server menu. From 30 minutes before they offer a server that stays up, and while the server is down, Play waits.
• The game's overlay warns 10 minutes before, with the note.
• No update starts on a server during its window, or the network's.

Cancelling a window takes it off launchers within a minute. More: docs/operations.md › Maintenance windows.

Players' reports

Reports players send from their launcher (Settings › Feedback) arrive in the admin UI's Reports, where you can reply (the player reads it in their launcher), with their logs (redacted on their PC) and the server's own log lines about that player, and can be posted to your alert chat. Sessions shows each player's time on the servers and what went wrong (refused sign-ins, failed joins, dropped matches, and reports the server refused, with why). Each joined room shows both players' ping to the server and whether the relay carried their traffic; Relayed round trip gives what the relay costs relayed players, over the period. Each stay says how it ended: left, removed by the host, the connection dropped, the game restarted or crashed, or timed out. Players' games also send their log when something goes wrong: Reports › Game logs, linked from their Sessions row.

From 0.4.6:

• Beta logs: a player with a beta feature on sends the game's whole log each time the game closes. These arrive under Reports › Automatic, tagged beta and the beta's name, up to 12 a day a player, apart from the player's own reports.
• The game's sessions: a game's Sessions row shows each change of its match and lobby sessions (Online sessions: match open, lobby locked, in a match). These are there to find what marks a Spies vs Mercs round under way.

Two things in 0.4.5 make these readings truer: a game that quits normally tells the server, so its connection going isn't counted as a drop or a restart and no log is asked for; and a room closing as a player joins it counts as one failed join, not two. A report's server lines leave out the packet traces, so they reach back to when the problem happened.

Which client builds your servers take

From 0.4.5 the launcher and the game add-on send a fingerprint of their own file (its SHA-256) when they sign in, and the coordinator knows the fingerprints of the project's signed releases. Security › Client builds sets what happens to anything else (a modified or development build, or a client older than 0.4.5):

• Report (the default): it's let in and noted on Sessions as Not one of the project's releases.
• Enforce: it's refused, except for identities you allow (developers: Allow other builds on their player page).

Check Sessions before enforcing, and enforce only once your servers refuse clients older than 0.4.5. A client can be made to send a release's fingerprint, so this stops honest modified builds and casual tampering, not a determined attacker. Whatever the setting, only the game's own sign-in lets the game in (a current launcher's no longer vouches for an outdated or refused game).

Support (the community network's coordinator only): each player's conversation with the admins, with their accounts and the logs they sent. Answer there and mark it waiting or resolved; Message on a player's page writes to them first.

Players' names

Every new name and rename is checked against the name rules (no slurs or hate, no web addresses; swearing is allowed). When a server starts, it checks the names it already has: a player whose name breaks the rules is asked to choose another, and the game isn't let in under that name until they do. The server's log lists them (player name(s) break the rules; asked to choose another), and Sessions shows their refused sign-ins (its name breaks the rules, and it hasn't chosen another).

On a player's page:

• Ask to rename flags a name the word list missed: the player's launcher and overlay ask them to choose another, and the game is kept out until they do.
• Allow name is for a name the rules flagged by mistake, usually after the player asks for a review in Support: the flag goes, and the name is kept as checked, so the check on start leaves it alone.
• Rename gives the player a name you choose. The word list doesn't apply to an admin's rename (the other rules do), so it's also how to grant a name the rules refused when the player tried it.

Privacy

The admin UI shows counts and cities, never players' addresses. Players' launchers report their ping to each server, from which only the city is noted. Player activity sent to the coordinator uses anonymised ids. The details: docs/operations.md › Privacy.

Listing your network

Public servers and networks can be listed in docs/community-servers.md with a pull request: one row, with an address players can type.