Operator Tokens

Operator Tokens

Operator tokens are bearer tokens for the operations routes. They are not Game Project API keys or player sessions. Make each token with openssl rand -hex 32. Each token except RELAY_METRICS_TOKEN also has a <NAME>_FILE form for a mounted secret file.

Token	Set on	Protects	Held by	If it leaks
LIFECYCLE_TOKEN	API servers and relay nodes	The /lifecycle routes of that process. See Process Lifecycle.	Deploy tooling	The holder can drain the process or put it back into service.
LIFECYCLE_READ_TOKEN	API servers and relay nodes, optional	GET /lifecycle of that process only. It never allows drain or resume. Needs v0.9.74 or later.	Probes and monitors. On relay nodes, the API servers hold it as RELAY_PROBE_TOKEN.	The holder can read the process state: lifecycle state, commit, work in progress, and public IP.
RELAY_PROBE_TOKEN	API servers	Nothing on the API server. The relay probe sends it to GET /lifecycle on each registered node. Set it to the relay nodes' LIFECYCLE_READ_TOKEN. The relay nodes' LIFECYCLE_TOKEN also works, but then every API server holds a token that can drain every relay. When it is unset, the probe sends no header. That works only for nodes with neither token.	API servers	With the read token, the holder can read relay state. With the full token, the holder can drain or resume relay nodes.
PLATFORM_ADMIN_TOKEN	API servers	/v1/admin/*, which registers, lists, and retires relay nodes. See Relay Operations.	Operators who deploy relay nodes	The holder can change the relays that clients use. The holder can make the API servers probe loopback or private addresses. If the holder registers a URL that they control, they also get RELAY_PROBE_TOKEN.
METRICS_AUTH_TOKEN	API servers	/metrics. Production requires it unless METRICS_AUTH_DISABLED=true.	Prometheus	The holder can read operational metrics.
RELAY_METRICS_TOKEN	Relay nodes	/metrics on the relay. Optional. When it is unset, the route is open.	Prometheus	The holder can read relay metrics.
ENTITLEMENT_API_TOKEN	API servers, optional	/internal/entitlements, which exists only with ENTITLEMENT_API_ENABLED=true. When it is unset, the server generates one and stores it in the database.	The billing service	The holder can change the tier class and features of any Account Tenant. A generated token is also in database backups.

You never set the control panel bootstrap token. The server generates it on a new install for the first sign-up. RELAY_SHARED_SECRET is not a bearer token. It is a signing key for TURN credentials that the API servers and relay nodes share.

Give relay nodes their own token

The probe sends RELAY_PROBE_TOKEN to each lifecycle_url that a platform admin registers. It often uses plain http on a private network. Assume that relay hosts and platform admins can get this token. So give the probe a token that cannot drain.

Do these steps:

1. On the relay nodes, set LIFECYCLE_READ_TOKEN (for the probe) and LIFECYCLE_TOKEN (for drain and resume). Use two different values, and do not reuse the tokens of the API servers.
2. Set RELAY_PROBE_TOKEN on the API servers to the relay nodes' LIFECYCLE_READ_TOKEN. Keep the relay nodes' LIFECYCLE_TOKEN in your deploy tooling only.
3. Keep RELAY_HEALTH_ADDR on a private network, or put it behind TLS. Then the token does not cross a network that you do not trust.

If the read token leaks, the holder can read relay state but cannot drain relays. An API server logs a warning at startup when RELAY_PROBE_TOKEN is equal to its own LIFECYCLE_TOKEN.

On v0.9.73 and earlier there is no read token. There, RELAY_PROBE_TOKEN must equal the relay nodes' LIFECYCLE_TOKEN, and a leak lets the holder drain relays.

Move the probe to the read token

GET /lifecycle accepts either relay token, so this change does not unlist any node:

1. Set LIFECYCLE_READ_TOKEN on each relay node and restart it.
2. Change RELAY_PROBE_TOKEN on the API servers to that value.
3. Check that every node is still eligible in GET /v1/admin/relay-nodes.

Rotate the read token

If RELAY_PROBE_TOKEN matches neither token of a node, the probes of that node get 401. Credentials then stop listing the node. One API server in the cluster logs the mismatch one time for each node. A node accepts one read token at a time, so a rotation unlists each node from its restart until the API servers have the new value. To avoid that gap, point RELAY_PROBE_TOKEN at the new value only after every relay node has it, and do the two changes close together.

What a typical install sets

• Single host with the relay embedded in the API server: METRICS_AUTH_TOKEN in production. Add LIFECYCLE_TOKEN if you drain the server for deploys.
• Dedicated relay nodes: on the API servers, set PLATFORM_ADMIN_TOKEN, their own LIFECYCLE_TOKEN, and RELAY_PROBE_TOKEN with the relay nodes' read token. On the relay nodes, set a different LIFECYCLE_TOKEN and a LIFECYCLE_READ_TOKEN. Add the metrics tokens that your scraper uses.