Relay Operations

Relay Operations

A relay node is a dedicated host that runs ggscale-server relay. It runs only the TURN listener. It has no database, HTTP API, or matchmaker. The API servers issue relay credentials. The relay node checks them with the shared secret.

The relay section of .env.example lists every setting. Configuration Reference lists the main ones. P2P Connectivity and TURN Relay covers the client side.

The API servers keep a registry of relay nodes. A platform admin registers and retires nodes through the admin API. A probe reads the GET /lifecycle route of each node every 10 seconds. Credentials list only nodes that are up and active. You do not restart the API servers to add, replace, or roll back a node.

Minimal configuration

On each relay node:

RELAY_PUBLIC_IP=203.0.113.10        # IPv4 address clients reach
RELAY_SHARED_SECRET=...             # >= 32 bytes, same value as the API servers
RELAY_HEALTH_ADDR=:9091             # /healthz, /metrics, lifecycle routes
LIFECYCLE_TOKEN=...                 # >= 32 bytes, enables drain and resume
LIFECYCLE_READ_TOKEN=...            # >= 32 bytes, read-only, for the probe

On the API servers:

PLATFORM_ADMIN_TOKEN=...            # >= 32 bytes, enables /v1/admin/relay-nodes
RELAY_PROBE_TOKEN=...               # the relay nodes' LIFECYCLE_READ_TOKEN

The API servers must reach RELAY_HEALTH_ADDR, because the probe reads GET /lifecycle there. Give the relay nodes their own tokens, and give the probe only the read token. See Give relay nodes their own token. LIFECYCLE_READ_TOKEN needs v0.9.74 or later; on v0.9.73, set RELAY_PROBE_TOKEN to the relay nodes' LIFECYCLE_TOKEN.

Ports

Port	Protocol	Setting	Default
TURN	UDP	RELAY_UDP_PORT	3478
TURN over TCP	TCP	RELAY_TCP_PORT	off
TURNS (TLS)	TCP	RELAY_TLS_PORT, with RELAY_TLS_CERT_FILE and RELAY_TLS_KEY_FILE	off
Relayed media	UDP	RELAY_MIN_PORT to RELAY_MAX_PORT	OS ephemeral range
Health, metrics, lifecycle	TCP	RELAY_HEALTH_ADDR	off

Set a media port range, so that the firewall opens only that range. RELAY_MAX_ALLOCATIONS defaults to 4000. Keep it at or below the width of the range. Keep RELAY_HEALTH_ADDR on a private network, for example a tailnet. Do not put it on the public interface.

With TURNS on, the node reads the certificate and key files each minute. When their content changes, it loads the new pair, so a renewal does not need a restart. If a pair does not load, the node keeps the last good certificate, logs the error, and tries again.

Health and metrics

GET /healthz returns 200 while the process runs, also while it drains. GET /readyz returns 200 only while the node accepts new allocations. GET /lifecycle returns the state, live allocations, public IP, and transports as JSON. Process Lifecycle documents these three routes. GET /metrics serves Prometheus metrics. When RELAY_METRICS_TOKEN is set, it needs that bearer token. RELAY_METRICS_TOKEN has no _FILE form.

Metric	Alert on
up (from Prometheus)	up == 0: the scrape target is down. Do not use ggscale_relay_up for this. It is 1 while the process serves, and a stopped process sends no value.
ggscale_relay_active_allocations	Approaching RELAY_MAX_ALLOCATIONS or the media port range width.
ggscale_relay_allocations_rejected_total	Allocations refused by the node cap.
ggscale_relay_auth_failures_total	Credential probing, or a shared secret that differs from the API servers'.
ggscale_relay_alloc_throttled_total	One credential hitting the per-player allocation limit.
ggscale_relay_peer_rejected_total	Attempts to relay into private address space.
ggscale_lifecycle_state	A node left in draining after a deploy.
ggscale_lifecycle_work_refused_total	Clients still sent to a draining node.
ggscale_relay_no_eligible_nodes_total	On the API servers: credential requests refused because no node was eligible.
ggscale_relay_registry_node_eligible	On the API servers: 0 for a node that credentials do not list. Each API server reports the same registry, so aggregate with max by (node_id).
ggscale_relay_registry_node_probe_ok	On the API servers: 0 for a node that serves but failed its last probe. Aggregate the same way.

Which nodes credentials list

A node is eligible when all of these are true:

• It is registered and not retired.
• Its last probe was less than 30 seconds ago.
• That probe got a 2xx with role: "relay" and state: "active".

POST /v1/relay/credentials?region=<name> lists the TURN URLs of each eligible node in that region. The region match ignores case. The nodes come in random order. The URLs of one node keep their registered order.

If the region has no eligible node, or the client sends no region, the response lists every eligible node. Then the region field is "" if those nodes are in more than one region. If no node is eligible, the request fails with 503 and Retry-After: 10.

A node drops out after its next probe if one of these occurs:

• It stops answering.
• It answers with a status other than 2xx.
• It reports a different role.
• It reports draining.

If no probes run, a node drops out when its last good probe is 30 seconds old.

A 401 means that RELAY_PROBE_TOKEN matches neither the LIFECYCLE_READ_TOKEN nor the LIFECYCLE_TOKEN of the node. One API server in the cluster logs this one time for each node. In the admin list, last_probe_error tells why the last probe failed, for example timeout, status 503, or unauthorized.

The probe is the River job relay_node_probe. It runs on its own queue, one time across the API server cluster, so slow jobs on the default queue cannot delay it. If River does not start, each API server logs a warning. It then probes the nodes itself every 10 seconds.

A change on a node shows in credentials after about 15 seconds:

• Up to 10 seconds until the next probe.
• Up to 3 seconds for the probe timeout.
• Up to 2 seconds while each API server uses its cached list of eligible nodes.

A delay in the probe job adds to this time. To see when a change applies, watch the node in the admin list. When its eligible and last_probe_at fields show the change, wait 2 more seconds.

Admin API

The routes exist only when PLATFORM_ADMIN_TOKEN is set, and they take it as a bearer token. The API reference lists them under Platform Admin.

API=https://api.example.com
ADMIN=...                           # PLATFORM_ADMIN_TOKEN

# Register a node. It is listed once a probe sees it active.
curl -fsS -X POST -H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' \
  "$API/v1/admin/relay-nodes" -d '{
    "region": "us-east",
    "turn_urls": [
      "turn:relay-2.example.com:3478?transport=udp",
      "turns:relay-2.example.com:5349?transport=tcp"
    ],
    "lifecycle_url": "http://10.0.0.6:9091/lifecycle"
  }'

# List nodes with their last probe and whether they are eligible.
curl -fsS -H "Authorization: Bearer $ADMIN" "$API/v1/admin/relay-nodes"

# Retire a node: never listed or probed again. The row is kept.
curl -fsS -X POST -H "Authorization: Bearer $ADMIN" "$API/v1/admin/relay-nodes/7/retire"

A region contains lowercase letters, digits, and dashes. Only one registered node can use a given lifecycle_url. You can register the URL of a retired node again.

lifecycle_url can point at a loopback or private address. The server refuses these addresses:

• Link-local addresses, including 169.254.169.254.
• Cloud metadata addresses.
• Multicast and unspecified addresses.

It checks at registration, and again after DNS resolution when the probe connects. The probe ignores proxy settings and does not follow redirects.

The relay embedded in an API server

An API server runs a TURN relay inside it when all of these are set:

• FEATURE_P2P_RELAY_ENABLED=true.
• RELAY_SHARED_SECRET.
• RELAY_PUBLIC_IP, as an IPv4 address.

The server advertises it as turn:<RELAY_PUBLIC_IP>:<RELAY_UDP_PORT>?transport=udp. Clients can reach it at another port or a hostname, for example when Docker publishes the UDP port on a different host port. Then list the URIs to advertise in RELAY_EMBEDDED_URLS.

The embedded relay has no region. A response lists it only in two cases:

• The response falls back to every eligible node. This includes the case where no registered node is eligible.
• The server cannot read the registry.

A single-host install therefore needs no registration.

Nothing probes the embedded relay, and you cannot drain it. Do not register it as a node. Its GET /lifecycle reports role: "api", so the probe marks it unhealthy. To stop listing it, unset RELAY_PUBLIC_IP on that API server.

Replacing a node (blue/green)

A draining relay refuses new TURN Allocate requests. Its current allocations continue to refresh and relay data, so calls in progress continue.

NEW=http://10.0.0.6:9091            # new node's RELAY_HEALTH_ADDR
OLD=http://10.0.0.5:9091            # old node's RELAY_HEALTH_ADDR
TOKEN=...                           # LIFECYCLE_TOKEN

1. Register the new node through the admin API.
2. Wait until GET $NEW/readyz returns 200 and the admin list shows the new node with "eligible": true. Then wait 2 more seconds.
3. Drain the old node with POST $OLD/lifecycle/drain. Wait until the admin list shows it with "eligible": false and last_lifecycle_state draining. Then wait 2 more seconds. New credentials no longer list it.
4. Wait for .details.active_allocations in GET $OLD/lifecycle to reach 0. Credentials issued before the drain list the old node until they expire after RELAY_CRED_TTL. The old node refuses new allocations from them. The next credential request of the client lists the new node.
5. Retire the old node with POST /v1/admin/relay-nodes/<old id>/retire, then stop it. With nothing left to drain, SIGTERM exits at once.

curl -fsS "$NEW/readyz"
curl -fsS -X POST -H "Authorization: Bearer $TOKEN" "$OLD/lifecycle/drain"
curl -fsS -H "Authorization: Bearer $TOKEN" "$OLD/lifecycle" | jq .details.active_allocations

To roll back before step 5, run POST $OLD/lifecycle/resume and POST $NEW/lifecycle/drain. Each change shows in credentials after a probe records it. Retire the new node when its active_allocations is 0. After step 5, register the old node again instead of resuming it.

DRAIN_TIMEOUT defaults to 0s on a relay node. For the reason and the container stop timeout, see Process Lifecycle.

Upgrading from RELAY_URLS

GG Scale no longer reads RELAY_URLS. An API server with RELAY_URLS set refuses to start. To upgrade:

1. On the API servers, set PLATFORM_ADMIN_TOKEN and RELAY_PROBE_TOKEN, and remove RELAY_URLS. Give each relay node a RELAY_HEALTH_ADDR that the API servers can reach. Set its LIFECYCLE_READ_TOKEN (or, on v0.9.73, its LIFECYCLE_TOKEN) to the value of RELAY_PROBE_TOKEN.
2. Deploy. Relay credential requests get 503 until a probe sees a registered node. An API server with an embedded relay is the exception.
3. Register every relay node that was in RELAY_URLS.

RELAY_STUN_URLS is unchanged.

Rotating the shared secret

The relay accepts each configured secret. The API servers sign with RELAY_SHARED_SECRET. You can therefore rotate the secret with no downtime:

1. Set the new value as RELAY_SHARED_SECRET_NEXT on every relay node and API server.
2. Everywhere, move the new value to RELAY_SHARED_SECRET and the old value to RELAY_SHARED_SECRET_NEXT.
3. After one RELAY_CRED_TTL, clear RELAY_SHARED_SECRET_NEXT.

Credentials issued before each step stay valid until they expire.