Process Lifecycle

Process Lifecycle

The API server (ggscale-server) and the relay node (ggscale-server relay) use the same lifecycle routes, JSON, environment variables, and token. A deploy tool or load balancer can take a process out of service and wait until it is empty. It can then put the process back without a restart.

Process	Routes are served on	Alongside
API server	The main HTTP_ADDR listener, at the root (no /v1 prefix)	/metrics
Relay node	RELAY_HEALTH_ADDR (off when unset)	/healthz, /metrics

Public edge

On the API server, the lifecycle routes use the same listener as the public API. They are not in the API reference. Apply this rule at your reverse proxy or load balancer:

Path	On the public edge
/v1/*	Public.
/readyz, /lifecycle, /lifecycle/*	Private. Only the load balancer and deploy tools reach them.
/internal/*	Private. The entitlement API uses this path.
/metrics	Public only when METRICS_AUTH_TOKEN is set. Otherwise private.
/mcp	Public when FEATURE_MCP_ENABLED=true, because coding agents connect to it.

On a relay node, keep RELAY_HEALTH_ADDR on a private network.

States

starting ──▶ active ◀──▶ draining ──▶ stopping

State	Meaning
starting	Booting; not ready for traffic.
active	Accepts new work.
draining	Refuses new work while work in flight continues. POST /lifecycle/drain or SIGTERM starts it.
stopping	The SIGTERM drain wait has ended and the process is shutting down.

Each process keeps its state in memory only. A restarted process starts in starting. It becomes active when its listeners are up, even if it was draining before the restart.

Routes

Route	Auth	Behavior
GET /readyz	Open	200 in active, 503 otherwise. The body is the state name.
GET /lifecycle	LIFECYCLE_TOKEN or LIFECYCLE_READ_TOKEN when either is set, else open	The JSON below.
POST /lifecycle/drain	LIFECYCLE_TOKEN, required	Moves to draining. A repeat returns 200 and changes nothing.
POST /lifecycle/resume	LIFECYCLE_TOKEN, required	Moves back to active. Idempotent. Returns 409 once SIGTERM has arrived, including during the drain wait.
GET /healthz (relay), GET /v1/healthz (API)	Open	Liveness: 200 in every state while the process runs.

Both POST routes return the same JSON as GET /lifecycle.

With LIFECYCLE_TOKEN set, send Authorization: Bearer <token>. A missing or wrong token gets 401. Without the token, GET /lifecycle is open and the POST routes return 404, so nobody can drain a process that has no token. The token must be 32 bytes or more. It supports LIFECYCLE_TOKEN_FILE.

LIFECYCLE_READ_TOKEN is an optional second token for probes and monitors. It allows GET /lifecycle and nothing else, so its holder cannot drain or resume the process. It must be 32 bytes or more and different from LIFECYCLE_TOKEN, and it supports LIFECYCLE_READ_TOKEN_FILE. When it is set, GET /lifecycle needs one of the two tokens, even if LIFECYCLE_TOKEN is empty. It needs v0.9.74 or later. See Operator Tokens.

{
  "role": "relay",
  "state": "draining",
  "commit": "abc1234",
  "started_at": "2026-10-26T10:00:00Z",
  "drain_started_at": "2026-10-26T10:30:00Z",
  "active_work": 12,
  "details": {
    "active_allocations": 12,
    "public_ip": "203.0.113.10",
    "transports": ["udp", "tcp"]
  }
}

role is relay or api. Times are UTC. drain_started_at is null when the process is not draining. A drain is complete when active_work reaches 0. details holds the role-specific counts in the next table.

What counts as work

	Relay node	API server
Draining refuses	New TURN Allocate requests.	New WebSocket upgrades on /v1/ws, with 503 and Retry-After.
Keeps working	Existing allocations: Refresh, CreatePermission, ChannelBind, and relayed data.	Open WebSockets. Plain HTTP requests also continue. The load balancer stops sending them when /readyz returns 503.
active_work	Each admitted Allocate, from admission until its allocation closes.	Realtime WebSockets from admission until cleanup finishes, plus in-flight HTTP requests other than the lifecycle routes.
details	active_allocations, public_ip, transports	realtime_connections, inflight_requests

On /v1/ws, the API server checks for a drain before authentication. A browser client that it refuses keeps its one-time ticket and can use it on another node.

Draining does not pause River jobs or the matchmaker worker. A River queue pause applies to the full cluster. Both stop cleanly during normal shutdown.

A drain of the API server does not drain the relay embedded in it. That relay runs when FEATURE_P2P_RELAY_ENABLED=true, RELAY_SHARED_SECRET is set, and RELAY_PUBLIC_IP is an IPv4 address. You cannot register it as a relay node. To drain relays, run dedicated relay nodes. See Relay Operations.

SIGTERM

1. The process enters draining: /readyz returns 503 and new work is refused.
2. It waits until active_work reaches 0 or DRAIN_TIMEOUT elapses. With nothing in flight, the wait ends immediately.
3. It enters stopping, runs its shutdown path, and exits 0.

Between steps 2 and 3, the API server closes the remaining WebSockets with code 1001 Going Away. Clients then connect to another node. The relay closes its remaining TURN allocations when its TURN server shuts down.

A second SIGTERM or SIGINT during the wait kills the process at once. Open sockets get no 1001 and deferred cleanup does not run.

DRAIN_TIMEOUT is a Go duration, for example 30s or 2m. The default is 30s on the API server and 0s on a relay node. The relay default is 0s for two reasons. You drain a relay with POST /lifecycle/drain before you stop it. Also, TURN allocations usually last longer than a short wait. An undrained relay that gets SIGTERM closes its allocations and exits at once.

Container stop timeout

Set the container stop timeout above DRAIN_TIMEOUT plus the shutdown steps. If it is shorter, the runtime kills the process during the drain or before cleanup. A relay node with DRAIN_TIMEOUT=0s stops within the Docker default of 10s. The API server needs more time. After the drain wait, it runs these steps in sequence.

Step	Bound
Close remaining realtime sockets	5s
http.Server.Shutdown	10s
Matchmaker worker	30s
River graceful stop	10s
Realtime connection-cap close	5s
Rate-limit store close	10s

The total is 70s after the drain, or 100s with the default DRAIN_TIMEOUT. The database pool and later closers have no time limit, so allow about 120s:

# docker compose
services:
  server:
    stop_grace_period: 120s

# Kubernetes pod spec
terminationGracePeriodSeconds: 120

Metrics

Both processes add these to their /metrics:

Metric	Type	Meaning
ggscale_lifecycle_state{state="..."}	gauge	1 for the current state, 0 for the others.
ggscale_lifecycle_work_refused_total	counter	New work refused while draining: Allocate requests on the relay, WebSocket upgrades on the API server.

Rolling a node

TOKEN=...                       # LIFECYCLE_TOKEN
NODE=http://10.0.0.5:9091       # relay: RELAY_HEALTH_ADDR. API: the main listener.

# 1. Drain. /readyz turns 503 and the load balancer stops routing here.
curl -fsS -X POST -H "Authorization: Bearer $TOKEN" "$NODE/lifecycle/drain"

# 2. Wait until active_work reaches 0, or give up after your own deadline.
curl -fsS -H "Authorization: Bearer $TOKEN" "$NODE/lifecycle" | jq .active_work

# 3a. Replace the node: SIGTERM finishes the drain and exits 0.
# 3b. Or put it back into service without a restart:
curl -fsS -X POST -H "Authorization: Bearer $TOKEN" "$NODE/lifecycle/resume"

For relay nodes, follow the blue/green sequence instead. That sequence also removes the node from the relay node registry. New credentials then stop listing it.