architecture

Architecture

ggscale is a multiplayer game backend that runs as a single Go service against one Postgres database. It needs no message broker or separate cache tier, and nothing runs as a sidecar. The service and its database are the whole system, which keeps a self-hosted deployment small enough for a single VPS. The same service scales out to a managed multi-host, multi-region deployment with no code change.

This page covers the core technical design and the reasoning behind the major decisions. For the API nouns (tenant, project, player, API key), read Core Concepts. For every environment variable, read Configuration Reference.

One service with flexible configuration

The service dispatches on its first argument to one of three commands:

• server (the default) runs the HTTP and WebSocket API, the control panel, the player website, the matchmaker, and any enabled optional subsystems.
• migrate inspects or repairs migration state (version, force <n>) without booting the server.
• relay runs a standalone TURN node for a dedicated relay VM. It holds no database and serves no application API, only the TURN listener and an optional health and metrics endpoint.

Server startup is a single flat function. It loads config, applies migrations, opens the database pools, constructs every subsystem in dependency order, and passes them into one router. There is no dependency-injection framework, so the whole object graph is visible in one file and reads top to bottom.

Optional subsystems are gated by config. The fleet manager, the relay credential issuer and its UDP listener, the server browser, the entitlement API, the billing handoff, the control panel, and the read replica are each built only when their flag or URL is set. When a subsystem is off, its value stays nil and its routes never mount, so the running surface matches the configuration exactly. The matchmaker is the exception. Its queue and worker always run, because a match-only ticket needs no fleet and no extra infrastructure; access to matchmaking is gated per API key by a scope, not by a startup switch.

Postgres as the source of truth

Every piece of shared correctness lives in Postgres. Caches and request rate limiters are deliberately process-local, held in memory on each instance. An earlier design used a distributed cache (Olric); it was removed, because a per-node cache with a short time-to-live converges fast enough for rate limits and feature-grant lookups, and dropping it removes an operational dependency and a failure mode.

One limiter genuinely needs cross-instance agreement: the per-tenant cap on concurrent realtime sockets. That one is a Postgres-backed lease keyed by deployment region. It admits sockets from process memory and touches the database only to lease or renew a block of the tenant's capacity. Isolation, capacity accounting, and quotas stay in Postgres. Anything where a few seconds of staleness is harmless stays in local memory.

Tenant isolation and the database trust model

Isolation is enforced in two independent layers, so a bug in either one still blocks cross-tenant access.

The first layer is application middleware. Each /v1 request carries a bearer API key. The tenant middleware hashes the key, resolves it to a tenant (and optionally a project), and puts those ids in the request context. Handlers read the tenant from context; they never trust a tenant id sent by the caller.

The second layer is Postgres Row-Level Security. Before running a handler's queries, the pool opens a transaction and sets app.tenant_id as a transaction-local variable. RLS policies on the 21 tenant-scoped tables filter every row against that setting, so a query that forgets its WHERE tenant_id = ... clause still returns only the caller's rows.

What makes RLS trustworthy is the database role, because a table owner is exempt from RLS. The application must never connect as an owner, and two roles keep this clean:

• An elevated migration role holds the rights to run DDL, create roles, and define policies. It is used once at startup and never on the request path. In production it comes from a separate DB_MIGRATE_URL.
• A least-privilege login owns nothing. Every pooled connection runs SET ROLE ggscale_app on connect, and a boot-time check confirms the effective role is the non-owning ggscale_app and, in production, that the session cannot reset back to an owner.

Fail-closed RLS means privileged and pre-tenant work needs explicit carve-outs, and the schema states them plainly:

• Bootstrap policies let the tenant middleware read an API key or tenant row before any tenant is known. They allow the read only while app.tenant_id is unset.
• Worker policies let the matchmaker background worker act across tenants. It runs with no tenant context, and its policies apply only when app.tenant_id is null.
• A control-panel policy lets an operator read the tenants they hold a membership in, keyed by their dashboard user id.
• A bootstrap check allows a new tenant row to be created when an explicit allow_tenant_bootstrap flag is set for that statement.

The wire API

The JSON API under /v1 is defined as typed operations with Huma on a chi router. openapi.yaml is generated from those operations in process, with no external analyzer, so it cannot be hand-edited or drift from the handlers. The Go and C# SDKs are generated from that spec, so the wire is a frozen contract. Error responses use application/problem+json with a stable shape.

A few surfaces sit outside the /v1 document on purpose. /metrics is versionless and, in production, sits behind a bearer token. An internal entitlement API lives under /internal so it never enters the spec or the SDKs and can be firewalled to a private network. The WebSocket route and the deliberately opaque session-verify endpoint are patched into the spec by hand, because neither fits huma's request and response model.

Authorization is layered, and each /v1 route sits under only the checks it needs:

API key (Bearer)     → which tenant / project is calling
  key type           → publishable (client) vs secret (server)
  key scope          → matchmaker, fleet, p2p_relay, ...
  RBAC permission    → e.g. submit a score, verify a session
  player session     → which player, via X-Session-Token (JWT)

Password sign-in keeps a fixed per-IP limit, because bcrypt costs the server real CPU per attempt no matter what tier a tenant is on. Score submission requires a secret key, so a publishable key shipped inside a game binary cannot write to a leaderboard.

Identity and authorization

ggscale has three kinds of identity. A player is scoped to one project and signs in through the auth endpoints. A global player account links a person's identities across several of your games. A control-panel user is a human operator with a role on a tenant. See Authentication and Players for the player model.

Player sessions are short-lived JWTs signed with an HMAC key. Each live request re-checks the token against a stored session epoch, so a revoked session stops working on its next request even while the JWT itself has not expired. Banning a player or disabling a tenant takes hold the same way.

Authorization above tenant isolation uses Casbin, with its policy stored in Postgres and reloaded on a short interval. Instances pick up a policy change within that interval, so there is no push channel to run. This is the same process-local approach used elsewhere.

Paid features require three independent conditions at once:

• an operator kill switch set in the environment,
• an entitlement grant on the tenant, and
• a scope on the API key.

The layers separate the operator's control, the tenant's billing entitlement, and each key's least privilege, so one leaked key cannot draw paid infrastructure on its own. Player-resource permissions encode the project id, so a grant authorizes only its own project.

Matchmaking and peer-to-peer play

The matchmaker treats Postgres as its queue. Players post tickets; a background worker claims a batch of the oldest tickets with a lease and a claim id while the rows stay in the queued state. The worker forms groups in memory, resolves each group's mode, then flips the whole group to matched in one transaction. That commit is all-or-none, so a player who cancels mid-claim can never land in a delivered roster, and the survivors go back to the queue and rematch.

Crash recovery falls out of that design. Because rows stay queued until commit, a worker that dies mid-flight strands nothing; its lease expires and a sweeper reclaims the tickets. The worker wakes on a post-commit notification with a fallback ticker, so a lost notification only adds a little latency.

Two of the three result modes hand back a peer-to-peer roster with a designated host, the group's longest-waiting player, so peers connect directly and ggscale stays out of the game's data path. NAT traversal, for the case where a direct connection fails, uses an optional TURN relay. Credentials are minted on the trusted application tier as standard time-limited TURN-REST tokens that any WebRTC client can use. The relay VMs only verify those credentials and forward media; they hold no database and no signing secret, so they stay stateless and cheap to run in each region. A key id inside each credential lets the shared secret rotate with no downtime.

The fleet_allocation mode and the game-server fleet behind it are beta. See Matchmaking, P2P Connectivity and TURN Relay, and Game Server Fleet.

Realtime delivery

Realtime uses a single in-process hub that maps a connected player to a socket writer. There is no cross-node fan-out, because a single-binary deployment does not need one. Every producer (a match result, a presence change, a game invite) writes a durable record first and then tries the socket, and a missing socket is treated as normal. Clients recover a missed message by polling the durable state, so socket delivery is best-effort while the persisted record carries the guarantee. Live sockets also re-check the session epoch and account state on each heartbeat, so a ban or an unlink reaches a socket that outlived its handshake.

Background jobs and shutdown

Periodic maintenance runs on River, a Postgres-backed job queue. Garbage collection of expired sessions, invites, trusted devices, and old matchmaker records runs as leader-elected periodic jobs, so each one runs once for the whole fleet on whichever instance holds leadership. If River cannot start, the error is logged and the server still boots; forgot-password delivery then falls back to in-process goroutines.

Shutdown is ordered. The HTTP server drains first, then the matchmaker worker is cancelled with its own deadline, so an allocation in flight can finish committing its match before the process exits.

Two smaller choices protect the server under load. The request deadline is kept below the server's write timeout, so a request stuck on a saturated connection pool fails fast with a 503 and Retry-After before the connection is force-closed. Staleness-tolerant reads (leaderboards, friends, presence, storage) route to a read pool that runs read-only transactions; when no replica is configured, that pool aliases the primary, so every host runs identical code.

The control panel and player website

Two surfaces are server-rendered HTML rather than JSON: the control panel for operators and the player website for global accounts. Both use templ templates and Pico CSS, with cookie sessions in place of bearer tokens. The control panel adds HTMX for interactivity; the player website runs no JavaScript at all, because its content-security policy blocks every script. All assets are self-hosted, with no CDN and no inline script or style. These surfaces mount only when enabled, and they share the same database and mailer as the API.

Self-hosting and the commercial boundary

A default deployment runs with no secrets configured. Keys for player sessions, 2FA, and email verification auto-generate into a global server_secrets table on first boot, using a race-safe insert and read-back so several instances starting at once settle on the same key. The internal entitlement and billing keys generate the same way, but only once those features are turned on. Providing a key through the environment is optional hardening that keeps it out of database backups. Quotas stay dormant unless a tenant is explicitly marked for enforcement, so a self-hoster is never capped by the tier ladder.

Production is stricter, by design. Config validation refuses to boot when the production posture is unsafe or contradictory. It requires an explicit region, a distinct migration URL, non-wildcard CORS, secure cookies, an HTTPS control-panel URL, a metrics token (unless metrics auth is explicitly disabled), and real player-JWT and email-verification keys. A single failed check stops the boot, on the reasoning that an unsafe production setting is an operator mistake worth surfacing at once. See Security and Run in Production.

Commercial billing lives in a separate closed repository. The open-source build exposes only a declarative entitlement API: an external billing service states a desired tier and feature set for a tenant, and the apply path reuses the same tier update, grant upsert, audit record, and admin email as a human operator's change request. Human and billing changes behave the same way and differ only by the recorded actor. A signed, short-lived handoff token carries a tenant id out to the billing service for an upgrade link. The relay and the game-server fleet are the two entitlement-gated features; the game-server fleet ships as beta.