Cross Shard Handoff

Cross-Shard Handoff

Audience: Engine Developer Status: ✅ Ready

A zone — and the players in it — can move between world servers with zero dropped messages and exactly-once input semantics. This is the backbone of scaling in and out and of rolling upgrades: a draining shard hands its zones off to a peer while players keep their connection. The mechanism is a fenced two-phase protocol (Prepare/Commit) plus a per-player ownership epoch and a signed snapshot, all resting on the single-writer zone model.

Related: Distributed Systems Model, Edge & Protocol (the gate-side redirect/replay), RPC & Protobuf (the Handoff service), Persistence & Durability.

Why it exists

Each zone is owned by exactly one goroutine on one world server, arbitrated by a time-fenced Redis CAS lease. Two things must be true through a migration: no state is lost, and there is never a moment with two writers or an ownerless zone. The handoff carries the authoritative in-memory snapshot directly over the wire (never through the datastore), and flips the lease atomically so ShardForZone never observes a gap.

The player handoff (a cross-zone walk)

When a player walks across a zone boundary whose destination is on another shard, the source freezes and snapshots the player and the world redirects the gate:

sequenceDiagram
    participant W1 as source shard
    participant W2 as dest shard
    participant G as telos-gate
    W1->>W1: freeze player, build signed PlayerSnapshot
    W1->>W2: Handoff.Prepare(snapshot, epoch, target, snapshot_sig)
    W2->>W2: verify sig, rehydrate PENDING (applyStateComponents)
    W2-->>W1: {handoff_token, target_shard_addr}
    W1-->>G: ServerFrame{Redirect: addr, token}
    G->>W2: re-dial Play stream + Attach{handoff_token}
    W2->>W1: Handoff.Commit(token) → activate
    W2-->>G: Attached (ack_input_seq = resume point)
    G->>W2: replay buffered input seq > ack (exactly-once)

Loading

The PlayerSnapshot (RPC & Protobuf) is the authoritative state so the destination resumes with zero DB round-trips: stats, vitals, inventory/equipment, affects, flags, plus state_version (the CAS base), applied_seq (the freeze-point input high-water that seeds the destination's dedup watermark), persist_id (so the destination CASes the same durable row), and tier.

• The gate side buffers un-acked input, freezes on the Redirect, re-dials the stream (the TCP socket never moves), and replays from the destination's ack_input_seq. The world dedups by seq, so a replayed line applies exactly once. See Edge & Protocol.
• Storage is bypassed: dumpStateJSON reuses the same serializer as the durable save (byte-identical), and a handed-off character is removed without a save so the destination's epoch/state_version is never raced by a stale flush from the source (Persistence & Durability).

Abort rolls the pending state back on failure/timeout; the pending record has a TTL.

Snapshot signing and the tier

The snapshot is Ed25519-signed over a canonical digest binding character, epoch, target, and state; the destination verifies it when it has a key. This is why the account trust tier rides the snapshot directly rather than the reserved capability flags: carrying the flags would be a forgeable escalation surface (a malicious snapshot could inject admin), so the destination re-derives the reserved flags from the signed tier via applyTierFlags on arrival. An admin or builder therefore keeps elevation across a shard walk — see Trust Tier Model. Two documented caveats: wizinvis is a session concealment (never tier-grantable), so it clears on arrival — a deliberate one-time presence "flicker" across the boundary; and elevation survives only on a signed path — a keyless dev/test shard that skips verification fails closed to baseline (those are single-shard and never hand off).

The zone handoff (graceful drain)

BeginDrain (on SIGTERM) moves whole zones off a shard while they're still live — the signal context is deliberately separate from the zone-lifetime context so flush + handoff precede loop teardown:

1. Set draining — reject new fresh logins, still accept inbound handoff binds.
2. For each hosted zone, choose a peer and atomically flip the lease via HandoverZone (a fenced CAS that flips owner only if the source is still the live owner and sets a fresh TTL in the same script — closing the ownerless-gap window), then post a drain message.
3. Each zone fans its players off in place — same zone id, same room, now owned by the peer — via the shared handoff path. The socket stays open; the player is redirected (zero drop).
4. Wait until every zone empties or the deadline; stragglers are durably flushed and left to resume from durable state on reconnect (counted as reclaimed, not zero-drop).

An unexpected lease loss takes a different path: the fence stops the world without a drain (you can't hand off zones you no longer own). See Running at Scale for what an operator triggers.

Drain-target selection is director-owned and serialized. Rather than "first live peer," the selector picks a target against a soft occupancy ceiling and atomically reserves its headroom in the directory, so two concurrent drains don't pile onto the same peer. If every peer is reservation-full it admits over the soft ceiling rather than stall (a dropped connection is worse than transient overload, which the rebalancer then corrects). See Running at Scale.