Cross Shard Handoff

Cross-Shard Handoff

Audience: Engine Developer Status: ✅ Ready

A zone — and the players in it — can move between world servers with zero dropped messages and exactly-once input semantics. This is the backbone of scaling in and out and of rolling upgrades: a draining shard hands its zones off to a peer while players keep their connection. The mechanism is a fenced two-phase protocol (Prepare/Commit) plus a per-player ownership epoch and a signed snapshot, all resting on the single-writer zone model.

Related: Distributed Systems Model, Edge & Protocol (the gate-side redirect/replay), RPC & Protobuf (the Handoff service), Persistence & Durability.

Why it exists

Each zone is owned by exactly one goroutine on one world server, arbitrated by a time-fenced Redis CAS lease. Two things must be true through a migration: no state is lost, and there is never a moment with two writers or an ownerless zone. The handoff carries the authoritative in-memory snapshot directly over the wire (never through the datastore), and flips the lease atomically so ShardForZone never observes a gap.

The player handoff (a cross-zone walk)

When a player walks across a zone boundary whose destination is on another shard, the source freezes and snapshots the player and the world redirects the gate:

sequenceDiagram
    participant W1 as source shard
    participant W2 as dest shard
    participant G as telos-gate
    W1->>W1: freeze player, build signed PlayerSnapshot
    W1->>W2: Handoff.Prepare(snapshot, epoch, target, snapshot_sig)
    W2->>W2: verify sig, rehydrate PENDING (applyStateComponents)
    W2-->>W1: {handoff_token, target_shard_addr}
    W1-->>G: ServerFrame{Redirect: addr, token}
    G->>W2: re-dial Play stream + Attach{handoff_token}
    W2->>W1: Handoff.Commit(token) → activate
    W2-->>G: Attached (ack_input_seq = resume point)
    G->>W2: replay buffered input seq > ack (exactly-once)

Loading

The PlayerSnapshot (RPC & Protobuf) is the authoritative state so the destination resumes with zero DB round-trips: stats, vitals, inventory/equipment, affects, flags, plus state_version (the CAS base), applied_seq (the freeze-point input high-water that seeds the destination's dedup watermark), persist_id (so the destination CASes the same durable row), and tier.

• The gate side buffers un-acked input, freezes on the Redirect, re-dials the stream (the TCP socket never moves), and replays from the destination's ack_input_seq. The world dedups by seq, so a replayed line applies exactly once. See Edge & Protocol.
• Storage is bypassed: dumpStateJSON reuses the same serializer as the durable save (byte-identical), and a handed-off character is removed without a save so the destination's epoch/state_version is never raced by a stale flush from the source (Persistence & Durability).

Abort rolls the pending state back on failure/timeout; the pending record has a TTL.

Snapshot signing and the tier

The snapshot is Ed25519-signed over a canonical digest binding character, epoch, target, and state; the destination verifies it when it has a key. This is why the account trust tier rides the snapshot directly rather than the reserved capability flags: carrying the flags would be a forgeable escalation surface (a malicious snapshot could inject admin), so the destination re-derives the reserved flags from the signed tier via applyTierFlags on arrival. An admin or builder therefore keeps elevation across a shard walk — see Trust Tier Model. Two documented caveats: wizinvis is a session concealment (never tier-grantable), so it clears on arrival — a deliberate one-time presence "flicker" across the boundary; and elevation survives only on a signed path — a keyless dev/test shard that skips verification fails closed to baseline (those are single-shard and never hand off).

The zone handoff (graceful drain)

BeginDrain (on SIGTERM) moves whole zones off a shard while they're still live — the signal context is deliberately separate from the zone-lifetime context so flush + handoff precede loop teardown:

1. Set draining — reject new fresh logins, still accept inbound handoff binds.
2. For each hosted zone, choose a peer and atomically flip the lease via HandoverZone (a fenced CAS that flips owner only if the source is still the live owner and sets a fresh TTL in the same script — closing the ownerless-gap window), then post a drain message.
3. Each zone fans its players off in place — same zone id, same room, now owned by the peer — via the shared handoff path. The socket stays open; the player is redirected (zero drop).
4. Wait until every zone empties or the deadline; stragglers are durably flushed and left to resume from durable state on reconnect (counted as reclaimed, not zero-drop).

An unexpected lease loss takes a different path: the fence stops the world without a drain (you can't hand off zones you no longer own). See Running at Scale for what an operator triggers.

Drain-target selection is director-owned and serialized. Rather than "first live peer," the selector picks a target against a soft occupancy ceiling and atomically reserves its headroom in the directory, so two concurrent drains don't pile onto the same peer. Each reservation carries its own expiry (on the Redis server's clock), so a crashed drainer's stale hold is pruned rather than inflating the reserved sum and starving the target. If every peer is genuinely reservation-full it admits over the soft ceiling rather than stall (a dropped connection is worse than transient overload, which the rebalancer then corrects). See Running at Scale.