Cross Shard Handoff

Cross-Shard Handoff

Audience: Engine Developer Status: ✅ Ready

A zone — and the players in it — can move between world servers with zero dropped messages and exactly-once input semantics. This is the backbone of scaling in and out and of rolling upgrades: a draining shard hands its zones off to a peer while players keep their connection. The mechanism is a fenced two-phase protocol (Prepare/Commit) plus a per-player ownership epoch and a signed snapshot, all resting on the single-writer zone model.

Related: Distributed Systems Model, Edge & Protocol (the gate-side redirect/replay), RPC & Protobuf (the Handoff service), Persistence & Durability.

Why it exists

Each zone is owned by exactly one goroutine on one world server, arbitrated by a time-fenced Redis CAS lease. Two things must be true through a migration: no state is lost, and there is never a moment with two writers or an ownerless zone. The handoff carries the authoritative in-memory snapshot directly over the wire (never through the datastore), and flips the lease atomically so ShardForZone never observes a gap.

The player handoff (a cross-zone walk)

When a player walks across a zone boundary whose destination is on another shard, the source freezes and snapshots the player and the world redirects the gate:

sequenceDiagram
    participant W1 as source shard
    participant W2 as dest shard
    participant G as telos-gate
    W1->>W1: freeze player, build signed PlayerSnapshot
    W1->>W2: Handoff.Prepare(snapshot, epoch, target, snapshot_sig)
    W2->>W2: verify sig, rehydrate PENDING (applyStateComponents)
    W2-->>W1: {handoff_token, target_shard_addr}
    W1-->>G: ServerFrame{Redirect: addr, token}
    G->>W2: re-dial Play stream + Attach{handoff_token}
    W2->>W1: Handoff.Commit(token) → activate
    W2-->>G: Attached (ack_input_seq = resume point)
    G->>W2: replay buffered input seq > ack (exactly-once)

Loading

The PlayerSnapshot (RPC & Protobuf) is the authoritative state so the destination resumes with zero DB round-trips: stats, vitals, inventory/equipment, affects, flags, plus state_version (the CAS base), applied_seq (the freeze-point input high-water that seeds the destination's dedup watermark), persist_id (so the destination CASes the same durable row), and tier.

• The gate side buffers un-acked input, freezes on the Redirect, re-dials the stream (the TCP socket never moves), and replays from the destination's ack_input_seq. The world dedups by seq, so a replayed line applies exactly once. See Edge & Protocol.
• Storage is bypassed: dumpStateJSON reuses the same serializer as the durable save (byte-identical), and a handed-off character is removed without a save so the destination's epoch/state_version is never raced by a stale flush from the source (Persistence & Durability).

Abort rolls the pending state back on failure/timeout; the pending record has a TTL.

Snapshot signing and the tier

The snapshot is Ed25519-signed over a canonical digest binding character, epoch, target, and state; the destination verifies it when it has a key. This is why the account trust tier rides the snapshot directly rather than the reserved capability flags: carrying the flags would be a forgeable escalation surface (a malicious snapshot could inject admin), so the destination re-derives the reserved flags from the signed tier via applyTierFlags on arrival. An admin or builder therefore keeps elevation across a shard walk — see Trust Tier Model. Two documented caveats: wizinvis is a session concealment (never tier-grantable), so it clears on arrival — a deliberate one-time presence "flicker" across the boundary; and elevation survives only on a signed path — a keyless dev/test shard that skips verification fails closed to baseline (those are single-shard and never hand off).

The zone handoff (graceful drain)

BeginDrain (on SIGTERM) moves whole zones off a shard while they're still live — the signal context is deliberately separate from the zone-lifetime context so flush + handoff precede loop teardown:

1. Set draining — reject new fresh logins, still accept inbound handoff binds.
2. For each hosted zone, choose a peer and atomically flip the lease via HandoverZone (a fenced CAS that flips owner only if the source is still the live owner and sets a fresh TTL in the same script — closing the ownerless-gap window), then post a drain message.
3. Each zone fans its players off in place — same zone id, same room, now owned by the peer — via the shared handoff path. The socket stays open; the player is redirected (zero drop).
4. Wait until every zone empties or the deadline; stragglers are durably flushed and left to resume from durable state on reconnect (counted as reclaimed, not zero-drop).

A gate-wedged player is deliberately not handed off, and not counted zero-drop. The Redirect frame travels to the gate over the player's out channel, so if the gate has stopped reading the Play stream (its buffer is full — the same "wedged" threshold that reclaims a slow client, see Edge & Protocol), the Redirect would drop like any other frame: the gate would never re-dial, and the player would actually drop and reconnect from durable state while the drain reported them as a clean redirect. So the drain skips a wedged player, leaving it resident to be reclaimed at the deadline (a clean reconnect) and classified as a client-fault straggler — the same shape as a link-dead player holding the drain to its deadline. (The complete fix is a gate ack on the Redirect; this is the interim heuristic.)

An unexpected lease loss takes a different path: the fence stops the world without a drain (you can't hand off zones you no longer own). See Running at Scale for what an operator triggers.

Zone adoption: build, confirm, or un-adopt

A rebalance moves a whole zone between shards, and the two ends are separated in time: AdoptZone makes the destination build and run the zone (rooms, resets, mob spawns, an actor goroutine, a scope subscription), and the source's HandoverZone lease flip, several steps later, is what actually transfers ownership. Three guards keep that window from leaking a permanently orphaned "zombie" zone:

• The destination requires from_shard_id to be the zone's live owner and refuses before doing any state work. This isn't a security barrier — owner and gen come from the same directory read, so anyone who can satisfy the generation fence already knows the owner — it enforces at the destination the precondition the source's flip asserts anyway, so a misnamed source (desynced, lagging, mid-partition, or buggy) can't make the destination build a zone it will never own.
• Adoption is confirmed by observing the flip land, not by the RPC returning. The adopting shard's lease-renewal loop tries to ClaimZone; a landed flip — from the source, or from any sibling that wins the CAS — makes that claim succeed and marks the adoption confirmed. The gate is "did the flip land?", observed via lease acquisition, never "is this RPC's context still alive?" (tearing down on ctx.Err() would delete a zone a concurrent sibling handoff had legitimately just flipped to us).
• An unconfirmed adoption is un-adopted when its confirm deadline expires. If the flip never lands — the source's drain deadline elapsed while the AdoptZone was in flight, or the source died mid-drain — the renewer runs the teardown (Zone Runtime & Actor Model). It is gated on an adopted flag, not merely on "unconfirmed": a boot zone that genuinely loses its lease must fence, not delete itself — only a zone built at runtime for an in-flight handoff is ever un-adopted. Teardown is best-effort: a refusal (the zone has since acquired a player, or owes a durable write) keeps the zone, because a recoverable leak beats a dropped player.

Drain-target selection is director-owned and serialized. Rather than "first live peer," the selector picks a target against a soft occupancy ceiling and atomically reserves its headroom in the directory, so two concurrent drains don't pile onto the same peer. Each reservation carries its own expiry (on the Redis server's clock), so a crashed drainer's stale hold is pruned rather than inflating the reserved sum and starving the target. If every peer is genuinely reservation-full it admits over the soft ceiling rather than stall (a dropped connection is worse than transient overload, which the rebalancer then corrects). See Running at Scale.