Pillar v0.2.0 — The Decision Engine
Pre-release0.1.0 taught the fleet to see each other and pass messages. 0.2.0 teaches it
to decide: nodes now publish their own health, and Pillar chooses where work
should land — filtering on hard caps, then balancing with power-of-two-choices
over an in-memory view. The reliability tier underneath grew up too.
Pre-release — the placement logic is complete and tested, but not yet wired
to player routing. That actuation lands in Iteration 3 (the MVP line).
Health & placement
- Health snapshots — each node publishes MSPT, memory, player/world counts, and pending-signal depth, TTL-bounded and self-healing like presence; any node reads the fleet's latest health.
- Eligibility (hard caps) — a node over its player, MSPT, or memory ceiling is filtered out before any optimization runs.
- Power-of-two-choices — samples two eligible nodes and keeps the lighter, over an in-memory snapshot: O(1), with no herd toward a single "best" node.
- In-flight reservations — TTL'd and counted into scoring, so a burst of concurrent placements spreads across the fleet instead of stampeding one node.
- Least-connections fallback — when health data is stale or absent, selection degrades automatically instead of routing blind.
- Simulation harness — synthetic bursts against a fleet larger than the real 2-node topology prove even distribution and cap adherence at volume.
Reliable dispatch
- Bounded worker pool — handlers run off the consumer thread; a saturated pool blocks the consumer, so backpressure is natural rather than dropped work.
- Ack-after-success —
XACKonly after a handler completes; failures stay pending. - PEL reclaim + dead-lettering — orphaned entries (e.g. after
kill -9) are drained and retried on restart, and dead-lettered after N attempts, so nothing accumulates unbounded. - At-least-once dedup — a redelivered entry is recognized and never re-executed; the dedup window is TTL-bounded.
Hardening
- Redis package split into
presence/transport/lifecyclefor clearer boundaries. - Fleet reads (
/pillar fleet, ping presence, tab-completion) served from an in-memory snapshot — noSCAN+MGETon a platform thread. JedisPoolsized explicitly (config-driven), replacing the library defaults that starve under login bursts and block callers forever.
Upgrading from 0.1.0
Drop-in. Two new optional keys under redis.pool (max-total, max-wait-millis) with safe defaults — set them only when tuning for load. No breaking config changes. Requirements unchanged: Paper 26.1.2 and/or Velocity 3.4.0, Java 25, Redis 7.
Known limitations
- Placement is decision-only here — the selector is not yet connected to player connections. It ships as pure, tested logic; the wiring and the typed no-eligible-node outcome land in Iteration 3 with its first real consumer.
- Cross-consumer PEL recovery — reclaiming a dead peer's pending entries (
XAUTOCLAIM) — is still pending; only a node's own orphans are reclaimed. - Carried from 0.1.0: dispatch-level self-send misroute; extracted
lang/files can shadow newer keys.
What's next (Iteration 3 — MVP)
Actuation. Player routing wires this engine to the Velocity connect event, reading health from an in-memory cache so the login path never touches Redis; a generic lease primitive (resource → owner, TTL, renewal) gives consumers atomic cross-server exclusivity; and player messaging completes FR-05. End of Iteration 3 = the MVP — a control plane that both decides and acts.