Map scheduler design

The design for the map scheduler: how every map in the process gets its ticks, on a shared pool of threads or a thread of its own, each to its own deadline. What a map does inside a tick is on Owners design, and its timers on Timer wheel design. Step 3's second stage, one character alone in the world, builds it; Roadmap says how far that has got.

The idea

In short

A map is ticked, never threaded: a pool of threads takes whichever map's deadline is earliest and has come, runs one tick to completion, and puts the map back with its next deadline. A map placed on a dedicated thread runs alone on it. No map ticks on two threads at once, and a deadline missed while a tick ran is skipped, not replayed.

Why

Every house is an instance, so a server holds far more maps than cores, and most of them are idle. A thread per map would wake thousands of threads for their own ticks and preempt them mid-tick once they outnumber the cores. A pool runs the long tail cheaply, and a dedicated thread is there for the few maps heavy enough to need one.

How it fits together

flowchart TD
    MD["Map directory"] -->|"register, unregister"| MS["Map scheduler"]
    MS -->|"earliest due deadline"| P["Pool threads"]
    MS -->|"placed dedicated"| D["Dedicated thread"]
    P -->|"one tick, to completion"| M1["Map"]
    P -->|"one tick, to completion"| M2["Map"]
    D -->|"its ticks only"| M3["Heavy map"]

Loading

Decisions

Locked

Maps share a pool, and a heavy map gets a thread of its own. The pool runs the long tail (housing plots, small instances, quiet zones), earliest deadline first, each tick run to completion. A map registered as dedicated gets its own thread while the dedicated budget has one free, and runs in the pool, with a warning, when it hasn't. Pool and dedicated threads together stay within the cores left after the gateway, the realm and account owners and the garbage collector.

A map's next deadline is the first one on its period not yet past. A deadline that passed while a tick ran is skipped and counted, never replayed, so a slow tick never turns into a burst of catch-up ticks.

Nothing ticks in step. Each map's first deadline is staggered by a random fraction of one tick, so maps registered together don't all tick in the same millisecond. Saves are staggered the same way, on Owners design.

The tick and the update interval are set per instance type, as data in the world files, with a runtime-settings override per type that takes effect at the next map load.

• The tick divides 250 ms (25, 50, 125 or 250 ms), so every effect tick on a multiple of 250 ms, 98% of them, still lands exactly.
• The update interval is a whole number of ticks inside the client's 25 to 1,000 ms, sent to the client, and fixed for the instance's lifetime.
• The open world, capitals, dungeons, raids and PvP start at a 25 ms tick and 125 ms updates; whether PvP needs a shorter interval is the load test's call. Housing plots and small social instances start at a 125 ms tick and 250 ms updates. All of them get tuned from play at milestone one.
• The second stage hosts open-world maps only, so it has one tick and one update interval for every map.

A tick that throws is logged and counted, and the map keeps its place. Faults inside a map, a message handler or a timer that throws, are counted per world as well. Repeated faults stop the map, disconnecting its players with their state as of the last confirmed save, as Owners design says.

Unregistering waits for the tick in progress. It completes at once for an idle map, and for a running one when its current tick ends; after that the map never ticks again. Stopping the server stops every map while the scheduler still ticks, so each map hands its characters back on its own last tick, and only then stops the scheduler.

Every tick is measured: its duration and the bytes it allocated, by world, beside counts of ticks, skipped ticks and faults. Recording them allocates nothing while nothing listens.

Metric	Records
nexustomorrow.map.ticks	Ticks completed, by world
nexustomorrow.map.tick.duration	Each tick's duration in milliseconds, by world
nexustomorrow.map.tick.allocated	The bytes each tick allocated, by world
nexustomorrow.map.ticks_skipped	Deadlines passed over
nexustomorrow.map.faults	Ticks that threw, and handler and timer faults by world

Note

On Windows a scheduler thread waits on a high-resolution waitable timer, because .NET's timed waits there step in 10 to 16 ms increments, too coarse for a 25 ms tick. On Linux it waits on its wake event with the deadline as the timeout, which keeps millisecond precision and also wakes a pool thread at once when a map registers.

Knobs

Knob	Default	Sets
Maps:PoolThreads	the cores less three, at least 1	The shared pool that runs the long tail of maps
Maps:DedicatedThreads	0	How many threads heavy maps may hold between them
Maps:TickMs	25	The map tick: 25, 50, 125 or 250 ms
Maps:ReplicationEveryTicks	5	Ticks per update interval; the interval has to fall within the client's 25 to 1,000 ms

Which map types start on a dedicated thread, and how many overruns promote a pool map, are data as well, and arrive with the load work.

Open

• Promotion: a pool map that keeps overrunning its tick moves to a dedicated thread and returns when it settles. The second stage places maps only where they register; promotion, and stopping a map after repeated faults, come with the load test.

Rejected

Idea	What killed it
A thread per map	Every house is an instance, so thousands of mostly idle threads would each wake for their own tick, and once they outnumber the cores they're preempted mid-tick

Verification status

Claim	Basis
The tick rates, 40 Hz and 8 Hz for housing	Chosen. Confirmed by measurement once the loop runs
The update interval's 25 to 1,000 ms range	Established from the client
98% of effect ticks on a multiple of 250 ms	Counted from the tables
The Windows timer holds a 25 ms tick	Owed: measured on a Windows machine once the second stage lands, from the tick and skip counts

Open questions

• Tick rates: measure 40 Hz for most maps and 8 Hz for housing once the loop runs.
• The load test: how many maps one pool thread carries, and the overrun count that should promote a map.