crash-loop

A 2D infrastructure & automation puzzle. You are a Site Reliability Engineer at a fictional cloud provider: each level hands you a failing distributed system, and you compose a topology of nodes — load balancers, services, caches, queues — that survives simulated traffic inside an error budget and a resource budget. No physics, no twitch, no 3D. Backend over bullets.

Built by Team Three-Way Merge (Gabriel Felipe Guarnieri · Hector Guarçoni Machado · Marcos Winícios Silva Martins) for Software Engineering for Games — 2026/1.

Run it

▶ Play it live: https://oguarni.github.io/crash-loop/ — deployed from main by GitHub Actions on every push (no install required).

To run it locally:

npm install
npm run dev      # vite dev server, opens the playable build

Other scripts:

npm run build      # type-check (tsc --noEmit) + production bundle into dist/
npm run preview    # serve the production build locally
npm run typecheck  # type-check only
npm run test       # full Vitest suite (sim, board rules, progress)
npm run coverage   # Vitest with v8 coverage thresholds (>=80 / >=95 core)
npm run test:sim   # headless deterministic sim-check smoke harness (L01–L06)

L01 — "boot" (the first playable level)

svc-cart takes public traffic on a single replica and folds under load. Incoming traffic is 30 req/tick; one service handles 10.

• ingress is a single public entry point — it can only feed one downstream node, so you must route it through a load-balancer to fan traffic out.
• The load-balancer splits its inflow evenly across every downstream service.
• Budget caps you at $5.00 / 8 CPU / 8 MEM, and dropped requests must stay within the error budget (20).

The dominant correct topology:

ingress ──> load-balancer ──┬──> service
                            ├──> service
                            └──> service

3 services × 10 = 30 req/tick (zero drops), at $4.50 — which also clears the gold tier (parCost).

Screen flow

The boot/title screen leads into a level-select menu — every level is a card showing its saved best, so a presenter can jump straight into any scenario (click a card, or press its number key 1–7). Inside a level, the rail's < menu affordance or a clean Esc returns to the select screen; you no longer have to clear levels in sequence to reach the later ones.

Controls

Action	How
Place	Click a component in the left rail, then click an empty spot in the work area
Wire	Select wire, click the source node, then the target node
Move	Select move, drag a node — a node dropped on another is nudged to the nearest free slot
Delete	Select delete, click a node or an edge
Run	Run > (or Enter) — simulate the traffic profile and score it
Pause	Pause (or Space / P) freezes a running sim; Resume continues
Help	? or H toggles an in-game help / legend overlay (controls + the node kinds this level uses)
Mute	M (or the rail's mute affordance) toggles all audio, including the ambient hum
Cancel / back	Esc cancels an in-progress wire or clears a selection; a clean Esc returns to the level menu

Clear a level and a Next > button appears on the result banner to advance.

Scoring & progress

Each run is graded into a tier — FAIL, PASS (error budget held), or GOLD (also at or under parCost). Beyond the tier, a run is scored on three axes, surfaced side by side on the result banner and carried as saved bests in the rail:

• cost — the dollar total of the topology (always live);
• cycles — request-ticks spent waiting in a buffer, so it only matters where a queue exists; other levels show a clean — rather than a misleading 0;
• coverage — the share of traffic that passed through a CI gate, live only where a gate is present (or required).

The best tier and the best of each relevant axis are kept per level in localStorage, so a cleared scenario shows its saved bests on return, and a run that beats a record flags a NEW BEST. Verdicts are never signalled by colour alone — a tier always pairs with a word (PASS / FAIL / GOLD). The title screen reports how many regions you've stabilised. Scoring is meta state only — it never feeds the deterministic simulation.

L02 — "first deploy"

A new svc-cart release cuts over to production, but every request must clear a canary deploy gate first. Traffic rises to 40 req/tick.

• requireBeforeSinks: ['gate'] — every path from ingress to a service must pass through a ci-gate, or the run is rejected ("untested traffic reached production"). It's a topology rule, checked before the traffic ever flows.
• A ci-gate forwards only 20 req/tick, so one gate throttles production — you need two, fanned out from the load-balancer.
• Budget caps you at $8.00 / 8 CPU / 8 MEM; the error budget is 40.

The dominant correct topology:

ingress ──> load-balancer ──┬──> ci-gate ──┬──> service
                            │              └──> service
                            └──> ci-gate ──┬──> service
                                           └──> service

lb splits 40 → 20/20 to the gates → 10/10 to four services (cap 10): zero drops, at $7.50 — which also clears the gold tier. A third gate or fifth service would breach the $8.00 budget, so this build is the unique solution.

L03 — "flapping cart"

svc-cart is flapping under a flood of repeated reads, and adding replicas is priced out. This level introduces the cache node: it serves a fixed fraction of its inflow locally (a cache hit) and forwards only the misses downstream — and, like a load-balancer, it splits those misses evenly.

• A cache has hitRate: 0.5: of 40 req/tick it serves 20 as hits and forwards 20 as misses. It's cheap in cost but heavy in memory (mem 2).
• Budget caps you at $4.50 / 6 CPU / 6 MEM; the error budget is 40.

The dominant correct topology:

ingress ──> cache ──┬──> service
                    └──> service

The cache serves 20 locally and forwards 20 → 10/10 into two services: zero drops, at $3.00 (gold). The cacheless brute force (lb + 4 services = $5.50) is over budget, so you can't out-spend the problem — you have to cache. Chaining a second cache does not help: it receives only the first one's misses — the reads that are not repeated — so it serves nothing and forwards all 20 into a replica that caps at 10.

L04 — "error budget"

This level flips the lesson: until now the goal was zero drops; here serving everything is deliberately unaffordable. Traffic holds at a steady 20 req/tick, spikes to 40 for five ticks, then settles back.

• Budget caps you at $5.00 / 6 CPU / 6 MEM; the error budget is 120.
• The zero-drop build (lb + 4 services = $5.50) is over budget — you cannot buy your way out of the spike.

The dominant correct topology:

ingress ──> load-balancer ──┬──> service
                            └──> service

Two services (cap 20) serve the steady 20 with zero drops; during the spike they shed 100 requests total, which sits inside the 120 error budget — at $3.50 (gold). A safer lb + 3 services ($4.50) passes with only 50 drops but misses gold. The lesson: spend the error budget instead of overspending on capacity.

L05 — "chaos friday"

It's Friday and chaos is loose: replicas crash mid-run, without warning. A seeded schedule knocks out one service at a time (two incidents, five ticks each), and while a replica is down its capacity is 0 — everything routed to it is dropped. Traffic is a steady 20 req/tick.

• Budget caps you at $7.00 / 8 CPU / 8 MEM; the error budget is 55.
• The incident schedule lives entirely in a per-level seed — you can't see the exact timing, so you build for the failure, not around it.

The dominant correct topology:

ingress ──> load-balancer ──┬──> service
                            ├──> service
                            ├──> service
                            └──> service

The lesson is resilience through redundancy. The load-balancer splits 20 evenly, so with four services each carries only 5 req/tick; when one crashes, only its 5/tick are shed — across two 5-tick incidents that's 50 dropped, inside the 55 error budget, at $5.50 (gold). Two or three services carry a bigger share (10 or ~7 per replica), so losing one blows the budget. Because the gold build is symmetric, which replica the seed picks never changes the outcome — so the run stays fully deterministic.

L06 — "back-pressure"

The counterpoint to L04: instead of dropping a spike, you buffer it. This level introduces the queue — the one stateful node, whose buffer carries across ticks. Traffic sits at a steady 10 req/tick, spikes to 40 for five ticks, then settles back, leaving room to drain.

• A queue drains up to 20 req/tick and holds up to 100 across ticks; when the buffer is full it sheds the overflow (back-pressure). Like a cache, it splits its released traffic evenly across downstream edges.
• Budget caps you at $5.00 / 6 CPU / 6 MEM; the error budget is 20.
• Peak provisioning (lb + 4 services = $5.50) is over budget — you must buffer, not out-spend, the spike.

The dominant correct topology:

ingress ──> queue ──┬──> service
                    └──> service

The queue releases 20/tick and buffers the surplus (peaking at exactly 100 during the spike), then drains it over the calm tail. Two services (cap 20) match the drain rate, so nothing is dropped, at $4.00 (gold). A single downstream service can't keep up with the queue's own drain and fails. Requests still buffered when the run ends count as dropped — you must drain in time.

L07 — "black friday"

The finale. It stacks every mechanic the campaign taught into one topology, so all three scoring axes are live at once. Peak read traffic sits at 32 req/tick, bursts to 56 for eight ticks (the Black Friday spike), then eases back over a long recovery tail; two seeded incidents knock a replica out mid-run.

• cache halves the heavy read load, so you provision for the misses, not the full arrival rate — without it the downstream is unaffordable;
• queue soaks the burst and bleeds the backlog off across the quiet ticks (this is the cycles axis — request-ticks spent waiting). It is required on every path to a replica: after the cache halves the load the burst is only 28 req/tick, so a second $1.00 gate would carry it unbuffered — a $7.00 build that beat par on every axis and dissolved the lesson;
• ci-gate is required before every replica, which drives coverage to 100%. Both rules ride on requireBeforeSinks: ['gate', 'queue'] — every path from ingress to a replica must cross each listed kind;
• chaos (as in L05) sheds a replica's share during each incident, so you spread the load across enough replicas to stay inside the error budget.
• Budget caps you at $9.00 / 10 CPU / 12 MEM; the error budget is 52.

The dominant correct topology:

ingress ──> cache ──> queue ──> ci-gate ──┬──> service
                                          ├──> service
                                          ├──> service
                                          └──> service

The cache halves the reads, the queue drains ≤20/tick and holds the burst, the gate forwards ≤20 to four replicas carrying ~4–5 req/tick each — so losing one to an incident sheds only its small share: 45 dropped, inside the 52 error budget, at $8.00 / 768 cycles / 100% coverage (gold). Three replicas shed too big a share on a crash (60 dropped) and fail; a fifth replica clears the drops but breaches the cost par.

Architecture

src/
  types.ts          shared domain types (incl. ChaosSpec)
  palette.ts        canonical Three-Way Merge palette
  layout.ts         geometry constants + hit-testing helpers
  sim/
    nodes.ts        per-kind specs (cost, capacity, fan-out, cache hit-rate, queue buffer)
    rng.ts          deterministic seeded PRNG (mulberry32) for chaos
    engine.ts       deterministic per-tick topological flow simulation
  levels/
    L01.ts          "boot"          — routing / load balancing
    L02.ts          "first deploy"  — deploy gate rule
    L03.ts          "flapping cart" — cache node
    L04.ts          "error budget"  — traffic spike, tight budget
    L05.ts          "chaos friday"  — seeded incident injection
    L06.ts          "back-pressure" — queue node (cross-tick buffering)
    L07.ts          "black friday"  — finale: cache + queue + gate + chaos
    index.ts        level register (played in order)
  game.ts           board state, editing rules, run/playback (framework-agnostic)
  progress.ts       persistent per-level scoring (localStorage, sim-independent)
  render.ts         all canvas drawing + shared hit-region layouts
  main.ts           DOM wiring, input, the playback loop
scripts/
  sim-check.ts      headless deterministic verification harness (npm run test:sim)

Design notes

• The simulation is fully deterministic — the same topology, traffic profile and seed always produce the same result (a design pillar). The chaos mechanic (L05) is a seeded incident injection, not wall-clock noise: the schedule is a pure function of the level seed, generated once before the tick loop.
• Traffic flows through the graph in topological order each tick; a cycle is rejected as an invalid topology (a real DAG constraint).
• Node behaviour is data-driven: fanOut nodes split evenly, a hitRate node (cache) serves a fraction and forwards the rest, a buffer node (queue) holds traffic across ticks and drains at its capacity, and plain sinks (services) handle up to capacity and drop the overflow.
• The queue is the only stateful node: its buffer persists between ticks. Requests still held when the run ends are counted as dropped, so conservation (served + dropped === arrived) always holds and a solution must drain in time.
• game.ts holds no rendering or DOM code, so the rules are unit-testable and the renderer is replaceable.

Roadmap

Shipped: L01 — boot · L02 — first deploy · L03 — flapping cart · L04 — error budget · L05 — chaos friday · L06 — back-pressure · L07 — black friday (finale). All six roadmap node kinds are live: ingress, load-balancer, service, ci-gate, cache, queue. Also shipped:

• Level select — a title-screen menu to jump into any scenario, no longer gated behind clearing levels in order.
• Multi-axis scoring — cost, cycles, and coverage surfaced side by side on the result banner and rail.
• In-game help / legend — a ? / H overlay with the controls and the node kinds each level uses.
• Terminal polish — CRT vignette + contrast pass and a low ambient hum; IBM Plex Mono is now self-hosted, so a live demo needs no network at all.

Planned:

• Infrastructure as Code — declare part of a topology from a script/template.
• Narrative & NPCs — diegetic incident briefings and the senior SRE mentor.
• Thematic campaign — group levels into worlds (Ingress, Queues, Services, CI/CD, Data/Cache, SRE panel) with boss scenarios.