Skip to content

Engine Core Developer Guide

Jump to bottom
Priyesh Patel edited this page Oct 17, 2026 · 8 revisions

Engine Core Developer Guide

This page describes the simulation engine core delivered for US-1.1 Deterministic event engine and how the remaining tasks connect to it. It is the starting point for anyone continuing work on the simulator.

Contents

Status

Branch: feature/engine-core. All 76 tests pass, and ruff check and ruff format --check are clean.

Task Issue State Code
1.1.1 Define event types and record schema #96 Done simulator/engine/events.py
1.1.2 Implement event queue and simulation clock #8 Done simulator/engine/event_queue.py
1.1.3 Implement seed management and stream separation #9 Open plugs into StreamFactory
1.1.4 Build the network model #97 Open plugs into ComponentFactory
1.1.5 Load and validate scenario configuration #98 Open plugs into Scenario
1.1.6 Implement the simulation runner #99 Done simulator/engine/runner.py, simulator/engine/event_log.py
1.1.7 Implement run manifest recording #10 Open writes next to the run output
1.1.8 Implement determinism test in CI #11 Open can reuse the runner smoke test

Terms

Term Meaning
Discrete-event simulation A simulation that jumps from one event to the next instead of advancing in fixed time steps. Nothing changes between events
Event Something that happens at one instant, such as a train arriving or a passenger boarding
Event queue The list of future events, always kept in processing order
Simulation clock The current simulated time in seconds. It starts at 0 and only moves to the time of the next event
Tie order The fixed order used when two events happen at the same time
Component A part of the simulation (train logic, passenger logic) that schedules events and reacts to them
Master seed The single number every random value in a run is derived from (Mathematical Foundations, section 9)
Substream An independent sequence of random values for one source of randomness, such as walking times
Byte-identical Two files with exactly the same content, byte for byte

Setup and running

The README section "Getting started" lists the setup commands. In short:

py -3.12 -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements-dev.txt
.\.venv\Scripts\python.exe -m pytest -v

The smoke test output (an event log and summary for a small McGill timetable) is kept with:

.\.venv\Scripts\python.exe -m pytest tests/simulator/engine/test_runner.py --basetemp=run-output

There is no command-line program yet. One needs a scenario file to read, which depends on #98.

How a run works

sequenceDiagram
    participant R as run_scenario
    participant F as Factories (#9, #97)
    participant S as Scheduler
    participant C as Components
    participant L as Event log

    R->>R: check duration_s and master_seed
    R->>F: build_streams(master_seed)
    R->>F: build_components(scenario, streams)
    R->>C: start(scheduler) for each component
    C->>S: schedule first events
    loop until no event at or before duration_s
        S->>R: next event (clock jumps to its time)
        R->>L: write event
        R->>C: handle(event, scheduler) for each component
        C->>S: schedule follow-up events
    end
    R->>L: rename events.jsonl.partial to events.jsonl
    R->>R: write summary.json
Loading

run_scenario(scenario, build_streams, build_components, output_dir) produces two files in output_dir:

  • events.jsonl: every processed event, one JSON record per line, in processing order
  • summary.json: master seed, duration, events processed, events still pending after the end of the period, time of the last event, and a count per event type

The same scenario and master seed always produce byte-identical files. The encoding uses a fixed key order, \n line endings on every platform, and no wall clock time or file paths.

While a run is in progress the log is written to events.jsonl.partial. It is renamed only when the run finishes, so a crashed run never leaves a log that looks complete.

Event schema

Defined in simulator/engine/events.py, SCHEMA_VERSION = 1. Every event is an immutable (frozen) dataclass and is validated when created. An invalid value raises EventSchemaError naming the field, for example Boarding.train_id must be a non-empty string. All times and durations are in seconds.

Every record starts with schema_version, event_type and time_s.

Event Fields after time_s Notes
TrainArrival train_id, station_id, platform_id, scheduled_s, disruption_delay_s Disruption delay only; hold delay is recorded separately
TrainDeparture train_id, station_id, platform_id, scheduled_s, disruption_delay_s, hold_s, boarded_count, alighted_count, onboard_load The three counts cover every passenger at the stop: B, A and L in Mathematical Foundations sections 1 and 4
Hold train_id, station_id, platform_id, requested_s, applied_s, reason applied_s can never exceed requested_s; matches DecisionAuditLog on the Architecture page
PassengerArrival passenger_id, group_id, platform_id A transferring passenger reaches the platform of the next train. group_id is the transfer burst
Boarding passenger_id, train_id, platform_id Only for passengers who fit through the doors
Alighting passenger_id, train_id, platform_id A transferring passenger leaves the train that brought them
MissedConnection passenger_id, train_id, platform_id train_id is the train that left without them

Only transferring passengers are modelled one by one. Each has their own walking time (pulsed arrivals, Mathematical Foundations section 2), and their delay is the difference between two of their event timestamps. Everyone else enters the math only as counts, so they appear only in the counts on TrainDeparture.

Event.to_record() gives the flat record written to the log, and event_from_record(record) rebuilds the event. Records from another schema version, with an unknown event type, or with missing or extra fields are rejected.

Tie order

Events at the same time are processed in this order, then in the order they were scheduled. The order follows what happens at one platform in one instant. It is defined once, in EVENT_PRIORITY in events.py.

Order Event Reason
1 TrainArrival The train reaches the platform
2 Alighting Passengers get off before anyone gets on
3 PassengerArrival A passenger reaching the platform at this instant can still board
4 Boarding Passengers board during the dwell
5 Hold The hold decision is made once boarding is known
6 TrainDeparture The train leaves
7 MissedConnection A connection is only missed once the train has left

Rules enforced by the scheduler

Scheduler in event_queue.py is the only way to schedule events.

  • schedule(event) rejects an event before the current clock time
  • It also rejects an event at the current time that comes earlier in the tie order than the event just processed. For example, a Boarding at 100 s cannot be scheduled while handling a TrainDeparture at 100 s, because boarding would then happen after the train left. Components must schedule follow-up events at the same time or later in the tie order
  • run_until(end_s, handler) processes an event at exactly end_s; later events stay in the queue and are counted as pending
  • The clock stays at the last processed event; it does not move to the end of the period
  • Event times are already guaranteed finite and non-negative by the event schema

Where the remaining tasks plug in

The runner depends only on the small interfaces in simulator/engine/runner.py. Each remaining task provides one of them. tests/conftest.py holds a stand-in for each, which is deleted once the real one is merged.

#98 Scenario configuration

The scenario object needs at least these two attributes. Extra attributes are fine.

class Scenario(Protocol):
    @property
    def duration_s(self) -> float: ...   # length of the simulated period, seconds
    @property
    def master_seed(self) -> int: ...    # non-negative integer

The runner rejects a negative, infinite or non-numeric duration and a negative or non-integer seed before creating any file, but full validation (types, ranges, the hours x replications cap from the Security page) belongs to #98. Stand-in to replace: MinimalScenario in tests/conftest.py.

#9 Seed management and stream separation

The runner calls a stream factory once with the master seed and passes the result to the component factory. It never draws random values itself.

StreamFactory = Callable[[int], object]   # master_seed -> random substreams

The substreams follow Mathematical Foundations section 9: one numpy.random.SeedSequence substream per source (walking times, group sizes, run times, disruptions, loads, prediction errors). Adding numpy means adding an exact pin to requirements.txt. Once #9 is merged, the object type in StreamFactory and ComponentFactory can be replaced by the real streams type. Stand-in to replace: SeedOnlyStreams in tests/conftest.py.

#97 Network model, then US-1.2 and US-1.3

The network, trains and passengers come in as components.

class Component(Protocol):
    def start(self, scheduler: Scheduler) -> None: ...                # schedule first events
    def handle(self, event: Event, scheduler: Scheduler) -> None: ... # react to every event

ComponentFactory = Callable[[Scenario, object], Sequence[Component]]

Every component sees every event, in the order of the list returned by the factory, so a component ignores events it does not care about. TimetableComponent in tests/conftest.py is a minimal example: it schedules each train's arrival in start, and in handle schedules the departure after the dwell. The network itself is read from config/, never from GTFS at runtime (see CLAUDE.md). Stand-ins to replace: TINY_TIMETABLE and TimetableComponent in tests/conftest.py, once the real network can provide an equally small test network.

#10 Run manifest

The runner writes into output_dir, so manifest.json (parameter set, seed, code version, GTFS feed version) can be written next to events.jsonl and summary.json without changing the runner signature. The manifest must not include anything that changes between identical runs (wall clock time, absolute paths), or the determinism check will fail.

#11 Determinism test in CI

test_same_scenario_and_seed_give_byte_identical_output in tests/simulator/engine/test_runner.py already compares two full runs byte for byte. The CI job needs to run pytest (with ruff check . and ruff format --check .) on every push and pull request and fail on any difference.

Changing the event schema

The schema is frozen so other components can rely on it. When a change is necessary:

  1. Add or change the field in events.py using the existing field helpers (_id_field, _seconds_field, _count_field, _text_field) so validation stays automatic
  2. Increase SCHEMA_VERSION by one
  3. Update the expected records in tests/simulator/engine/test_events.py and the sample events
  4. Update the event schema table on this page
  5. Mention the change and the version bump in the pull request

Testing conventions

  • Tests live in tests/, mirroring the source tree
  • Expected values are worked out by hand and written down in a comment or docstring, never computed with the code under test. test_runner.py shows the worked timeline for the tiny timetable
  • Every task adds a test of the main behaviour and at least one edge case (invalid input, boundary, failure)
  • Hypothesis is used for properties that must always hold (processing order never goes backwards, records survive a round trip)
  • Fixtures available from tests/conftest.py: minimal_scenario, build_streams, build_tiny_network
  • No unseeded randomness and no access to Supabase or the network in tests
  • Every source and config file starts with an AI declaration line (Code Quality Rubric section 8): # AI contribution: No substantial AI-generated code, # AI contribution: Below 50% AI-generated or # AI contribution: 50% or more AI-generated. An existing declaration is never lowered
  • Functions stay at or under 50 lines and files at or under 500 lines. Ruff enforces complexity and statement count, but not function length, so length is checked in review

Design decisions

2026-10-17 - Custom heapq event loop instead of SimPy

Context: #8 needs a queue whose tie order is fixed and documented, and US-1.1 needs byte-identical replays. Options considered:

  • SimPy: less code for processes and resources; tie order and the event record format would need extra wrapping, and process-based code makes exact replay harder to reason about
  • Custom queue on Python's heapq: about 160 lines; full control of ordering, the event schema and the clock Decision: custom heapq queue. Rationale: determinism and a frozen event schema are the core of US-1.1; both are simpler to guarantee and test with a small queue the team owns. Consequences: the Architecture and Infrastructure and Tools pages still name SimPy and need updating (see below). Evidence: simulator/engine/event_queue.py, tests/simulator/engine/test_event_queue.py.

2026-10-17 - Per-passenger events for transferring passengers only

Context: #96 defines the passenger events. Transferring passengers number in the low hundreds per train arrival; a peak Metro train carries around a thousand people. Options considered:

  • Group events with a count: compact, but per-passenger walking times and delays are lost
  • Per-passenger events for everyone: complete, but millions of records per run would threaten the 30 s full-day target on the Performance page
  • Per-passenger events for transferring passengers, counts for everyone else Decision: the third option. Rationale: the metrics (per-passenger delay, helped vs harmed) only need individual records for transferring passengers. Everyone else appears in the math only as counts (L, B, A). Consequences: TrainDeparture carries boarded_count, alighted_count and onboard_load. Evidence: simulator/engine/events.py.

2026-10-17 - Scheduler rejects events that break the tie order

Context: a component handling an event could schedule another event at the same time but earlier in the tie order, such as boarding after the train has already left. Options considered:

  • Allow it and process it next: the documented order would silently be broken
  • Reject it with a clear error Decision: reject. Rationale: such an event is always a bug in the component, and catching it early keeps the event log consistent with the tie order. Evidence: test_scheduling_earlier_in_tie_order_at_current_time_is_rejected.

Open documentation updates

These pages still describe choices that changed during US-1.1:

  • Architecture: lines naming SimPy in the high-level diagram, the system layers diagram and the sequence diagram; a row in the Architecture changes table
  • Infrastructure and Tools: the SimPy row
  • Naming Conventions: Python follows PEP 8 (snake_case functions, variables and modules); camelCase applies to TypeScript
  • Testing Plan and Continuous Integration: Python tests use pytest and Hypothesis, not unittest

AI acknowledgement: drafted with Claude, reviewed and edited by Priyesh Patel.

Clone this wiki locally