When one train runs late and ninety minutes later six trains are backed up, the honest answer from a control room today is "train 12658 was late" — that's not an explanation, it's the last domino. Signal Failure names the upstream decision that turned a routine five-minute delay into a six-train pileup — while the cascade is still forming, and in time for the system to act on its own diagnosis.
Built for the Google DeepMind Bangalore Hackathon — Problem Statement 2: Autonomous Orchestration with Managed Agents (iAPI). One system, two faces: a control-room screen for the person watching the whole line, and a phone (Rail Buddy) for the person just trying to catch a train.
- The problem
- What it does
- The novel architecture — live structural attribution
- The autonomous agent swarm
- Dual-mode: Connected (Gemini) + Offline edge (Gemma)
- Feature tour
- System design & data flow
- Technical stack
- Repository layout
- Running locally
- Configuration
- API reference
- Testing
- What is real vs. simulated (honesty note)
- Novelty — stated honestly
- Roadmap
- Credits & sources
Indian Railways delay cascades are a universal, lived experience: one train runs late, platforms fall into conflict, a crew runs out of duty hours mid-route, and passengers on several trains have no idea why. The controller has no way to know which upstream decision turned a small delay into a pileup. On the passenger side it's worse: a board that says "delayed," no reason, no action taken on their behalf.
Any credible autonomous railway-operations system is a multi-agent system, and multi-agent systems fail in a specific, well-documented way: the agent whose output is visibly wrong is almost never the one at fault. The real error is introduced upstream by an agent whose decision looked locally reasonable but was subtly wrong given context nobody downstream could see. That error is inherited, restated, and compounded until it surfaces far from its cause.
This is an active research problem. Zhang et al. (ICML 2025, "Which Agent Causes Task Failures and When?") introduced automated failure attribution and the Who&When dataset; their best method reached 53.5% accuracy naming the responsible agent and only 14.2% pinpointing the failure step — barely above chance.
The core reframe this project is built on: don't read what agents said — read how the shared state structurally shifted at each handoff. Failure attribution becomes tractable the moment you ask "how far did this handoff diverge from a correct handoff" instead of trying to semantically interpret a transcript.
- Runs an autonomous multi-agent section-operations swarm — five agents plan, delegate, and execute real scheduling/dispatch decisions (platform allocation, signalling, crew, passenger rebooking, turnout integrity) with no human approval loop.
- Instruments every agent-to-agent handoff and runs a live structural-attribution engine that, while an incident is still forming, identifies the exact decision that is the structural origin of a developing cascade.
- Surfaces that to two very different humans:
- Controller — a calm-by-default control-room screen with a live map, a spoken Black Box narrator, a natural-language command console (type or speak), and direct authority (freeze time, hold a train, override, replay the counterfactual).
- Passenger (Rail Buddy) — told the truth, specifically, whenever something changes, and left alone the rest of the day. Spoken, multilingual, and ahead of the problem.
- Uses Google models throughout: Gemini for agent reasoning, the Black Box narration, the command parser and live Q&A; Gemini TTS for voice; Gemini for translation; and Gemma for the offline edge fallback.
- Runs fully offline with zero API keys (deterministic agent policies + templated narration). A Gemini key lights up the LLM reasoning and voice; the system degrades gracefully at every layer.
The attribution engine is the research-grade core. It never reads agent "explanations" — it measures how the shared state slice (platform table, crew buffers, delay vector) structurally shifted at each handoff, against a counterfactual "clean" run.
The section simulator is fully deterministic given a fault-injection point. That lets us compute, offline, a clean counterfactual trajectory — what a correct run would have looked like — for comparison. The flagship scenario: a crew-handover conflict at Vadodara where platform-allocation greedily keeps train 12658 on Platform 3 (locally reasonable, but it ignores the 14:24 crew handover buffer), turning a routine reassignment into a three-train cascade. The correct call was to hold it to Platform 6.
At every handoff we snapshot the state slice before and after the agent's decision, and score it against the clean run's slice at the same turn:
| Signal | Definition | What it captures |
|---|---|---|
| divergence (present) | distance(faulty_out, clean_out) |
Cumulative drift present after the handoff. Downstream agents that merely inherited the error keep this high. |
| introduced (new) | distance(out) − distance(in), clamped ≥ 0 |
The new drift this handoff added. Isolates the agent that caused the shift from the ones that only restated it. |
Ranking candidates by introduced divergence is what makes attribution specific instead of a false-confident scapegoat — it is the GraphTracer-style information-flow signal (see §15). In the flagship scenario Platform Allocation, turn 1 gets introduced ≈ 0.45 while every downstream agent gets ≈ 0.
The state slice is built purely from the committed platform decisions (crew buffers and the delay vector are derived from which platform the anchor is on), so the faulty and clean runs are byte-identical until an agent actually commits a different platform — which is precisely the handoff the divergence should localise.
A maturity ramp scales the score by how far the committed decisions have physically "set"
(idle → 0.1, watching → 0.55, attention → 0.9, problem/postmortem → 1.0). This is what
lets the same anomaly surface live — flagged at the attention phase, before the
cascade is visible (predictive = true) — then again fully post-hoc. Attribution as an
early-warning system, not a forensic tool.
- Backward walk — the attributed origin is the earliest turn whose cumulative divergence
crosses the threshold (
0.30). - Calibrated distribution — a softmax over introduced divergence gives a probability per candidate agent/turn, not a single false-confident answer (directly engaging why the 14.2% step-accuracy baseline is so low).
- Minimal sufficient cause — the smallest set of handoffs whose drift, if reverted, drops the total below threshold — honest about "combinations of individually-fine decisions" when more than one applies.
When the controller overrides, the engine forces the correct platform, the cascade is prevented, and the divergence collapses to zero — attribution stops being an external observer and becomes part of how the system governs itself.
tick phase maturity detected predictive origin turn conf
0 idle 0.10 false false — — 0.00
10 watching 0.55 false false — — 0.00
20 attention 0.90 true TRUE platform-allocation 1 1.00 ← caught early
30 problem 1.00 true false platform-allocation 1 1.00
45 postmortem 1.00 true false platform-allocation 1 1.00
Five agents, each owning a narrow, real tool surface against the shared section state. Every inter-agent call is a logged handoff — the payload each agent received and produced — which is the attribution engine's raw dataset.
| # | Agent | Owns | Tool surface |
|---|---|---|---|
| 0 | Turnout-integrity | Forced speed restrictions at turnouts (points & crossings) | flag_turnout_restriction(turnout_id, speed_kmh, duration) |
| 1 | Platform-allocation | Assigning platforms to arriving trains | assign_platform(train_id, station_id) |
| 2 | Signal-priority | Block/signal sequencing given the committed platform plan | set_block_sequence(...) |
| 3 | Crew-scheduling | Duty-hour & crew-availability constraints | reassign_crew(train_id, crew_id) |
| 4 | Passenger-rebooking | Downstream passenger impact once delay crosses threshold | rebook_passenger(pnr, new_train_id) |
Why turnouts? Railway Board data for FY2025–26 shows ~65% of track-related derailments occurred at turnouts — a real, citable statistic. A forced 15 km/h restriction at a turnout isn't a mistake, it's a mandatory safety response; but if the scheduling agents around it don't account for how it reshapes the section's timing, that correct local decision becomes the upstream cause of a cascade. Having both "an agent was wrong" (crew-handover) and "an agent was right but the consequence still cascaded" (turnout) in one demo proves the mechanism generalises across genuinely different structural signatures.
Deterministic first, LLM on top. Each agent has a deterministic policy (the "dumb pipeline"
the build order asks for) and an optional Gemini path. With AGENTS_USE_LLM=true, the agent
asks Gemini to phrase its rationale — genuine Google-model reasoning over the same committed
decision, so the live cascade and the attribution result stay reproducible either way.
The orchestrator runs the swarm twice per tick: once with the correct policies (the clean counterfactual) and once with the faulty policy (what actually happened), threading state slices agent → agent and logging every handoff.
A genuinely "100% offline" system cannot simultaneously be "built on hosted managed agents" — so the honest, and stronger, story is a dual-mode system, resilient to the exact real-world condition (monsoon-driven regional collapse) that makes turnout cascades most dangerous.
- Mode 1 — Connected (primary). Agents reason via Gemini, orchestration and handoffs are cloud-mediated. This is the fully-built core.
- Mode 2 — Offline edge (Gemma). Toggle "regional network down" and a lightweight local agent
running Gemma (via a local Ollama server, no cloud) takes over the same block sector's core
loop — a full sense → decide → act → check loop, not a single-turn chatbot:
- Sense local track-circuit + turnout telemetry;
- Decide with on-device Gemma (deterministic local policy fallback);
- Act — re-sequence local signals, reroute freight to loop lines;
- Check — flag anomalies (e.g. a crew-duty expiry it couldn't see), shift to a fallback, narrate the root cause via local TTS.
- Live section schematic — stations left-to-right, trains as colour-coded markers, a
five-state machine:
Nominal → Watching → Needs attention → Active problem → Resolved. - The Black Box narrator — grounded, mostly-templated spoken narration for each phase, fed through Gemini TTS (multi-voice: Kore, Puck, Charon, Fenrir…). Postmortem names the exact agent and turn.
- ⌘ Command Console (conversational control room) — type or speak (Web Speech mic) a
plain-language instruction; Gemini parses the intent and the engine executes it:
- "freeze the section" → pauses time · "hold the Duronto" → holds train 22209 · "add a 12-minute delay to 12009" → injects a fault · "override" → kills the cascade · "how sure are you?" → spoken LLM answer from the live attribution · "announce platform 6 in Hindi" → translates + speaks.
- Every reply is spoken back through Gemini TTS. Voice + narration-language pickers included.
- "Brief me" — freezes the section and Gemini generates a live spoken situational report from the actual frozen state (not a script).
- Direct authority — freeze/resume, hold/release any train, manual override, counterfactual toggle (real cascade in red vs. what should have happened in green).
- Judge-injected live fault — hand a judge control: pick any train, add any delay, and watch the system catch and attribute its own cascade unscripted.
- Incident log — every flagged event, its outcome, the attributed agent/turn, and whether the controller let it run or intervened.
- Leave-Now nudge — a spoken "Leave now for the 12658" at the actually-correct moment, driven by trajectory confidence, not a countdown. Spoken in the reader's language.
- "Already handled" rebooking — the first notification about a problem is the fix ("moved to Platform 6, departing 14:34"), spoken, with a Why? explanation and an English / हिंदी / मराठी / ગુજરાતી / தமிழ் / বাংলা switcher that translates + speaks live.
- Journey Guardian (solves: missing your stop, night-time safety, elderly/solo travel):
- Wake-Me-There — a delay-adjusted station-by-station progress rail and a spoken, escalating wake-up as your stop nears.
- Autonomous buddy watch — Boarded → Halfway → Nearing your stop → Arrived safely, marked "sent" to a trusted contact with zero effort.
- "I need help" — Gemini composes a calm, precise, translated alert (train, coach, seat, nearest station, helpline 139 / RPF 182).
- Coach Compass (solves: overcrowding / unreserved boarding of reserved coaches):
- A live crowd heat-map of the whole rake; unreserved overflow visibly spills into the nearest sleeper coaches as the journey progresses.
- Recommends a calmer coach within reach and the exact platform boarding marker, spoken.
- Flag overcrowding to the TTE — Gemini composes a translated report.
A dark, brutalist front door that states the thesis, a capability band, and two doors — and, when the backend is connected, a live status that ticks through the real phases.
┌───────────────────────────────────────────────┐
│ SimulationEngine │
│ (autoplay loop · per-tick recompute · WS fan) │
└───────────────────────────────────────────────┘
simulator ───▶ orchestrator ───▶ attribution ───▶ narrator ───▶ API
(Layer 1) (Layer 2) (Layer 3) (Layer 4) REST + WS
deterministic iAPI-style divergence + Black Box │
graph, tick(), handoff routing backward walk narration + │
inject_fault(), (clean + faulty + calibrated live Q&A ▼
counterfactual runs, full distribution ┌──────────┐
handoff log) │ Frontend │
agents (5, Pydantic AI + Gemini) ────┘ edge (Gemma) ◀───────│ WS/REST │
└──────────┘
Every tick (~1 s) the engine advances the sim, re-runs both agent chains, recomputes
attribution + narration, appends incidents on phase transitions, and broadcasts the full
EngineState to every connected websocket — so the controller map and the passenger app stay
live without polling. Every REST control also triggers a broadcast, so REST and WS never drift.
Backend — Python 3.12 · FastAPI (REST + WebSocket) · Pydantic v2 domain models ·
Pydantic AI with the Google provider (Gemini) for the agent swarm · raw httpx for
Gemini TTS (PCM → WAV) and translation · optional MongoDB (state is in-memory by
default). Everything LLM-related is lazy and guarded — the server starts and the deterministic
pipeline works even without pydantic-ai installed or a key configured.
Frontend — React 19 · react-router · a shared SimProvider context (simStore)
that either drives a self-contained scripted timeline or streams live EngineState over
WebSocket when REACT_APP_BACKEND_URL is set · Web Speech recognition for the mic ·
<audio> playback of Gemini TTS WAVs · a warm/brutalist bespoke design system (Fraunces,
Work Sans, Newsreader, JetBrains Mono).
Google models used
| Purpose | Model | Where |
|---|---|---|
| Agent reasoning / rationale | gemini-3.5-flash |
AGENT_MODEL |
| Black Box narration, live Q&A, command parsing, briefing | gemini-3.5-flash |
NARRATOR_MODEL |
| Text-to-speech (narration + passenger voice) | gemini-3.1-flash-tts-preview |
TTS_MODEL |
| Translation (Hindi + regional) | gemini-3.5-flash |
TRANSLATE_MODEL |
| Offline edge agent | gemma3:4b (Ollama) / gemma-4-* |
GEMMA_MODEL |
railways/
├── backend/
│ ├── server.py # FastAPI entrypoint (uvicorn server:app)
│ ├── requirements.txt
│ ├── .env.example # copy → .env; runs with zero keys
│ ├── README.md # backend-specific deep dive + full API table
│ ├── app/
│ │ ├── config.py # env-driven settings, graceful flags
│ │ ├── scenario.py # seed data (stations, trains, crew, PNR, agents)
│ │ ├── domain.py # Pydantic models (the shared vocabulary)
│ │ ├── simulator.py # deterministic graph, tick, inject_fault, counterfactual
│ │ ├── llm.py # Pydantic AI + Gemini factory (lazy, guarded)
│ │ ├── agents.py # the 5 agents (policy + optional Gemini rationale)
│ │ ├── orchestrator.py # iAPI-style handoff router (clean + faulty runs)
│ │ ├── attribution.py # divergence, introduced, backward walk, distribution, MSC
│ │ ├── narrator.py # Black Box narration + live controller Q&A
│ │ ├── command.py # NL command parser + situational briefing (Gemini)
│ │ ├── voice.py # Gemini TTS (PCM→WAV) + translation + voices
│ │ ├── passenger.py # Journey Guardian safety-message composer
│ │ ├── edge.py # Gemma offline edge loop (Ollama)
│ │ ├── engine.py # SimulationEngine (conductor, autoplay, WS)
│ │ └── api.py # REST + WebSocket routes
│ └── tests/test_attribution.py # locks the research core (5 tests)
├── frontend/
│ ├── .env.example # REACT_APP_BACKEND_URL → connects to backend
│ └── src/
│ ├── pages/ # Landing, Controller, Passenger
│ ├── components/
│ │ ├── controller/ # SectionMap, Sidebar, OpsPanels, CommandConsole, TheFork
│ │ └── passenger/ # JourneyGuardian, CoachCompass, Speak, RippleRadar, …
│ ├── hooks/useSpeech.js # Gemini TTS w/ Web Speech fallback
│ └── lib/
│ ├── simStore.js # shared context; WS-connected or scripted
│ ├── backend.js # REST/WS client + voice prefs
│ ├── journey.js # delay-adjusted progress/ETA
│ └── scenario.js # frontend seed (mirrors backend)
└── signal-failure-spec (1).md # the full design spec
You need two terminals. Everything runs locally; the app works with zero API keys (deterministic policies + templated narration), and a Gemini key lights up the LLM + voice.
Windows note: on this machine
pythonis the Microsoft Store stub — use thepylauncher. Port 8000 was already in use, so we run the backend on 8080.
cd backend
py -m pip install -r requirements.txt # one-time
cp .env.example .env # optional — add your Gemini key
py -m uvicorn server:app --reload --port 8080Check http://localhost:8080/api/health · interactive docs at http://localhost:8080/docs.
cd frontend
# point the UI at the backend:
# .env -> REACT_APP_BACKEND_URL=http://localhost:8080
npm install --legacy-peer-deps # React 19 needs this flag
npm startOpens http://localhost:3000 → / landing, /controller, /passenger.
Prefer yarn?
corepack enable && yarn install && yarn start. If npm hits anajv/ajv-keywordserror, yarn (which honours theresolutionsinpackage.json) is the clean fix.
Driving the demo: the section is a ~60-second timeline. Use the control strip at the
bottom of any page — ▶/❚❚, the scrub slider, ⚡ Judge: Inject fault, and ⟲ Reset. Hit
Reset and watch it play through Nominal → Watching → Attention → Cascade → Resolved.
All via backend/.env (see .env.example). Key switches:
| Variable | Default | Purpose |
|---|---|---|
GEMINI_API_KEY |
(empty) | Enables all Google-model reasoning + voice. Get one at aistudio.google.com/apikey. |
AGENT_MODEL / NARRATOR_MODEL |
gemini-3.5-flash |
Swarm reasoning / narration + Q&A. |
AGENTS_USE_LLM |
false |
Let agents phrase rationale with Gemini (cascade stays deterministic). |
NARRATOR_USE_LLM |
true |
Gemini for narration + live Q&A when a key is present. |
TTS_MODEL / TTS_VOICE |
gemini-3.1-flash-tts-preview / Kore |
Spoken narration + passenger voice. |
TRANSLATE_MODEL |
gemini-3.5-flash |
Text translation (Hindi + regional). |
GEMMA_ENABLED / GEMMA_MODEL / GEMMA_BASE_URL |
false / gemma3:4b / localhost:11434 |
Offline edge mode via Ollama. |
MONGO_URL |
(empty) | Optional persistence; state is in-memory without it. |
SIM_TICK_SECONDS / SIM_MAX_TICKS / SIM_AUTOPLAY |
1.0 / 60 / true |
Simulator pacing. |
Frontend: REACT_APP_BACKEND_URL (unset → self-contained scripted demo; set → live backend).
Base path /api; interactive docs at /docs. The full table (bodies included) lives in
backend/README.md. Highlights:
| Method | Path | Purpose |
|---|---|---|
| GET | /health, /state, /snapshot, /counterfactual, /attribution, /handoffs, /incidents |
State & attribution |
| POST | /sim/play · /pause · /reset · /seek · /inject_fault |
Simulator + judge control |
| POST | /controller/action · /controller/command · /controller/brief · /controller/hold |
Controller authority + NL command |
| POST | /narrator/ask · /narrator/speak · /translate |
Q&A · Gemini TTS · translation |
| GET | /voices · /languages |
Pickable voices / languages |
| GET/POST | /passenger/trip · /passenger/safety |
Passenger view · Journey Guardian |
| POST | /edge/toggle · /edge/cycle |
Gemma offline edge mode |
| WS | /ws |
Live EngineState stream (on connect + every tick) |
cd backend
py -m pytest -q # 5 tests, all offline/deterministictests/test_attribution.py locks the research core: no attribution before the fault matures,
the predictive flag firing at the attention phase, the origin isolated to
platform-allocation (turn 1) with the minimal sufficient cause, a controller override
killing the cascade, and the judge-fault reusing the same machinery.
For integrity in front of a technical panel:
- Real & computed: the deterministic simulator, the agent handoff log, the entire attribution math (divergence, introduced-drift, backward walk, distribution, minimal sufficient cause), the counterfactual, and every Gemini call (agent rationale, narration, command parsing, briefing, TTS audio, translation, safety-message composition).
- Simulated for the demo: the section itself is a scripted 6-station / 5-train scenario (the spec's reproducible spine). Outbound delivery of passenger messages — the "TTE alerted" / "sent to your contact" confirmations — currently compose and display the message but do not dispatch to a real person or channel; wiring them into a live controller "operations inbox" (so a report actually lands on the controller's screen) is the honest next step, not a fake we claim works.
This is deliberately careful so the pitch doesn't overclaim.
Not novel (cited as prior work): structural drift instead of transcript-reading in general (spectrum-analysis methods, GraphTracer's information-dependency graph); counterfactual replay / fault injection as attribution (AgenTracer); recurring-pattern transfer (CORRECT); agent-to-agent trust/anomaly graphs (SentinelAgent).
What is genuinely unclaimed: every method above operates offline, on a static, already-finished trajectory log, after a failure has happened, on text/tool-use trajectories. Nobody in the published literature runs structural attribution live, inside a still-operating autonomous system, fast enough to act on its own diagnosis before a physical consequence locks in, and grounded in a physically-consequential, non-text environment where state is irreversible in a way a codebase or chat log is not (a train given the wrong platform cannot be quietly reverted like a bad commit).
The honest one-line claim: Existing failure attribution for multi-agent systems is forensic and text-only. This project takes a validated structural-attribution approach (closest to GraphTracer) and is the first to run it live, inside a physically-grounded autonomous railway operations system, where attribution has to be fast and specific enough for the system to act on its own diagnosis before the physical consequence locks in.
The real contribution is live deployment + physical grounding + closing the loop into control, not the core attribution algorithm.
- Passenger → live controller inbox — route "I need help" / "Flag overcrowding" reports over the existing WebSocket into a real controller-side panel (honest delivery, stronger demo).
- Live-API voice — true real-time barge-in translation via the Gemini Live API (the current translate/TTS is one-shot REST).
- Judge-fault generality — expose scenario selection so the cascade can be triggered on any train from the UI.
- Handback reconciliation — reconcile a locally-made Gemma edge decision back into the central counterfactual as a second attribution challenge.
- Zhang, S. et al., "Which Agent Causes Task Failures and When?", ICML 2025 — the 53.5% / 14.2% baseline and the Who&When dataset.
- AgenTracer (arXiv:2509.03312) — counterfactual replay + programmed fault injection.
- GraphTracer — information-dependency-graph cross-agent error propagation.
- CORRECT, SentinelAgent — pattern transfer / graph-based root-cause localization.
- Railway Board letter to zones (reported May 2026) — 18 of 28 track-related derailments in FY2025–26 (~65%) at turnouts. CAG Report No. 22 of 2022 — broader derailment-cause context.
Built within the hackathon window: the simulator, agents, attribution engine, and both app surfaces. Prior structural-drift research is disclosed as background the team brought in, not presented as built live.
See backend/README.md for the backend deep-dive and the full API table,
and signal-failure-spec (1).md for the complete design spec.