title	WaitCost
emoji	⏳
colorFrom	indigo
colorTo	gray
sdk	docker
app_port	7860
pinned	false

WaitCost — The Cost of Doing Nothing (Agents + Simulator)

USAII Global AI Hackathon 2026 · Graduate Track · Challenge 6, Direction A.

A multi-agent policy-analyst system on top of a system-dynamics simulator. Ask a plain-English question about delaying homelessness intervention; the agent classifies the question, runs calibrated simulations with uncertainty, explains the drivers, picks the right chart, and writes a decision brief — stopping short of any binding allocation, which stays with a human.

Calibrated for CA-600 Los Angeles on real public data; the same trained model scores 17 US cities.

How it works — the data flow (read this first)

The one rule: the deterministic engine owns every number. The LLM only routes the question and writes prose; a regex number-guard rejects any figure the engine didn't compute. So the agent can sound human while staying as trustworthy as a spreadsheet — every headline is traceable to a simulation output.

A plain-English question flows left-to-right through eight stages. Blue = the LLM (Claude Sonnet 4.6) proposes; green = deterministic code decides. The guardrails (safety rail, data-support check, Tier-2 gate, number-guard, evaluator) sit downstream of the brain and can never be bypassed.

The whole system on one line: five public sources → verified data files → ML inflow model + Monte-Carlo simulator → FastAPI → 5 agents → React. Blue = the AI / learned components.

The diagrams below render the same flow interactively on GitHub.

flowchart TD
    Q["💬 Plain-English question<br/><i>'What if we wait 3 years on a $15M program?'</i>"]
    Q --> ORCH["🚪 Analyst agent — single front door /ask<br/>agent/orchestrator.py"]

    subgraph ROUTE["1 · ROUTE — LLM proposes, rules veto"]
      direction TB
      P["🧠 Claude planner<br/>semantic intent → list of typed tool calls"]
      RAIL{"🛡️ Safety rail<br/>individual / sub-CoC?"}
      NORM["⚙️ Deterministic normalizer<br/>re-parses every budget &amp; delay · no silent drop"]
      P --> RAIL
      RAIL -- "yes" --> OOS["out_of_scope — forced"]
      RAIL -- "no" --> NORM
    end
    ORCH --> P

    NORM --> SUPPORT{"2 · Enough data?<br/>check_data_support()"}
    SUPPORT -- "too thin / synthetic" --> DECLINE["🙅 Decline — honestly"]
    SUPPORT -- "ok" --> KIND{"3 · Kind of question?"}

    KIND -- "concept / data lookup" --> RET["📚 Retrieval — cited, no engine<br/>agent/retrieval.py"]
    KIND -- "city situation / plan" --> CB["🏙️ City Brief agent<br/>agent/city_brief.py"]
    KIND -- "quantitative" --> ENGINE

    subgraph ENGINE["4 · DETERMINISTIC ENGINE — owns every number"]
      direction TB
      MC["🎲 Monte-Carlo scenarios<br/>model/simulate · montecarlo · cost"]
      MET["📐 Metrics: cost-of-waiting · break-even · ROI · sensitivity"]
      BT["✅ Backtest + effect band — uncertainty"]
      MC --> MET --> BT
    end

    ENGINE --> DEC["🧭 5 · Decision agent<br/>act-now / wait + confidence on direction"]
    RET --> GUARD
    CB --> GUARD
    DEC --> TIER{"6 · Tier-2 gate<br/>binding allocation?"}
    TIER -- "yes" --> HUMAN["✋ Stop → human approval"]
    TIER -- "no" --> GUARD

    GUARD["🔒 7 · Number-guard<br/>reject any figure the engine didn't compute"]
    GUARD --> EVAL["🔎 8 · Evaluator agent · 6 checks<br/>grounding · scope · params · confidence · chart↔text · Q-match"]
    EVAL -- "hard fail" --> RETRY["🔁 1-retry self-correct → else decline"]
    EVAL -- "pass / warn" --> OUT

    OUT["📄 Decision brief + matching chart<br/><b>every number traceable to a sim output</b>"]

    classDef llm fill:#dbeafe,stroke:#2563eb,color:#1e3a5f;
    classDef det fill:#dcfce7,stroke:#16a34a,color:#14532d;
    classDef guard fill:#fef9c3,stroke:#ca8a04,color:#713f12;
    class P,DEC llm;
    class ORCH,NORM,MC,MET,BT,RET,CB,OUT det;
    class RAIL,SUPPORT,TIER,GUARD,EVAL,HUMAN,OOS,DECLINE,RETRY guard;

Loading

Why a judge can trust the numbers — the guard, isolated

The single mechanism that makes an LLM-written brief as reliable as a spreadsheet: Claude may only phrase facts the engine already computed. Anything else is dropped.

flowchart LR
    ENG["⚙️ Deterministic engine<br/>the ONLY source of numbers"] -- "computed figures" --> FACTS[("📦 Engine facts")]
    FACTS --> CLAUDE["🧠 Claude writes the brief<br/>using only these facts"]
    CLAUDE --> NG{"🔒 Number-guard<br/>every figure ∈ engine facts?"}
    NG -- "yes ✅" --> SHOW["Ship Claude's brief"]
    NG -- "no ❌" --> FALLBACK["Drop it → deterministic brief"]

    classDef llm fill:#dbeafe,stroke:#2563eb,color:#1e3a5f;
    classDef det fill:#dcfce7,stroke:#16a34a,color:#14532d;
    classDef guard fill:#fef9c3,stroke:#ca8a04,color:#713f12;
    class ENG,SHOW,FALLBACK det;
    class CLAUDE llm;
    class NG guard;

Loading

One engine, every surface (nothing else does math)

The same Python skills back the CLI, the API, the React app, the Streamlit app, and the installable Agent Skill — so they can never disagree on a number.

flowchart TD
    subgraph CORE["🐍 Engine (Python) — the asset"]
      SK["5 agents · 16 capabilities · 16 skills · 18 charts<br/>model/ · analysis/ · agent/ · data/"]
    end
    SK --- API["⚡ FastAPI bridge<br/>api/main.py"]
    SK --- CLI["⌨️ run_demo.py / eval<br/>62 pytest"]
    SK --- SKILL["🧩 Agent Skill<br/>skills/waitcost/"]
    API -- "JSON / HTTP" --> REACT["⚛️ React + TypeScript<br/>frontend/"]
    API --- ST["📊 Streamlit<br/>app/dashboard.py"]

    classDef det fill:#dcfce7,stroke:#16a34a,color:#14532d;
    classDef ui fill:#ede9fe,stroke:#7c3aed,color:#4c1d95;
    class SK,API,CLI,SKILL det;
    class REACT,ST ui;

Loading

Quickstart

pip install -r requirements.txt
export ANTHROPIC_API_KEY=sk-...                                  # Claude planner + narrator (default)
python run_demo.py "What if we wait 3 years on a $15M program?"   # CLI agent
pytest -q                                                         # deterministic test suite
python eval/routing_benchmark.py                                  # 108-Q routing accuracy report
python eval/chart_coverage.py                                     # every question -> a buildable chart
streamlit run app/dashboard.py                                    # interactive dashboard (5 tabs)
uvicorn api.main:app --reload --port 8000                         # JSON API -> http://localhost:8000/docs

Claude Sonnet 4.6 is the default brain. It plans and narrates via the Anthropic API — but no PII is even possible (the system holds only public, aggregate HUD/Census data and refuses individual-level questions), so nothing sensitive can ever be sent. The planner runs in mode WAITCOST_PLANNER ∈:

• auto (default) — Claude when a key is present; if there's no key or the API errors it falls back to the rule-based planner silently (never stalls, never crashes). The result carries planner (claude / rule_based_fallback) and brief_author (claude / deterministic) so you can see which brain ran.
• rule — pure deterministic, never touches the network (the air-gapped, can't-fail demo mode).
• claude — force Claude (errors loudly if the API is unreachable — to prove the path).

export WAITCOST_PLANNER=rule    # guaranteed-reproducible, network-free demo
export WAITCOST_AGENT=toolloop  # (opt-in) Claude orchestrates the engine over multiple tool calls

Claude (a) routes the question into a plan and (b) writes the one-page brief — but every number stays the engine's: a regex number-guard rejects any figure the model emits that the engine didn't compute, falling back to the deterministic brief. The model never invents or alters a headline figure.

Routing — a layered router (LLM proposes, rules veto)

Understanding which question a user is asking is the job we put the model on:

1. Claude (primary) — in auto/claude mode Claude classifies the intent semantically from the registry's when_to_use + contrastive few-shots, so it disambiguates things keywords can't ("the ROI of the city's plan" → roi, not care_plan).
2. Deterministic safety rail (authoritative) — individual / sub-CoC profiling is forced to out_of_scope no matter what the model says. The "never profile individuals" promise cannot depend on the LLM (planner._apply_safety_rail).
3. Rule fallback (reproducible) — a registry-ordered regex walk when no key is set, and the can't-fail demo mode. An unknown LLM label defers to it, not to a blind default.

Routing quality is measured, not assumed: a ~100-question, judge-style benchmark (eval/routing_cases.py) scores accuracy with a regression gate — 100% on the clear+paraphrase tier, 99% overall in rule mode (python eval/routing_benchmark.py). And every question type yields a render-ready graphic — eval/chart_coverage.py proves 108/108 questions build a valid chart.

Five agents

How one question becomes an answer: Claude Sonnet 4.6 routes & narrates; the deterministic engine owns every number; guardrails + the Evaluator (the 5th agent) verify before the user sees it — falling back offline to a deterministic rule planner.

• Analyst agent (agent/orchestrator.py) — the loop: classify the question → call the right tools (the "sandbox" of deterministic Python) → narrate → log to MEMORY.md. Autonomy is bounded by Action Tiers: Tier 0–1 automatic; Tier 2 (recommending an allocation) needs human approval. Declines out-of-scope or thin-data questions instead of guessing.
• Visualization agent (analysis/viz.py) — picks the right decision chart for the question and builds a render-ready spec from real engine output (18 charts).
• City Brief agent (agent/city_brief.py) — answers qualitative questions ("what's the situation in Seattle?", "what is San Diego's plan?") from a curated, cited source registry (data/city_sources.json) + the engine's own indicators. Numbers come only from the engine, narrative + strategy only from cited sources; every brief is labelled "general context — not the calibrated cost model." Offline by default; an opt-in WAITCOST_ONLINE=1 flag can refresh from the plan URL.
• Decision agent (agent/decision.py) — turns the simulator's raw scenarios into a plain-English recommendation a non-technical director can act on: the call (act now / wait) with a confidence on the direction (separate from the dollar magnitude), and the framing people misread — the multi-billion 10-year baseline is mostly unavoidable; the timing decision only moves the smaller cost-of-waiting slice. It invents no figure (number-guarded); it rides along in every analytic answer and leads the decision brief.
• Evaluator agent (agent/evaluator.py) — the post-answer critic that checks every answer before the user sees it, across six dimensions (grounding, scope, parameter fidelity, data confidence, chart↔text consistency, and an LLM question-match judge). It can only flag / annotate / repair / decline — never alter an engine number. On a hard failure it triggers a bounded 1-retry self-correction, then declines rather than show a confident wrong answer; uncertainty becomes a warn (a caveat), not a wrongful refusal. Its verdict ships to the UI as a per-dimension Response Check. With confidence-gated routing, when the LLM and rule routers disagree at low confidence the system asks instead of guessing (clarify). See EVALUATOR_AND_GUARDRAILS.md.

The orchestrator is the single front door (/ask): quantitative questions run the simulator; city_situation / care_plan questions are handed off to the City Brief agent.

What the user actually sees: the Evaluator's six-dimension Response Check ships with every answer (grounding · scope · params · confidence · chart↔text · question-match).

Registry of capabilities is explicit (agent/tools.py, exposed at /tools): 5 agents · 16 capabilities · 16 skills · 18 charts.

The planner is a tool-calling router: one question becomes a list of typed calls (plan.calls), so compound asks fan out (e.g. "$1M and $15M" → two cost-of-waiting calls) and a deterministic normalizer guarantees no parsed value is ever silently dropped. See ARCHITECTURE.md.

One capability registry (Agent-Skills-style)

Every answerable question type is declared once in agent/capabilities/specs.py (its when-to-use, regex triggers, tier, handler, and chart). The structures that used to be hand-maintained in five places — planner.INTENTS, classify_intent, the Claude _PLAN_SYS planner prompt, the orchestrator's intent dispatch, tools.CAPABILITIES, the Anthropic tool schemas (caps.anthropic_tools()), and viz.INTENT_CHART — are now derived from that registry, so they can't drift. Adding an analysis means one registry entry + one handler in agent/handlers.py.

Run it as an installable Agent Skill

The engine is also packaged as an Anthropic Agent Skill so Claude Code / the Skills API can drive it directly — offline, deterministic, and with the Tier-2 human-approval gate and number-guard preserved in code:

python skills/waitcost/scripts/waitcost_cli.py ask "What if we wait 3 years on a $15M program?" --coc CA-600

See skills/waitcost/SKILL.md. The CLI is a thin wrapper over api/payloads.py, so the skill, the API, and the agent can never disagree on a number.

What it answers (multi-question intents)

📋 QUESTIONS.md is the full, copy-pasteable list of everything you can ask (auto-generated from the registry: python scripts/gen_questions.py).

cost-of-waiting (incl. multi-budget sweeps) · break-even · savings-vs-nothing · outcome-at-horizon · compare-budgets · compare-mix · sensitivity · ROI / benefit–cost · cost-per-person + people-helped · regional (multi-city ranking) · uncertainty / explain-this-number · city-context · equity (racial disparities, population-level — never individual) · concept_qa (define a concept, cited, no engine) · data_lookup (data source / vintage / methodology, cited) · greeting/meta → friendly guide · unmapped in-scope → clarify · out-of-scope → declines. Each answer's recommended chart shows the same figure the text states (numbers are identical, not re-sampled).

The AI inside

• Learned inflow model (model/inflow_model.py): Ridge on real Census ACS → HUD PIT across 17 cities, leave-one-CoC-out R²≈0.36, exact additive SHAP (housing cost is the top driver). Sets the simulator's inflow + its uncertainty.
• Face-validity backtest (model/backtest.py): seed real 2023 PIT, reproduce observed 2024 within band (~4% error for CA-600).
• Claude Sonnet 4.6 planner + narrator for natural-language understanding, number-guarded prose, and (opt-in WAITCOST_AGENT=toolloop) a real tool-use loop that orchestrates the engine over multiple steps — with a deterministic rule fallback so the demo never depends on the network.

Honest by design: the inflow model is scored on a city it never trained on (leave-one-CoC-out R²≈0.36), explained with exact SHAP, and backtested against the real 2024 count (~4% off, inside the band).

The model evidence behind those claims (regenerated by the notebook in notebooks/):

What the engine learned	Does the model hold up?
SHAP attribution — exact additive contributions; housing cost dominates inflow.	Backtest — predicted vs. observed 2024 PIT, ~4% error for CA-600.
Leave-one-CoC-out fit — honest out-of-sample R²≈0.36 across 17 cities.	Decision charts — the visualization agent renders the same figures the brief states.

Data — all real, all sourced (see data/SOURCES.md)

• Homeless counts: HUD 2024 PIT CoC Population & Subpopulation reports.
• Economic signals: US Census ACS 2024 1-yr.
• Flow rates (CA-600): HUD System Performance Measures FY2023.
• Per-person costs: Economic Roundtable "Where We Sleep" (2024$).

The ACS economic features are reproducible from the official Census API with one command (CENSUS_API_KEY=… python scripts/fetch_acs.py) and independently verified: re-fetching from the live API matched the panel exactly — 0 of 119 values differed by ≥5%. See data_sources/METHODOLOGY.md.

How the data is sourced & coordinated

Four public sources, joined on HUD's CoC region code, then turned into dollars — every headline figure traces back to one of these inputs. (Diagrams live in data_sources/.)

Provenance with real LA (CA-600) figures: four public sources → anchored on the CoC code → joined one row per city → calibrated → dollars. Green = pulled live from the Census API and verified.

The six steps that assemble coc_panel.csv — the 17-city panel of HUD counts + Census economics.

The same pipeline at file·function granularity — which module owns each step from raw source to served answer.

Layout

config/params.yaml      calibrated CA-600 assumptions (source + confidence per value)
model/                  simulate · montecarlo · cost · interventions · inflow_model · backtest · coc_registry · states
analysis/               metrics (cost-of-waiting, break-even, compare, sensitivity) · viz (charts) · equity
agent/                  skills (tools) · planner (Claude + rule fallback) · llm (Anthropic provider seam) · orchestrator (loop) · tools (registry)
api/                    FastAPI bridge: /ask · /ask/stream (SSE) · /brief/export (PDF/Word) · /compare-cities · …
agent/retrieval.py      concept_qa + data_lookup (cited, offline, no engine)
eval/                   rubric.json (weighted criteria) + verifier.py (62 tests total)
scripts/gen_questions.py  regenerates QUESTIONS.md from the capability registry
app/dashboard.py        Streamlit UI (Ask · Explore · Visualize · Where's the AI · Governance)
data/                   coc_panel.csv (17 cities) · equity_*.csv · SOURCES.md
scripts/                fetch_acs.py (reproducible ACS pull from the Census API) · train_inflow.py · backtest.py
run_demo.py             end-to-end CLI entry point

Scope & honesty

• Real & calibrated (CA-600): initial homeless counts (HUD PIT), per-person costs (Economic Roundtable), key flow rates + inflow (HUD SPM), the ACS→PIT learned model. Backtested against observed 2024.
• Documented priors: remaining transition rates (pending fuller HUD SPM fit), at_risk/housed_stable (ACS proxies), intervention effect sizes (the headline is reported as a band over ±50% of these).
• Other 16 cities: real PIT + ACS-driven inflow; flow rates & costs are CA-600 priors (illustrative until locally calibrated).
• This tool informs a budget-timing tradeoff. It does not decide allocations or forecast individuals, surfaces equity at the population level only, and all outputs are ranges, not predictions.

Deploy

Two ready-to-go paths (both run the Claude brain when ANTHROPIC_API_KEY is set, and fall back silently to the deterministic rule planner otherwise):

• Vercel (full-stack: React static + FastAPI Python serverless, one URL) — VERCEL_DEPLOY.md. Wired via vercel.json / api/index.py; needs the Pro plan for the Claude-live function timeout (or rule mode on Hobby).
• Hugging Face Spaces (single Docker container, FastAPI serves the app + API on port 7860) — DEPLOY_HF.md. No function-timeout limit. The root Dockerfile builds it; set ANTHROPIC_API_KEY as a Space secret.