Testing

Every check is a standalone script in scripts/ with a docstring explaining what it covers. No CI required: each is offline and hermetic unless noted.

Commands

uv run test         # fast offline suite: quick wins, cost estimate, run history, setup wizard
uv run test chat    # one specific group by name
uv run test all     # every group, including the browser smoke test

Each check runs in its own subprocess because several set environment variables (like a temp DB_PATH) at import time.

The catalog

Script	What it verifies
check_quick_wins.py	Indicators, forecast normalizers (TimeGPT + TSFM/Chronos-2), portfolio accounting + migration, risk gate
check_risk.py	The risk gate, plus a full mock-mode analysis end-to-end (needs network for market data)
check_sentiment.py	The sentiment analyst: thin-volume brake, provenance, hermetic e2e
check_streaming.py	The live reasoning stream: sink, caps, transient events, e2e
check_chat.py	The manager chat: dossier, validation, mock + history fallback
check_decision_brief.py	The decision brief: staleness by outlook, coverage, gate change
check_ui.py	The shared UI layer: single-source helpers, nav markers, no manual ?v= stamps, token resolution, module import/export consistency
check_run_controls.py	Run controls: cancel, rerun, partial results
check_lifecycle.py	Concurrency and run lifecycle (P2.8): autopilot session lock, serialized submit guards (5 concurrent submits, 1 POST), BUY position re-check, run deadline, hung in-flight cleanup, SSE replay snapshot, finished-run eviction, corrupt session rows
check_incognito.py	Incognito mode: no portfolio fetch, no memory read/write, nothing persisted, cache isolation, unattributed API run, chat, discard (including mid-run), pruning
check_calibration.py	Confidence calibration: buckets, sample gate, Brier, API
check_cost.py	The per-run model cost estimate: price table, math, unpriced floors, persistence
check_run_history.py	Durable run history: save/restore, owner scoping, indexes
check_run_speed.py	Run pipeline speed (P2.14): forecast overlap, news-fallback gating, market-context cache, provider timeout caps, stage timings on a full offline replay
check_setup_wizard.py	The setup wizard: preset, mock, and merge paths; key never leaks into output
check_llm_reliability.py	Per-role LLMs, failure classification, retry paths
check_analysis_cache.py	The one-hour analysis cache
check_changes.py	The What-changed diff engine: deterministic lines, attach, previous-call lookup; price-history endpoint
check_accuracy.py	The accuracy report engine: verdict rule, grading, aggregates, pending counts, outcome persistence, script rendering
check_watchlist.py	Experimental watchlist (env-gated on): baseline vs newest call, change detection, owner isolation, API
check_search_export.py	Runs/portfolio filtering, print brief, JSON and CSV exports
check_paper_trading.py	Alpaca paper adapter: connection, submission guards, idempotency, performance (fake client, zero broker calls)
check_memory.py	Decision memory: recording, grading, reflections
check_automation.py	Autopilot sessions: gate ladder, discovery + held merge, decision-to-order gating, timeouts, skip recording (fake broker, discovery, and run store)
check_broker_sync.py	Broker reconciliation (P2.3): startup sweep settles pending/unknown rows, partial fills sync, drift check + /api/broker/sync, sweep-failure recording, classified read retries (fake client, zero broker calls)
check_settings.py	Settings store and API: masking, keychain round-trip, clamps and choices, blank-secret-means-unchanged, reset, keychain fallback, drive-by guard (fake keyring)
check_manager_reasoning.py	Per-role reasoning-effort overrides reach the actual provider requests
check_trust_ui.py	Provenance, backward compatibility, and trust-first UI hooks
check_backtest.py	The backtest harness: replay, as-filed SEC EDGAR fundamentals, CLI depth tiers, holdout split, paired mode, memory isolation
backtest_smoke.py	Regenerates the 3-ticker baseline backtest report
check_browser_smoke.py	Automated browser smoke test: keyboard journey, focus, live regions, reflow (system Edge/Chrome via the dev-only playwright dependency; skips with exit 0 when unavailable)
smoke_llm.py	One-shot check that the configured LLM endpoint answers

Other developer commands

uv run python -m app.calibration   # regenerate the confidence calibration report, no LLM calls
uv run python scripts/smoke_llm.py # does the configured LLM endpoint answer?