feat(red-team): zero-friction report dashboard — auto-save + CLI + docs

[Aryansharma28]

Stacks on #340 (paper fidelity, merged into base). Adds the zero-friction report pipeline: auto-save via pytest plugin, scenario redteam-report CLI, TS parity, and a docs page.

User flow (before → after)

Before — three friction points in every red-team test:

from scenario.report import save_redteam_report   # user must know this exists
result = await scenario.run(...)
save_redteam_report(result, red_team=rt, test_name="...")  # and remember to call
# then: streamlit run <long-path-to-app.py> -- --batch-dir <longer-path>

After — three commands, zero boilerplate in test code:

pip install 'langwatch-scenario[report]'
pytest path/to/tests.py        # reports save automatically
scenario redteam-report        # auto-discovers latest batch

End-to-end data flow

 TEST RUNS                    AUTO-SAVE                        DASHBOARD
┌─────────────┐              ┌────────────────┐              ┌───────────┐
│  pytest →   │              │ pytest plugin  │              │ scenario  │
│  scenario   │──wraps──▶    │  wraps         │              │ redteam-  │
│  .run(...)  │              │ ScenarioExec   │              │ report    │
└─────────────┘              │  .run()        │              └─────┬─────┘
                             └────────┬───────┘                    │
                                      │                            │
                                      ▼                            ▼
                             ┌────────────────┐              ┌───────────┐
                             │ save_redteam_  │              │ _find_    │
                             │ report():      │              │ batch_    │
                             │  - LLM analyze │              │ dir()     │
                             │  - write JSON  │              └─────┬─────┘
                             └────────┬───────┘                    │
                                      ▼                            ▼
                             ┌─────────────────────────────────────────┐
                             │ ./redteam-reports/<ts>/                 │
                             │   1776_test1_goat.json                  │
                             │   1776_test2_crescendo.json             │
                             │   _aggregated_fixes.json (on-demand)    │
                             └─────────────┬───────────────────────────┘
                                           │
                                           ▼
                             ┌─────────────────────────┐
                             │ streamlit app.py        │
                             │  Overview | Findings    │
                             │  | Prioritized Fixes    │
                             └─────────────────────────┘

What's in this PR

M1 — Auto-save via pytest plugin

• python/scenario/pytest_plugin.py::_auto_save_redteam_report hooks the existing auto_reporting_run wrapper.
• Detects isinstance(agent, RedTeamAgent) in the agents list — no marker/decorator needed.
• Env vars: SCENARIO_REDTEAM_REPORT=0 disables, ..._REPORT_DIR=path overrides the batch root.
• Errors swallowed with warning — reporting failures never break tests.

M2 — scenario redteam-report CLI

• New python/scenario/cli.py: _find_batch_dir() picks the latest timestamped dir via lexicographic sort.
• setup.py + pyproject.toml register the console script and [report] extras (streamlit, plotly, pandas).
• Flags: --latest N, --batch <ts>, --dir <path>, --port, --no-browser.

M3 — TypeScript auto-save parity

• New javascript/src/red-team-report.ts — same JSON shape as Python, duck-typed isRedTeamAgent().
• runner/run.ts calls it after execution.execute() when a RedTeamAgent is present.
• Skips the LLM-based severity/suggestion pass at save time (dashboard's aggregator fills on demand) to keep TS test runs fast.

M4 — Docs page

• docs/docs/pages/advanced/red-teaming/report.mdx — quick start, dashboard viewing, JSON shape, env-var config, CI/CD snippet, headless/SSH running, advanced manual save, troubleshooting (7 common failure modes).
• docs/vocs.config.tsx nav entry under Red Teaming.

Report module (previously untracked)

• python/scenario/report/{__init__,_save,_aggregate,app}.py moved into tracking.

Key design decisions

Decision	Why
Module-level _BATCH_DIR, lazy-init	One dir per process run. No "which test owns which file" bookkeeping.
Timestamp-named dirs (YYYYMMDD_HHMMSS)	Lexicographic sort = chronological sort; no metadata file needed.
LLM analysis at save time (Python only)	Dashboards open instantly — running 20 LLM calls when you click "open" would be unusable.
Separate "prioritized fixes" aggregator	Cross-batch aggregation is expensive; cached in _aggregated_fixes.json, refresh via button.
isinstance detection instead of @pytest.mark.redteam	Markers are boilerplate users forget. If you use RedTeamAgent, the report Just Appears.
Save errors swallowed with warning	Reporting is observability, not correctness. A file-write failure must never break a test.
Same JSON shape in Py and TS	One dashboard serves both. No need to port the 976-line Streamlit app to Node.

Verified end-to-end

Ran test_goat_short_run_produces_report (3-turn GOAT vs. mock always-refuse agent):

• Passed in 71s.
• Report auto-saved to ./redteam-reports/<ts>/<ts>_goat_report_verification_goat.json.
• Payload contains all analysis fields populated: severity=high, break_severity=none, failure_summary (non-empty), 4 suggestions, correct failing_turn_index=null (agent held).
• _find_batch_dir resolves --latest 1/2/3 correctly.
• 116 existing JS tests still pass.

Related

• Epic: epic: scenarios red teaming langwatch#1713
• Stacks on the GOAT + paper-fidelity work already merged into feat/red-team-dynamic-techniques.

🤖 Generated with Claude Code

[github-actions]

Automated low-risk assessment

This PR was evaluated against the repository's Low-Risk Pull Requests procedure and does not qualify as low risk.

This PR introduces new integrations with external LLM services (use of litellm in _save.py and _aggregate.py) and new runtime behavior that affects test execution (pytest auto-save hook and runner auto-save). It also adds a new CLI/console script, packaging extras, and a Streamlit dashboard — changes that interact with third‑party libraries and change runtime/test behavior, so it does not meet the low-risk criteria.

This PR requires a manual review before merging.