feat(detectors): integrated rca into evaluation workflow

[poshinchen]

Summary

Integrates the detection pipeline (detect_failures → analyze_root_cause) into the Experiment evaluation workflow so that failing cases are automatically diagnosed and fix recommendations surface directly in the report.

What's new

• diagnose_session() — single-call pipeline that runs failure detection followed by root cause analysis on a Session. Returns a DiagnosisResult with failures, root causes, and deduplicated recommendations.
• DiagnosisResult — Pydantic model with a recommendations property that extracts and deduplicates fix strings from all root causes.
• DiagnosisConfig — groups all diagnosis-related parameters into a single config object:
  • trigger: DiagnosisTrigger = "on_failure" — when to run diagnosis ("on_failure" or "always")
  • model: Model | str | None = None — override the LLM used for diagnosis
  • confidence_threshold: ConfidenceLevel = "medium" — minimum confidence for failure detection
• Experiment integration — accepts an optional diagnosis_config: DiagnosisConfig parameter. When None (the default), diagnosis is disabled. When provided, diagnosis runs according to the trigger setting.
• EvaluationReport — two new fields: diagnoses: list[dict | None] and recommendations: list[str | None], populated per-case (shared across evaluator reports).
• Diagnosis runs once per failing case (not per evaluator) to avoid duplicate LLM calls, then results are shared across all evaluator reports for that case.

Usage

Standalone diagnosis:

from strands_evals.detectors import diagnose_session

result = diagnose_session(session, confidence_threshold="low")

for rc in result.root_causes:
    print(f"{rc.root_cause_explanation}")
    print(f"  Fix ({rc.fix_type}): {rc.fix_recommendation}")

for rec in result.recommendations:
    print(rec)

Integrated with Experiment:

from strands_evals import DiagnosisConfig, Experiment

# Diagnose on failure (default trigger)
experiment = Experiment(
    cases=test_cases,
    evaluators=[evaluator],
    diagnosis_config=DiagnosisConfig(),
)

# Always diagnose, with custom model
experiment = Experiment(
    cases=test_cases,
    evaluators=[evaluator],
    diagnosis_config=DiagnosisConfig(trigger="always", model="claude-haiku"),
)

reports = experiment.run_evaluations(task_fn)
report = reports[0]

for i in range(len(report.scores)):
    if not report.test_passes[i]:
        print(f"Recommendation: {report.recommendations[i]}")
        print(f"Diagnosis: {report.diagnoses[i]}")

Testing

• 5 unit tests for diagnose_session() (skips RCA when no failures, passes model/threshold, serialization round-trip)
• 2 unit tests for DiagnosisResult.recommendations (deduplication, empty case)
• 6 unit tests for Experiment integration (disabled when no config, calls diagnosis for failing cases, skips passing cases, handles non-Session trajectory, shared across evaluators, exception handling)

pytest tests/strands_evals/detectors/test_diagnosis.py tests/strands_evals/detectors/test_types.py tests/strands_evals/test_experiment.py::TestDiagnoseOnFailure -v

All 13 tests pass.

Type of Change

New feature

Checklist

• I have added any necessary tests that prove my fix is effective or my feature works
• My changes generate no new warnings
• Any dependent changes have been merged and published