feat: Phase 11 — Component Diagnosis

[himanshu231204]

Summary

Implements the Component Diagnosis system (Phase 11) that attributes blame when RAG evaluations fail — telling the user WHERE it failed: retrieval, generation, or chunking.

Motivation

90% of production RAG failures are retrieval problems, but teams don't know which component caused the failure. This module provides blame attribution with actionable recommendations, saving hours of guesswork.

Files Modified

New files:

• openagent_eval/diagnosis/init.py — Module exports
• openagent_eval/diagnosis/models.py — FailureMode enum (8 modes), BlameTarget, ComponentScores, DiagnosisReport, etc.
• openagent_eval/diagnosis/blame.py — BlameAttribution class analyzing retrieval/generation/chunking scores
• openagent_eval/diagnosis/chunking.py — ChunkingQualityAnalyzer detecting overlap, inconsistent sizes, content gaps
• openagent_eval/diagnosis/analyzer.py — DiagnosisAnalyzer orchestrator producing DiagnosisReport with recommendations
• openagent_eval/exceptions/diagnosis.py — DiagnosisError, DiagnosisExecutionError, BlameAttributionError
• openagent_eval/cli/commands/diagnose.py — \oaeval diagnose\ CLI command

Modified files:

• openagent_eval/cli/main.py — Registered diagnose command
• openagent_eval/exceptions/init.py — Added diagnosis exception exports

Tests:

• ests/unit/test_diagnosis/ — 62 tests covering models, blame attribution, chunking analysis, and full pipeline integration

Architectural Decisions

• Follows D019 (Blame Attribution) decision from DECISIONS.md
• BlameAttribution uses weighted scoring: count × average_confidence per blame target
• Chunking analysis uses Jaccard similarity for overlap detection and length variance for inconsistency
• DiagnosisAnalyzer orchestrates blame + chunking analysis and generates actionable recommendations
• All 62 new tests pass; 651 existing tests continue to pass

Testing

• 62 new unit and integration tests in \ ests/unit/test_diagnosis/\
• All existing tests (651) continue to pass

Remaining TODOs

• Wire DiagnosisAnalyzer into the evaluation pipeline (optional post-evaluation diagnosis)
• Add \DiagnosisConfig\ to configuration models for threshold tuning
• Phase 12 (Synthetic Test Data) and Phase 13 (CI/CD Integration) are next