Diagnose cross-modal retrieval gaps

[brianmeyer]

Summary

• Add an offline cross-modal diagnostics CLI that reads saved ablation JSON and ranks weak or under-sampled categories.
• Generate checked-in JSON and Markdown diagnostics for the current benchmark artifact.
• Preserve video_query_path in benchmark per-query rows and document the diagnostics flow in README/release docs.

Research grounding

• BEIR-style qrels/results metrics: https://github.com/beir-cellar/beir
• MTEB retrieval evaluation framework: https://github.com/embeddings-benchmark/mteb
• ViDoRe pipeline evaluation patterns: https://github.com/illuin-tech/vidore-benchmark

Tests

• python3 -m pytest -q tests/test_cross_modal_diagnostics.py tests/test_cross_modal_benchmark_defs.py
• python3 -m pytest -q

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95529a3f97

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Require a true majority before tagging generic-query artifact

Use a ceiling threshold here; with len(query_values) // 2, odd-sized categories are misclassified. For example, with 3 media queries and only 1 placeholder, generic_count >= 1 becomes true, so the tool emits generic_query_artifact and can inflate priority_score/follow-up actions even when generic prompts are not the majority. This skews the ranked diagnosis for small categories and should use a strict majority check (e.g., >= ceil(n/2) or > n/2).

Useful? React with 👍 / 👎.