fix(metrics): compute occurrence-based ROUGE recall fallback

[Nithyaviswak]

The ROUGE fallback documented itself as unigram recall but deduplicated both sides into sets, so repetition in the reference was ignored and a term repeated many times counted once. Use a Counter intersection over the reference's total token count so recall is sensitive to how many occurrences are recovered.

Fixes #282

Description

A clear and concise description of what this PR does.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Refactoring (no functional changes)
• Test update
• CI/CD update

Related Issues

Closes #(issue_number)

How Has This Been Tested?

Describe the tests that you ran to verify your changes.

• Unit tests pass (uv run pytest)
• Linter passes (uv run ruff check .)
• Type checker passes (uv run mypy openagent_eval/)
• Manual testing performed

Checklist

• My code follows the project's coding standards
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

Screenshots (if applicable)

Add screenshots to demonstrate visual changes.

Additional Notes

Add any other notes about the PR here.

[himanshu231204]

/oc review pr

[opencode-agent]

Review Summary

Approve. The fix correctly addresses #282.

The bug

The fallback documented itself as "simple unigram recall" but converted both sides to set, so recall = len(overlap) / len(truth_words) deduplicated repeated terms on both numerator and denominator. Repetition in the reference was ignored entirely (e.g. cat vs cat cat cat scored 1.0).

The fix

rouge.py:85-88 now uses Counter:

• answer_counts & truth_counts is a multiset intersection (min of per-token counts), so repeated terms count per occurrence.
• Recall = matched occurrences / total reference occurrences — matching official ROUGE-N recall semantics.

Verification

• cat vs cat cat cat → 1/3; cat cat cat vs cat cat cat → 1.0 ✓
• Edge case cat cat cat cat vs cat cat caps correctly at 1.0 (Counter & min semantics) ✓
• Empty ground_truth path still guarded (if not truth_words works for lists) ✓
• Docstring updated to match new behavior; overlap metadata stays an int ✓

Tests

test_fallback_sensitive_to_repetition (test_generation.py:175) is well-targeted: it forces the fallback path deterministically via monkeypatch and asserts both the 1/3 partial score and 1.0 full score. pytest is imported at the top of the file, so pytest.approx and the fixture are available.

Couldn't run pytest/ruff locally (not installed in this env), but the logic was verified manually with Python.

Nits (non-blocking): none material. The fallback is now stricter than the HF path (which returns F1), but that divergence is pre-existing and out of scope.

Recommendation: merge.

opencode session  |  github run

[github-actions]

🎉 Congratulations @Nithyaviswak!

Your pull request has been successfully merged into main. 🚀

Thank you for contributing to OpenAgentHQ and helping improve the project.

We truly appreciate your contribution and hope to see you back with more amazing PRs!

Happy Open Sourcing! ❤️

🌟 This is your first merged contribution to this repository.
Welcome to the OpenAgentHQ contributors family!