Skip to content

feat(benchmarks): add PRBench - #1665

Merged
Yunnglin merged 1 commit into
mainfrom
feat/prbench
Aug 28, 2026
Merged

feat(benchmarks): add PRBench#1665
Yunnglin merged 1 commit into
mainfrom
feat/prbench

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Summary

  • add the ScaleAI/PRBench benchmark with all four official Finance and Legal splits
  • reproduce the official multi-turn prompt construction, reference-text handling, rubric judge contract, and weighted scoring formulas
  • report clipped_score:clipped_mean as the primary metric and normalized_score:mean independently, without a cross-split OVERALL row because hard splits overlap full splits
  • add generated metadata/docs, metric semantics, benchmark tests, and judge fail-closed coverage

Validation

  • make lint
  • python -m pytest tests/benchmark/test_prbench.py tests/api/judge/test_migrated_adapters.py tests/api/judge/test_gates.py -q (34 passed)
  • python -m pytest tests/cli/test_metric_semantics_e2e.py -q -p no:benchmark (30 passed)
  • python -m pytest tests/cli/test_all.py::TestRun::test_ci_lite -q -p no:warnings (passed)
  • one-sample real API E2E: 18/18 rubric verdicts parsed; independently recomputed scores matched the report

Copilot AI lite review requested due to automatic review settings August 28, 2026 03:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Yunnglin
Yunnglin merged commit d8c022a into main Aug 28, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants