fix(benchmark): handle empty OmniDocBench page metrics - #1666
Conversation
|
Thanks for the focused fix. Treating the official scorer's valid empty Before merging, could you please address two consistency issues?
It would also be helpful to add a regression test with one valid page and one excluded page, asserting the resulting denominators and IDs. With these changes, the PR should be ready to merge. |
|
Updated in 83f57bd: bumped the evaluation version and limited normalized_score num and ids to pages that contribute to Overall. The mixed valid and excluded page case covers the denominator and IDs. |
Fixes #1650
The OmniDocBench v1.6 scorer can return an empty metric object for a page with no supported metric-bearing content. Such pages are now marked excluded instead of aborting the whole run, while malformed and invalid scorer results still fail closed.
The normalized score count and IDs only include pages that contribute to the official Overall components. The benchmark evaluation version moves to v1.1.