Skip to content

fix(benchmark): handle empty OmniDocBench page metrics - #1666

Merged
Yunnglin merged 3 commits into
modelscope:mainfrom
git-jxj:codex/fix-omnidoc-empty-page-metrics
Aug 30, 2026
Merged

fix(benchmark): handle empty OmniDocBench page metrics#1666
Yunnglin merged 3 commits into
modelscope:mainfrom
git-jxj:codex/fix-omnidoc-empty-page-metrics

Conversation

@git-jxj

@git-jxj git-jxj commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Fixes #1650

The OmniDocBench v1.6 scorer can return an empty metric object for a page with no supported metric-bearing content. Such pages are now marked excluded instead of aborting the whole run, while malformed and invalid scorer results still fail closed.

The normalized score count and IDs only include pages that contribute to the official Overall components. The benchmark evaluation version moves to v1.1.

@Yunnglin

Copy link
Copy Markdown
Collaborator

Thanks for the focused fix. Treating the official scorer's valid empty {} result as an excluded page is the right approach.

Before merging, could you please address two consistency issues?

  1. Please bump BenchmarkMeta.evaluation_version to v1.1. This changes the scoring semantics from aborting the evaluation to excluding the page, so cache and report identities should distinguish the two behaviors.

  2. aggregate_scores() currently includes excluded pages in normalized_score.num and ids via len(sample_scores), even though those pages did not contribute to the score. Please restrict them to the pages that actually contributed to the Overall components.

It would also be helpful to add a regression test with one valid page and one excluded page, asserting the resulting denominators and IDs.

With these changes, the PR should be ready to merge.

@git-jxj

git-jxj commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

Updated in 83f57bd: bumped the evaluation version and limited normalized_score num and ids to pages that contribute to Overall. The mixed valid and excluded page case covers the denominator and IDs.

@Yunnglin Yunnglin left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Yunnglin
Yunnglin merged commit 9c180e1 into modelscope:main Aug 30, 2026
3 checks passed
@git-jxj
git-jxj deleted the codex/fix-omnidoc-empty-page-metrics branch September 3, 2026 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] omni_doc_bench_v1_6测评,沙箱容器报错

2 participants