Skip to content

feat: add VisFactor benchmark - #1661

Merged
Yunnglin merged 1 commit into
mainfrom
feat/visfactor
Aug 28, 2026
Merged

feat: add VisFactor benchmark#1661
Yunnglin merged 1 commit into
mainfrom
feat/visfactor

Conversation

@Yunnglin

Copy link
Copy Markdown
Collaborator

Summary

  • add the VisFactor multimodal benchmark using the ModelScope lmms-lab-encoder/visfactor dataset
  • preserve the official interleaved multi-image prompts and category-specific deterministic answer normalization
  • reproduce the official two-level metric: logical AND across rows in each (category_id, eval_index) item, then macro-average accuracy across subtests
  • add focused scorer/aggregation tests, VLM smoke tests, generated metadata, and bilingual documentation

Research and alignment

  • Official repository: CUHK-ARISE/VisFactor (vlmeval/dataset/visfactor.py)
  • Paper: arXiv:2502.16435
  • Cross-checked against the merged lmms-eval VisFactor task implementation
  • Verified ModelScope schema and standard loader support: 3,046 rows, 808 logical items, 20 subtests, and 6,048 images

Validation

  • make lint
  • pytest tests/benchmark/test_visfactor.py tests/benchmark/test_vlm.py::TestVLMBenchmark::test_visfactor_mock -q
  • pytest tests/api/test_benchmark_meta.py -q
  • pytest tests/cli/test_all.py::TestRun::test_ci_lite -v -s -p no:warnings
  • real API smoke run with qwen-vl-plus, 20 shuffled rows, eval_batch_size=5
  • independently re-scored the 20 real predictions with the lmms-eval reference scorer: 0 mismatches; 20/20 outputs followed the JSON answer contract
  • compared 9,138 generated prediction cases over all 3,046 rows with the lmms-eval scorer: 0 mismatches

Copilot AI lite review requested due to automatic review settings August 28, 2026 03:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Yunnglin
Yunnglin merged commit 9270cfc into main Aug 28, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants