add UT & doc for mathvision#288
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds the MathVision dataset to the documentation and introduces unit tests for the MathVision dataset and evaluator. Key feedback includes correcting Chinese text in the English documentation, addressing an unused import and lack of test coverage for the main dataset entry point, and fixing a mock object in the unit tests that fails type checks and causes incorrect record loading.
| | MMStar | Multimodal Understanding (Image + Text) | [Detailed Introduction](https://github.com/AISBench/benchmark/tree/master/ais_bench/benchmark/configs/datasets/mmstar/README_en.md) | | ||
| | Video-MME | Multimodal Understanding (video + Text) | [Detailed Introduction](https://github.com/AISBench/benchmark/tree/master/ais_bench/benchmark/configs/datasets/videomme/README_en.md) | | ||
| | OCRBench_v2 | Multimodal Understanding (Image + Text, OCR Evaluation) | [Detailed Introduction](https://github.com/AISBench/benchmark/tree/master/ais_bench/benchmark/configs/datasets/ocrbench_v2/README_en.md) | | ||
| | MathVision | Multimodal Understanding (Image + Text) | [详细介绍](https://github.com/AISBench/benchmark/tree/master/ais_bench/benchmark/configs/datasets/mathvision/README_en.md) | |
There was a problem hiding this comment.
The text "详细介绍" is in Chinese. In the English documentation, it should be "Detailed Introduction" to maintain consistency with the rest of the table.
| | MathVision | Multimodal Understanding (Image + Text) | [详细介绍](https://github.com/AISBench/benchmark/tree/master/ais_bench/benchmark/configs/datasets/mathvision/README_en.md) | | |
| | MathVision | Multimodal Understanding (Image + Text) | [Detailed Introduction](https://github.com/AISBench/benchmark/tree/master/ais_bench/benchmark/configs/datasets/mathvision/README_en.md) | |
| from datasets import Dataset | ||
|
|
||
| from ais_bench.benchmark.datasets.mathvision import ( | ||
| MathVisionDataset, |
There was a problem hiding this comment.
The MathVisionDataset class is imported but not used in any of the test cases. The main entry point MathVisionDataset.load should be covered by unit tests to ensure the integration of path resolution, image dumping, and prompt building logic.
| mock_resolve.return_value = (f.name, True) | ||
| mock_load.return_value = DatasetDict_mock({'test': [{'question': 'Q?', 'answer': '42'}]}) | ||
| records, path, is_local = _load_records(f.name, split='test') | ||
| self.assertTrue(is_local) |
There was a problem hiding this comment.
The DatasetDict_mock used here does not inherit from datasets.DatasetDict. In mathvision.py (line 111), there is an isinstance(dataset, DatasetDict) check. Since this mock fails that check, the code skips the dataset = dataset[split] logic. Consequently, list(dataset) returns a list of keys (e.g., ['test']) instead of the actual records. The test currently passes only because it does not verify the content of records.
2 participants
Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。
PR Type / PR类型
Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)
🔍 Motivation / 变更动机
补充mathvision数据集的资料和UT
📝 Modification / 修改内容
补充mathvision数据集的资料和UT
📐 Associated Test Results / 关联测试结果
Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接，例如 CI 管道、测试报告等。
Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改？如果是，请描述它如何破坏兼容性，以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。
If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。
🌟 Use cases (Optional) / 使用案例（可选）
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。
✅ Checklist / 检查列表
Before PR:
After PR:
👥 Collaboration Info / 协作信息
🌟 Useful CI Command / 实用的CI命令
/gemini review/gemini summary/gemini help/readthedocs build