[bugfix] Add postprocessor to extract last option for MMMU datasets

[SijieFu]

Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。

PR Type / PR类型

• Feature（功能新增）
• Bugfix（Bug 修复）
• Docs（文档更新）
• CI/CD（持续集成/持续部署）
• Refactor（代码重构）
• Perf（性能优化）
• Dependency（依赖项更新）
• Test-Cases（测试用例更新）
• Other（其他）

Related Issue | 关联 Issue
Relates to #224

🔍 Motivation / 变更动机

Previously, when benchmarking accuracy with the MMMU dataset, the postprocessing of the original predictions from models is missing/incorrect. The postprocessed predictions were the same as the original predictions. Therefore, even if the model outputs the correct answer, the evaluation process still count it as incorrect, resulting in a low accuracy score. For example,

"0": {
            "prompt": [
                {
                    "role": "HUMAN",
                    "prompt": [
                        {
                            "text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\n\nEach of the following situations relates to a different company. ",
                            "type": "text"
                        },
                        {
                            "image_url": {
                                "url": "file://{project_root}/ais_bench/datasets/mmmu/MMMU_images/1_1.jpg"
                            },
                            "type": "image_url"
                        },
                        {
                            "text": " For company B, find the missing amounts.A. $63,020\nB. $58,410\nC. $71,320\nD. $77,490\n",
                            "type": "text"
                        }
                    ]
                }
            ],
            "origin_prediction": "Let's solve this step by step.\n\n......\n\nANSWER: D",
            "predictions": [
                "Let's solve this step by step.\n\n......\n\nANSWER: D"
            ],
            "references": [
                {
                    "answer": "D",
                    "category": "Business",
                    "choices": "{\"A\": \"$63,020\", \"B\": \"$58,410\", \"C\": \"$71,320\", \"D\": \"$77,490\"}",
                    "l2-category": "Accounting",
                    "split": "dev"
                }
            ],
            "correct": [
                false
            ]
        }

📝 Modification / 修改内容

This PR adds a pred_postprocessor to the MMMU-related evaluation configurations.

📐 Associated Test Results / 关联测试结果

Instead of showing the original predictions, the postprocessed predictions is showing a single choice letter:

"0": {
            "prompt": [
                {
                    "role": "HUMAN",
                    "prompt": [
                        {
                            "text": "Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: [LETTER]' (without quotes) where [LETTER] is one of A,B,C,D. Think step by step before answering.\n\nEach of the following situations relates to a different company. ",
                            "type": "text"
                        },
                        {
                            "image_url": {
                                "url": "file://{project_root}/ais_bench/datasets/mmmu/MMMU_images/1_1.jpg"
                            },
                            "type": "image_url"
                        },
                        {
                            "text": " For company B, find the missing amounts.A. $63,020\nB. $58,410\nC. $71,320\nD. $77,490\n",
                            "type": "text"
                        }
                    ]
                }
            ],
            "origin_prediction": "Let's solve this step by step.\n\n......\n\nANSWER: D",
            "predictions": [
                "D"
            ],
            "references": [
                {
                    "answer": "D",
                    "category": "Business",
                    "choices": "{\"A\": \"$63,020\", \"B\": \"$58,410\", \"C\": \"$71,320\", \"D\": \"$77,490\"}",
                    "l2-category": "Accounting",
                    "split": "dev"
                }
            ],
            "correct": [
                true
            ]
        }

✅ Checklist / 检查列表

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues. / 使用预提交或其他 linting 工具来修复潜在的 lint 问题。
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖，导致 Bug 的情况应在单元测试中添加。
• The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是，请添加更多单元测试以确保正确性。
• All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档（API 文档、文档字符串、示例教程）已更新以反映这些更改。

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects. / 如果此拉取请求对下游或其他相关项目有潜在影响，应在那些项目中测试此 PR。
• CLA has been signed and all committers have signed the CLA in this PR. / CLA 已签署，且本 PR 中的所有提交者均已签署 CLA。

[gemini-code-assist]

Code Review

This pull request introduces the last_option_postprocess function into the evaluation configurations for MMMU and MMMU-Pro datasets. Reviewers identified several issues with this approach, noting that the post-processor is highly susceptible to false positives due to its lack of word boundaries and case-insensitivity. This is especially concerning for Chain-of-Thought (CoT) responses and datasets using extended option sets where common letters like 'I' might be incorrectly matched within the model's reasoning or standard text. It is recommended to implement a more robust extraction logic that targets specific answer patterns and to ensure the function is correctly registered within the project's post-processor registry.

[gemini-code-assist]

The last_option_postprocess function is highly susceptible to false positives because its implementation (re.findall(rf'([{options}])', text)) matches any occurrence of the option letters, even when they are part of other words (e.g., matching 'B' in 'Business'). This is risky as MMMU outputs often contain category names or reasoning that may include these letters. Additionally, the postprocessor is case-sensitive and will fail to extract lowercase answers. Consider improving the postprocessor to use word boundaries (e.g., rf'\b([{options}])\b') and case-insensitivity. Also, note that last_option_postprocess is not registered in the TEXT_POSTPROCESSORS registry, which is inconsistent with other postprocessors in the project.

[gemini-code-assist]

For Chain-of-Thought (CoT) responses, using a naive last_option_postprocess is particularly risky as the model might mention multiple options during reasoning. Since the prompt explicitly requests the format ANSWER: [LETTER], it would be much more robust to use a postprocessor that specifically targets this pattern rather than just finding the last occurrence of any option letter. Furthermore, the current implementation is case-sensitive and can match letters inside words.

[gemini-code-assist]

The risk of false positives with last_option_postprocess is even higher here because the options string ABCDEFGHIJ includes the letter 'I', which frequently appears as a standalone pronoun or within words in model responses. This naive extraction will likely return 'I' as the answer if it appears after the actual choice. A more specific pattern matching the requested 'Answer: $LETTER' format should be used instead.

[gemini-code-assist]

The last_option_postprocess function is prone to incorrect extractions, especially with the extended option set ABCDEFGHIJ. It can match letters within words and is case-sensitive. Given that MMMU-Pro evaluation relies on precise option extraction, consider using a more robust method that enforces word boundaries and handles case-insensitivity.

[gemini-code-assist]

Using last_option_postprocess for CoT evaluation is risky as it may pick up letters from the model's reasoning steps. Since the prompt specifies a clear format ('Answer: $LETTER'), the postprocessor should ideally look for that specific pattern to ensure accuracy, rather than simply taking the last matching character in the text.

[gemini-code-assist]

The last_option_postprocess implementation is too naive for reliable evaluation. It lacks word boundary checks and case-insensitivity, which can lead to incorrect results if the model output contains words like 'Business' or 'Image' (containing 'B' or 'I'). A more robust extraction logic is recommended.

[Copilot]

Pull request overview

Adds prediction postprocessing to MMMU-family dataset evaluation configs so that generated answers are reduced to a final multiple-choice option letter before scoring, fixing cases where verbose model outputs were previously mis-scored.

Changes:

• Add last_option_postprocess as pred_postprocessor for MMMU (A–D) evaluation configs.
• Add last_option_postprocess as pred_postprocessor for MMMU-Pro evaluation configs (A–J).
• Import the postprocessor into the affected dataset config files.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
ais_bench/benchmark/configs/datasets/mmmu/mmmu_gen.py	Adds eval-time pred_postprocessor to extract final A–D option.
ais_bench/benchmark/configs/datasets/mmmu/mmmu_gen_cot.py	Adds eval-time pred_postprocessor to extract final A–D option for CoT outputs.
ais_bench/benchmark/configs/datasets/mmmu_pro/mmmu_pro_vision_gen.py	Adds eval-time pred_postprocessor to extract final A–J option for MMMU-Pro vision.
ais_bench/benchmark/configs/datasets/mmmu_pro/mmmu_pro_vision_cot_gen.py	Adds eval-time pred_postprocessor to extract final A–J option for MMMU-Pro vision CoT.
ais_bench/benchmark/configs/datasets/mmmu_pro/mmmu_pro_options10_gen.py	Adds eval-time pred_postprocessor to extract final A–J option for MMMU-Pro options10.
ais_bench/benchmark/configs/datasets/mmmu_pro/mmmu_pro_options10_cot_gen.py	Adds eval-time pred_postprocessor to extract final A–J option for MMMU-Pro options10 CoT.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.