Skip to content

[feature] support aime26 dataset#274

Merged
SJTUyh merged 3 commits into
AISBench:masterfrom
yejj710:aime26
May 7, 2026
Merged

[feature] support aime26 dataset#274
SJTUyh merged 3 commits into
AISBench:masterfrom
yejj710:aime26

Conversation

@yejj710

@yejj710 yejj710 commented May 6, 2026

Copy link
Copy Markdown
Collaborator

PR Type / PR类型

  • Feature(功能新增)
  • Bugfix(Bug 修复)
  • Docs(文档更新)
  • CI/CD(持续集成/持续部署)
  • Refactor(代码重构)
  • Perf(性能优化)
  • Dependency(依赖项更新)
  • Test-Cases(测试用例更新)
  • Other(其他)

Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)

🔍 Motivation / 变更动机

支持aime26数据集。

📐 Associated Test Results / 关联测试结果

Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接,例如 CI 管道、测试报告等。

✅ Checklist / 检查列表

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues. / 使用预提交或其他 linting 工具来修复潜在的 lint 问题。
  • Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖,导致 Bug 的情况应在单元测试中添加。
  • The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是,请添加更多单元测试以确保正确性。
  • All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档(API 文档、文档字符串、示例教程)已更新以反映这些更改。

🌟 Useful CI Command / 实用的CI命令

Command / 命令 Introduction / 介绍
/gemini review Performs a code review for the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 执行代码审核。
/gemini summary Provides a summary of the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 提供摘要。
/gemini help Displays a list of available commands of Gemini. / 显示 Gemini 可用命令的列表。
/readthedocs build Triggers a build of the documentation for the current pull request in its current state by Read the Docs. / 触发当前拉取请求在当前状态下由 Read the Docs 构建文档。

@yejj710 yejj710 had a problem deploying to smoke-test-approval May 6, 2026 04:27 — with GitHub Actions Failure

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces the AIME2026 dataset to the benchmark suite, providing documentation, configuration files for 0-shot chat and string prompts, and a new smoke test for accuracy verification. The review feedback suggests aligning dataset abbreviations in the configuration files with the test execution script, refactoring the dataset loading method to correctly override the base class while adding robustness for file encoding and empty lines, and using variables instead of hardcoded strings in the smoke test shell script.

with read_base():
from .aime2026_gen_0_shot_chat_prompt import aime2026_datasets

aime2026_datasets[0]['abbr'] = 'aime2026_0_shot_chat'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The abbr should match the dataset name passed to the ais_bench command in run.sh. Since run.sh uses ${CASE_NAME}_0_shot_chat (which evaluates to accuracy_aime2026_0_shot_chat), the abbreviation here should be updated to match. Otherwise, the benchmark might fail to find the dataset or produce output files with unexpected names.

Suggested change
aime2026_datasets[0]['abbr'] = 'aime2026_0_shot_chat'
aime2026_datasets[0]['abbr'] = 'accuracy_aime2026_0_shot_chat'

with read_base():
from .aime2026_gen_0_shot_str import aime2026_datasets

aime2026_datasets[0]['abbr'] = 'aime2026_0_shot_str'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The abbr should match the dataset name used in run.sh. It should be updated to accuracy_aime2026_0_shot_str to ensure consistency with the test execution command.

Suggested change
aime2026_datasets[0]['abbr'] = 'aime2026_0_shot_str'
aime2026_datasets[0]['abbr'] = 'accuracy_aime2026_0_shot_str'

Comment on lines +11 to +18
def load(path, **kwargs):
path = get_data_path(path)
dataset = []
with open(path, 'r') as f:
for line in f:
line = json.loads(line.strip())
dataset.append(line)
return Dataset.from_list(dataset) No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The load method should be an instance method to correctly override the abstract method in BaseDataset. Additionally, it is recommended to specify encoding='utf-8' when opening the file and to skip empty lines in the JSONL file to ensure robustness against malformed data.

Suggested change
def load(path, **kwargs):
path = get_data_path(path)
dataset = []
with open(path, 'r') as f:
for line in f:
line = json.loads(line.strip())
dataset.append(line)
return Dataset.from_list(dataset)
def load(self, path, **kwargs):
path = get_data_path(path)
dataset = []
with open(path, 'r', encoding='utf-8') as f:
for line in f:
line = line.strip()
if line:
dataset.append(json.loads(line))
return Dataset.from_list(dataset)

TIMESTAMP="${WORK_DIR_INFO##*/}"

# 数据集abbr列表,用于文件检查
dataset_abbr_list=(aime2026_0_shot_chat aime2026_0_shot_str)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Instead of hardcoding the dataset abbreviations, it is better to use the ${CASE_NAME} variable to maintain consistency with the ais_bench command and the config overrides.

Suggested change
dataset_abbr_list=(aime2026_0_shot_chat aime2026_0_shot_str)
dataset_abbr_list=(${CASE_NAME}_0_shot_chat ${CASE_NAME}_0_shot_str)

@yejj710 yejj710 had a problem deploying to smoke-test-approval May 6, 2026 06:28 — with GitHub Actions Error
@yejj710 yejj710 had a problem deploying to smoke-test-approval May 6, 2026 07:04 — with GitHub Actions Failure
@yejj710 yejj710 had a problem deploying to smoke-test-approval May 6, 2026 07:56 — with GitHub Actions Failure
@yejj710 yejj710 had a problem deploying to smoke-test-approval May 6, 2026 08:24 — with GitHub Actions Failure
@yejj710 yejj710 had a problem deploying to smoke-test-approval May 6, 2026 08:36 — with GitHub Actions Failure
@yejj710 yejj710 changed the title [new-dataset] support aime26 [feature] support aime26 dataset May 7, 2026
@yejj710 yejj710 had a problem deploying to smoke-test-approval May 7, 2026 06:16 — with GitHub Actions Failure
@yejj710 yejj710 temporarily deployed to smoke-test-approval May 7, 2026 06:28 — with GitHub Actions Inactive
@SJTUyh SJTUyh merged commit 46306b0 into AISBench:master May 7, 2026
7 of 9 checks passed
@yejj710 yejj710 deleted the aime26 branch July 1, 2026 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants