[docs]Add mini dataset docs#302
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces 'mini' subsets for SWE-Bench, TAU2-Bench, and VBench 1.0 to significantly reduce evaluation costs and time. The changes include updated documentation in both English and Chinese, new configuration examples, and improved formatting in the main README. Review feedback highlights several critical issues in the documentation: there are inconsistencies and typos in the repository URLs for the SWE-Bench multilingual mini dataset, and the current instructions for TAU2-Bench and VBench suggest destructive manual file replacements within library or third-party directories. It is recommended to fix the broken links and prioritize non-destructive configuration methods (such as using configurable data paths) over manual file overwrites.
| - `mini_swe_agent_swe_bench_verified_mini.py`: SWE-bench Verified Mini (`MariusHobbhahn/swe-bench-verified-mini`, **50** instances) — a community subset of Verified designed to be much cheaper to run; see the dataset card and the subset construction repo: `https://huggingface.co/datasets/MariusHobbhahn/swe-bench-verified-mini` and `https://github.com/mariushobbhahn/make_swe_bench_verified_mini`. | ||
| - `mini_swe_agent_swe_bench_full.py`: SWE-bench Full (`princeton-nlp/SWE-Bench`) — the full test set. | ||
| - `mini_swe_agent_swe_bench_multilingual.py`: SWE-bench Multilingual (`SWE-bench/SWE-bench_Multilingual`) — multilingual issue statements. | ||
| - `mini_swe_agent_swe_bench_multilingual_mini.py`: SWE-bench Multilingual Mini (`MariusHobbhahn/swe-bench-multilingual-mini`, **15**/**30**/**60** instances) — an AISBench-constructed Multilingual subset designed to significantly reduce evaluation cost; see the dataset card and construction repository: `https://modelers.cn/datasets/AISBench/SWE-Bench_Multilingual_mini` and `https://github.com/AISBench/datasets/tree/main/mini_datasets/swe_bench_multiligual_mini`. |
There was a problem hiding this comment.
There are two issues in this line:
- Inconsistency: The text mentions
MariusHobbhahn/swe-bench-multilingual-mini, but the URLs point toAISBenchrepositories. Please verify the correct owner and update the text if necessary. - Typo: The GitHub URL contains
multiligualinstead ofmultilingual. This likely results in a broken link if the repository name is spelled correctly.
| - `mini_swe_agent_swe_bench_verified_mini.py`：SWE-bench Verified Mini（`MariusHobbhahn/swe-bench-verified-mini`，**50** 条），社区构造的 Verified 子集，用于显著降低评测成本；子集筛选/构造方式见数据集卡与构造仓库：`https://huggingface.co/datasets/MariusHobbhahn/swe-bench-verified-mini`、`https://github.com/mariushobbhahn/make_swe_bench_verified_mini`。 | ||
| - `mini_swe_agent_swe_bench_full.py`：SWE-bench Full（`princeton-nlp/SWE-Bench`），完整测试集。 | ||
| - `mini_swe_agent_swe_bench_multilingual.py`：SWE-bench Multilingual（`SWE-bench/SWE-bench_Multilingual`），包含多语言 issue 描述的数据集。 | ||
| - `mini_swe_agent_swe_bench_multilingual_mini.py`：SWE-bench Multilingual Mini（`MariusHobbhahn/swe-bench-multilingual-mini`，**15**/**30**/**60** 条），AISBench官方构造的 Multilingual 子集，用于显著降低评测成本；子集筛选/构造方式见数据集卡与构造仓库：`https://modelers.cn/datasets/AISBench/SWE-Bench_Multilingual_mini`、`https://github.com/AISBench/datasets/tree/main/mini_datasets/swe_bench_multiligual_mini`。 |
There was a problem hiding this comment.
此行存在两个问题：
- 不一致性：文本提到了
MariusHobbhahn/swe-bench-multilingual-mini，但 URL 指向的是AISBench仓库。请确认正确的持有者并相应更新文本。 - 拼写错误：GitHub URL 中包含
multiligual而非multilingual。如果仓库名称拼写正确，这可能会导致链接失效。
| VBench-1.0-mini provides a condensed `VBench_full_info.json` corresponding to the sampled prompts. AISBench reads `ais_bench/third_party/vbench/VBench_full_info.json` by default, so you need to overwrite it with the mini version: | ||
|
|
||
| ```bash | ||
| cp <MINI_ROOT>/VBench_full_info.json ais_bench/third_party/vbench/VBench_full_info.json |
There was a problem hiding this comment.
Overwriting files in the third_party directory is a destructive operation that can complicate environment management and switching between dataset versions. It is recommended to promote the non-destructive configuration method (using the full_json_dir field in the config file) as the primary approach instead of manual file replacement.
| VBench-1.0-mini 提供了与采样 prompt 对应的精简版 `VBench_full_info.json`。AISBench 默认读取 `ais_bench/third_party/vbench/VBench_full_info.json`，因此需要将 mini 版本的该文件覆盖到该路径： | ||
|
|
||
| ```bash | ||
| cp <MINI_ROOT>/VBench_full_info.json ais_bench/third_party/vbench/VBench_full_info.json |
There was a problem hiding this comment.
覆盖 third_party 目录中的文件是一种破坏性操作，会增加环境管理和版本切换的复杂性。建议将非破坏性的配置方法（在配置文件中使用 full_json_dir 字段）作为首选方案，而不是手动替换文件。
| Replace tau2's original dataset files with the TAU2-mini files (**back up** `{tau2_root}/src/benchmark/tau2/data/tau2/domains` first): | ||
|
|
||
| ```bash | ||
| cp -r <TAU2_MINI_ROOT>/tau2_subsets/* {tau2_root}/src/benchmark/tau2/data/tau2/domains |
There was a problem hiding this comment.
The instruction to manually replace files within the installed package directory ({tau2_root}/src/benchmark/tau2/data/tau2/domains) is risky and difficult to manage. This makes the installation stateful and requires manual backups/restores to switch datasets. Ideally, the integration should support a configurable data path to avoid modifying the library's internal files.
There was a problem hiding this comment.
tau2 bench本身数据集就在依赖中，没有其他途径可以替换，只能这样
| 将 TAU2-mini 数据中的文件替换 tau2 中的原始数据集文件（**请提前备份好** `{tau2安装路径}/src/benchmark/tau2/data/tau2/domains`）： | ||
|
|
||
| ```bash | ||
| cp -r <TAU2_MINI_ROOT>/tau2_subsets/* {tau2安装路径}/src/benchmark/tau2/data/tau2/domains |
There was a problem hiding this comment.
手动替换已安装包目录中的文件具有较高风险。这使得安装状态变得不可控，且切换数据集时需要手动备份/恢复。理想情况下，集成方案应支持可配置的数据路径，以避免修改库的内部文件。
There was a problem hiding this comment.
tau2 bench本身数据集就在依赖中，没有其他途径可以替换，只能这样
2 participants
Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。
PR Type / PR类型
Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)
🔍 Motivation / 变更动机
Please describe the motivation of this PR and the goal you want to achieve through this PR.
请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。
Add mini dataset (SWEBench multilingual, tau2, VBench) docs
📝 Modification / 修改内容
Please briefly describe what modification is made in this PR.
请简要描述此拉取请求中进行的修改。
Add mini dataset (SWEBench multilingual, tau2, VBench) docs
📐 Associated Test Results / 关联测试结果
Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接，例如 CI 管道、测试报告等。
Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改？如果是，请描述它如何破坏兼容性，以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。
If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。
🌟 Use cases (Optional) / 使用案例（可选）
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。
✅ Checklist / 检查列表
Before PR:
After PR:
👥 Collaboration Info / 协作信息
🌟 Useful CI Command / 实用的CI命令
/gemini review/gemini summary/gemini help/readthedocs build