【feature】Support RealworldQA dataset#268
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request integrates the RealworldQA dataset into the benchmark suite, adding necessary configurations, documentation, and implementation for data loading and evaluation. Review feedback points out a critical discrepancy between the dataset loader's output keys and the expected input for prompt templates, which would result in runtime errors. Additionally, the logic for loading parquet files should be updated to respect specific file paths provided in the configuration.
| content = AIS_TEXT_START + question + AIS_CONTENT_TAG + AIS_IMAGE_START + image_url + AIS_CONTENT_TAG | ||
|
|
||
| dataset.append({ | ||
| "content": content, | ||
| "answer": answer, | ||
| }) |
There was a problem hiding this comment.
RealworldQADataset.load 返回的字典键值与配置文件（如 realworldqa_gen.py）中的 reader_cfg 和 prompt_template 不匹配。配置文件期望 question 和 image 字段，而此处返回的是 content。这会导致推理阶段 MMPromptTemplate 格式化时因找不到字段而报错。此外，手动构建 content 字符串与 MMPromptTemplate 的功能重复，建议直接返回原始字段。
dataset.append({
"question": question,
"image": image_url,
"answer": answer,
})| data_dir = os.path.dirname(path) | ||
| parquet_files = sorted([f for f in os.listdir(data_dir) if f.endswith('.parquet')]) | ||
|
|
||
| if not parquet_files: | ||
| raise ValueError(f"No parquet files found in {data_dir}") | ||
|
|
||
| dfs = [pd.read_parquet(os.path.join(data_dir, f)) for f in parquet_files] | ||
| data = pd.concat(dfs, ignore_index=True) |
There was a problem hiding this comment.
目前的 load 方法逻辑会加载 path 所在目录下的所有 .parquet 文件，这忽略了参数中可能指定的具体文件名。在配置文件（如 realworldqa_gen.py）中，path 指向的是一个具体的文件，目前的实现会导致加载该目录下所有的 parquet 文件，不符合预期。建议增加对 path 是否为文件的判断，以支持加载单个文件或整个目录。
| data_dir = os.path.dirname(path) | |
| parquet_files = sorted([f for f in os.listdir(data_dir) if f.endswith('.parquet')]) | |
| if not parquet_files: | |
| raise ValueError(f"No parquet files found in {data_dir}") | |
| dfs = [pd.read_parquet(os.path.join(data_dir, f)) for f in parquet_files] | |
| data = pd.concat(dfs, ignore_index=True) | |
| if os.path.isfile(path): | |
| data_dir = os.path.dirname(path) | |
| parquet_files = [os.path.basename(path)] | |
| else: | |
| data_dir = path | |
| parquet_files = sorted([f for f in os.listdir(data_dir) if f.endswith('.parquet')]) | |
| if not parquet_files: | |
| raise ValueError(f"No parquet files found in {data_dir}") | |
| dfs = [pd.read_parquet(os.path.join(data_dir, f)) for f in parquet_files] |
There was a problem hiding this comment.
数据集格式是写死的，可以认为此问题忽略
| from .base import BaseDataset | ||
|
|
||
| # 提示词模板,参考evalscope | ||
| OPEN_PROMPT = ( |
There was a problem hiding this comment.
【review】提示词模板应该通过配置文件指定，此处硬编码实现，会限制提示词工程的可拓展性
There was a problem hiding this comment.
此处参考evalscope实现，对齐外部测评工具
3 participants
Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。
PR Type / PR类型
新增RealWorldQA数据集
Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)
🔍 Motivation / 变更动机
Please describe the motivation of this PR and the goal you want to achieve through this PR.
请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。
📝 Modification / 修改内容
Please briefly describe what modification is made in this PR.
请简要描述此拉取请求中进行的修改。
📐 Associated Test Results / 关联测试结果
Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接，例如 CI 管道、测试报告等。
Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改？如果是，请描述它如何破坏兼容性，以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。
If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。
🌟 Use cases (Optional) / 使用案例（可选）
If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能，最好在此处列出一些用例并更新文档。
✅ Checklist / 检查列表
Before PR:
After PR:
👥 Collaboration Info / 协作信息
🌟 Useful CI Command / 实用的CI命令
/gemini review/gemini summary/gemini help/readthedocs build