【Feature】Support Vbench third party and license#152
Merged
Conversation
Contributor
Summary of ChangesHello @GaoHuaZhang, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces comprehensive support for the VBench eval benchmark within AISBench. It includes new tasks, dataset configurations, and necessary documentation to facilitate video and image quality assessments on both GPU and NPU, while maintaining compatibility with existing AISBench workflows. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Activity
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
这个 PR 增加了对 VBench 评估基准的支持，这是一个很棒的功能。代码结构清晰，主要逻辑集中在新的 VBenchEvalTask 中。新增的数据集配置文件使得 VBench 的不同评估维度可以方便地被调用。
我发现了一些可以改进的地方：
- 多个数据配置文件中硬编码了用户特定的绝对路径，这不利于代码的可移植性。
vbench_eval.py中读取 JSON 文件时未指定编码，可能导致在不同平台上的兼容性问题。vbench_eval.py中定义了一个默认的维度列表，该列表与vbench_standard.py中的定义重复，建议提取到公共位置以方便维护。- 有一个配置文件末尾缺少换行符。
除了这些小问题，整体实现质量很高。
| load_ckpt_from_local=True, | ||
| ) | ||
|
|
||
| _BASE_PATH = '/data/zhanggaohua/datasets/vbench/CogVideoX-5B-mini' |
Contributor
There was a problem hiding this comment.
此配置文件中硬编码了一个绝对路径 _BASE_PATH，这似乎包含了个人用户信息 (/data/zhanggaohua/...)。这种做法会给其他用户在不同环境中使用该配置带来不便。建议将其修改为占位符路径，例如 '/path/to/your/video/folder'，并在注释中说明用户需要修改此路径。此问题同样存在于本次PR新增的大部分 vbench 数据集配置文件中。
Suggested change
| _BASE_PATH = '/data/zhanggaohua/datasets/vbench/CogVideoX-5B-mini' | |
| _BASE_PATH = '/path/to/your/video/folder' # 请替换为您的视频目录 |
| dimension_list=['subject_consistency'], | ||
| ), | ||
| ) | ||
| ] No newline at end of file |
Contributor
There was a problem hiding this comment.
文件末尾缺少一个换行符。根据 PEP 8 规范，所有源文件应以一个换行符结尾。
Comment on lines
+122
to
+129
| if not dimension_list: | ||
| dimension_list = [ | ||
| 'subject_consistency', 'background_consistency', 'aesthetic_quality', | ||
| 'imaging_quality', 'object_class', 'multiple_objects', 'color', | ||
| 'spatial_relationship', 'scene', 'temporal_style', 'overall_consistency', | ||
| 'human_action', 'temporal_flickering', 'motion_smoothness', 'dynamic_degree', | ||
| 'appearance_style', | ||
| ] |
Contributor
There was a problem hiding this comment.
这个默认的 dimension_list 与 ais_bench/benchmark/configs/datasets/vbench/vbench_standard.py 文件中定义的 VBENCH_DEFAULT_DIMENSIONS 列表内容相同。为了保持代码的 DRY (Don't Repeat Yourself) 原则并方便维护，建议将此列表定义在一个公共位置（例如 vbench_standard.py 中），然后在此处导入使用。这样可以避免两处代码不一致导致的潜在问题。
| prompt_list = dataset_cfg.get('prompt_list') or eval_cfg.get('prompt_list') or [] | ||
| prompt_file = eval_cfg.get('prompt_file') | ||
| if prompt_file and osp.isfile(prompt_file): | ||
| with open(prompt_file, 'r') as f: |
Contributor
There was a problem hiding this comment.
在读取 JSON 文件时，建议明确指定编码格式为 utf-8，以避免在不同系统环境下可能出现的编码问题，并与文件中其他部分的文件写入操作保持一致。
Suggested change
| with open(prompt_file, 'r') as f: | |
| with open(prompt_file, 'r', encoding='utf-8') as f: |
GaoHuaZhang
temporarily deployed
to
smoke-test-approval
GitHub Actions
Inactive
GaoHuaZhang
temporarily deployed
to
smoke-test-approval
GitHub Actions
Inactive
GaoHuaZhang
temporarily deployed
to
smoke-test-approval
GitHub Actions
Inactive
GaoHuaZhang
temporarily deployed
to
smoke-test-approval
GitHub Actions
Inactive
GaoHuaZhang
temporarily deployed
to
smoke-test-approval
GitHub Actions
Inactive
GaoHuaZhang
temporarily deployed
to
smoke-test-approval
GitHub Actions
Inactive
GaoHuaZhang
temporarily deployed
to
smoke-test-approval
GitHub Actions
Inactive
GaoHuaZhang
changed the title
SJTUyh
approved these changes
May 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献，我们非常重视。以下说明将使您的拉取请求更健康，更易于获得反馈。如果您不理解某些项目，请不要担心，只需提交拉取请求并从维护人员那里寻求帮助即可。
PR Type / PR类型
Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)
🔍 Motivation / 变更动机
本 PR 主要解决两件事：
pip安装路径以及内部子第三方版本的硬编码依赖，便于在 NPU / 离线环境一键拉起。LICENSE，配合既有 NOTICE 完成开源合规归属，便于发版前的开源审计。📝 Modification / 修改内容
本 PR 包含两个 commit，按职责拆分如下：
1)
46e6427 add third_party（+120,106 行 / 451 个文件）—— 引入 vendored 第三方源码新增 ais_bench/third_party/detectron2/：Facebook Detectron2 完整源码（含
LICENSE），用于满足 VBench 中 GRiT 物体检测、密集描述等维度对 detectron2 的依赖。运行前可通过下面命令以可编辑方式安装：新增 ais_bench/third_party/vbench/：VBench 评测库及其内部嵌套的子第三方源码，覆盖
RAFT、ViCLIP、amt、grit_src、tag2Text、umt，以及prompts/augmented_prompts/、prompts/prompts_per_dimension/等评测所需 prompt 数据集。供 ais_bench/configs/vbench_examples/ 的 Standard / Custom 模式直接调用，输出与上游cal_final_score.py对齐。2)
65766f6 add license（+230 / -1）—— 补齐 Apache 2.0 许可证与第三方归属ais_bench/third_party/*/LICENSE获取第三方归属；同时把 Apache 2.0 模板末尾的Copyright [yyyy] [name of copyright owner]落实为Copyright 2026 AISBench Authors。Copyright (c) 2023 Vchitect。mindie_sd/下 vendored Python 源代码遵循 Apache 2.0，详细的逐组件归属信息汇总至根 NOTICE（涵盖 QwenImage pipeline、FlowMatchEulerDiscrete scheduler、xDiT / vLLM GroupCoordinator、Microsoft sequence-parallel all-to-all 等）。根目录 NOTICE 已撰写完毕（覆盖 Detectron2 / VBench / MindIE SD Qwen Image Edit 三块第三方归属），但当前仍是 untracked 状态、未包含在本 PR 这两个 commit 中。计划在合入前补一个提交，确保发版包含完整的第三方归属说明。
📐 Associated Test Results / 关联测试结果
未引入 API / 配置层面的不兼容变更，但有两个工程层面的影响点需要下游知悉：
If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降，请描述性能下降的影响和预期的性能改进。
✅ Checklist / 检查列表
Before PR:
After PR:
👥 Collaboration Info / 协作信息
🌟 Useful CI Command / 实用的CI命令
/gemini review/gemini summary/gemini help/readthedocs build