Skip to content

[UT]: add UT coverage for swebench#316

Merged
SJTUyh merged 1 commit into
AISBench:masterfrom
wanlongze:feat/add-ut-coverage-swebench
Jun 2, 2026
Merged

[UT]: add UT coverage for swebench#316
SJTUyh merged 1 commit into
AISBench:masterfrom
wanlongze:feat/add-ut-coverage-swebench

Conversation

@wanlongze

@wanlongze wanlongze commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

概述

ais_bench/benchmark/summarizers/swebench.py 模块添加单元测试,实现 100% 代码覆盖率。

变更内容

新增文件

  • tests/UT/summarizers/test_swebench.py - SWEBenchSummarizer 单元测试

测试用例 (12 个)

测试用例 描述
test_pick_up_results_valid_aggregate 测试有效的聚合结果解析
test_pick_up_results_no_dir 测试结果目录不存在的情况
test_pick_up_results_no_aggregate_file 测试聚合文件不存在的情况
test_pick_up_results_zero_total_instances 测试总实例数为 0 的边界情况
test_pick_up_results_invalid_data 测试无效数据格式的处理
test_pick_up_results_missing_fields 测试缺少必要字段的情况
test_pick_up_results_negative_total 测试负数总数的异常情况
test_pick_up_results_load_exception 测试文件加载异常的处理
test_pick_up_results_100_percent 测试 100% 正确率的情况
test_pick_up_results_string_total_instances 测试字符串类型总数的处理
test_pick_up_results_empty_configs 测试空配置的情况
test_pick_up_results_resolved_greater_than_total 测试解析数大于总数的情况

UT测试用例通过情况

image

全量覆盖率

image

- Add test_swebench.py with 12 test cases covering all code paths
  - Test valid aggregate results, zero total, negative total
  - Test missing files, invalid data, load exceptions
  - Test empty configs, string total instances, etc.
  - Achieves 100% line coverage for swebench.py

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive suite of unit tests for the SWEBenchSummarizer class in tests/UT/summarizers/test_swebench.py, covering various edge cases for the _pick_up_results method. The review feedback suggests several improvements: removing the unused MagicMock import, targeting mocks locally within the module under test rather than globally on os.path and mmengine to prevent side effects, and adding a trailing newline at the end of the file to adhere to PEP 8 standards.

import os
import tempfile
import shutil
from unittest.mock import patch, MagicMock

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

建议:移除未使用的导入 MagicMock

在当前测试文件中,并没有使用到 MagicMock。为了保持代码的整洁和可维护性,建议将其移除。

Suggested change
from unittest.mock import patch, MagicMock
from unittest.mock import patch

Comment on lines +40 to +45
@patch('os.path.isdir')
@patch('os.path.isfile')
@patch('mmengine.load')
@patch('ais_bench.benchmark.summarizers.swebench.model_abbr_from_cfg_used_in_summarizer')
@patch('ais_bench.benchmark.summarizers.swebench.dataset_abbr_from_cfg')
@patch('ais_bench.benchmark.summarizers.swebench.get_infer_output_path')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

建议:将全局的 os.pathmmengine 补丁(patch)替换为局部模块补丁。

原因:
直接对 os.path.isdiros.path.isfilemmengine.load 进行全局 Mock 会污染全局命名空间。如果在测试运行期间,测试框架或其他标准库函数(例如 shutil.rmtreetempfile 相关操作)调用了这些方法,可能会导致不可预测的副作用或测试套件不稳定。

根据 Python unittest.mock 的最佳实践,应该在使用该函数的地方(即目标模块 ais_bench.benchmark.summarizers.swebench)进行 Mock,而不是在其定义的地方。

改进方案:
os.path.isdir 替换为 ais_bench.benchmark.summarizers.swebench.osp.isdir,将 os.path.isfile 替换为 ais_bench.benchmark.summarizers.swebench.osp.isfile,将 mmengine.load 替换为 ais_bench.benchmark.summarizers.swebench.mmengine.load

(注:此建议同样适用于文件中其他测试用例的 @patch 装饰器)

Suggested change
@patch('os.path.isdir')
@patch('os.path.isfile')
@patch('mmengine.load')
@patch('ais_bench.benchmark.summarizers.swebench.model_abbr_from_cfg_used_in_summarizer')
@patch('ais_bench.benchmark.summarizers.swebench.dataset_abbr_from_cfg')
@patch('ais_bench.benchmark.summarizers.swebench.get_infer_output_path')
@patch('ais_bench.benchmark.summarizers.swebench.osp.isdir')
@patch('ais_bench.benchmark.summarizers.swebench.osp.isfile')
@patch('ais_bench.benchmark.summarizers.swebench.mmengine.load')
@patch('ais_bench.benchmark.summarizers.swebench.model_abbr_from_cfg_used_in_summarizer')
@patch('ais_bench.benchmark.summarizers.swebench.dataset_abbr_from_cfg')
@patch('ais_bench.benchmark.summarizers.swebench.get_infer_output_path')



if __name__ == '__main__':
unittest.main() No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

建议:在文件末尾添加一个空行。

根据 PEP 8 编码规范,所有源文件都应该以一个空行(newline)结尾。这可以避免一些工具在处理文件时产生警告,并符合 POSIX 标准。

Suggested change
unittest.main()
unittest.main()

@wanlongze wanlongze changed the title test: add UT coverage for swebench summarizer [UT]: add UT coverage for swebench summarizer Jun 1, 2026
@wanlongze wanlongze changed the title [UT]: add UT coverage for swebench summarizer [UT]: add UT coverage for swebench Jun 2, 2026
@SJTUyh SJTUyh merged commit 5531432 into AISBench:master Jun 2, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants