v1.9.0

中文版

基准测试数据集

• 长上下文与记忆评测: 新增 LongMemEval、LoCoMo QA 等长上下文记忆类基准测试
• Agent 与代码评测: 新增 BrowseComp、SWE-bench Multilingual agentic、BigCodeBench、BigCodeBench-Hard、GDPval、MCP-Atlas、SkillsBench、DeepSWE 等 Agent、代码和工具使用能力评测
• 多模态评测: 新增 ERQA、WorldVQA、CharXiv、BabyVision、EmbSpatial-Bench、MeasureBench 等多模态基准测试
• 数学与推理评测: 新增 arxivmath、cmath、hmmt26、imo_answerbench、AGIEval、ARC-AGI-2、KINA 等数学和通用推理基准测试
• 办公与文档评测: 新增 OfficeQA 办公场景问答评测

功能增强

• Agent Runner: 新增 OpenCode 和 OpenHands runner，并提供对应 Dockerfile 支持
• SkillsBench: 新增原生 SkillsBench runner 支持
• Agent API: 将 run_agent_loop 移动到 evalscope.api.agent，作为公开 API 使用
• 适配器架构: 重构 benchmark adapter 架构，新增 AudioLanguageAdapter、统一 FunctionCallAdapter，并合并 AgentLoopAdapter
• 性能测试: 支持 CPU 密集型请求生成并行化，提升 perf 请求构造效率
• 性能测试数据加载: 统一通过 --data-source 参数加载 perf 数据源，简化多数据集配置

文档优化

• 新增和更新多个 benchmark 文档、supported dataset 列表、Agent 使用文档和 SkillsBench 第三方文档
• 更新性能测试相关参数文档和多轮压测说明
• 更新 ThinkEval 相关最佳实践文档

问题修复

• 修复 SWE-bench 架构选择问题和 sandbox 登录 shell 执行问题
• 修复 ASR 评测中 filters 未在 WER 评分前生效的问题
• 修复 perf 在 OpenAI usage 缺少 completion_tokens 时的兼容性问题
• 修复 perf 多轮压测绝对时间速率调度和 event loop 关闭问题
• 修复 LiveCodeBench stdin buffer 支持问题
• 修复 ThinkBench 对新 ReviewResult 格式的适配问题
• 修复 GPQA answer choices 被括号清理正则错误截断的问题
• 修复 OpenAI-compatible streaming 聚合问题
• 修复 tau3_bench 在 completion usage 缺失时崩溃的问题

---

English Version

Benchmark Datasets

• Long-context and Memory Evaluation: Added LongMemEval, LoCoMo QA and other long-context memory benchmarks
• Agent and Code Evaluation: Added BrowseComp, SWE-bench Multilingual agentic, BigCodeBench, BigCodeBench-Hard, GDPval, MCP-Atlas, SkillsBench, DeepSWE and other agent, coding, and tool-use benchmarks
• Multimodal Evaluation: Added ERQA, WorldVQA, CharXiv, BabyVision, EmbSpatial-Bench, MeasureBench and other multimodal benchmarks
• Math and Reasoning Evaluation: Added arxivmath, cmath, hmmt26, imo_answerbench, AGIEval, ARC-AGI-2, KINA and other math and reasoning benchmarks
• Office and Document Evaluation: Added OfficeQA for office-scenario question answering

Feature Enhancements

• Agent Runners: Added OpenCode and OpenHands runners with Dockerfile support
• SkillsBench: Added native SkillsBench runner support
• Agent API: Moved run_agent_loop to evalscope.api.agent as a public API
• Adapter Architecture: Refactored benchmark adapter architecture with AudioLanguageAdapter, unified FunctionCallAdapter, and merged AgentLoopAdapter
• Performance Testing: Parallelized CPU-bound request generation to improve perf workload preparation
• Performance Data Loading: Unified perf dataset loading through the --data-source parameter for simpler dataset configuration

Documentation

• Added and updated benchmark documentation, supported dataset lists, Agent user guides, and SkillsBench third-party documentation
• Updated performance testing parameter documentation and multi-turn stress testing guides
• Updated ThinkEval best practice documentation

Bug Fixes

• Fixed SWE-bench architecture selection and login shell execution in sandboxes
• Fixed ASR filters so they are applied before WER scoring
• Fixed compatibility when OpenAI usage blocks are missing completion_tokens
• Fixed absolute-time rate scheduling and event loop closing for multi-turn perf tests
• Fixed stdin buffer support in LiveCodeBench
• Fixed ThinkBench compatibility with the new ReviewResult format
• Fixed GPQA answer choice corruption caused by bracket-stripping regex
• Fixed OpenAI-compatible streaming aggregation
• Fixed tau3_bench crash when completion usage is missing

What's Changed

• fix SWE-bench architecture selection by @Yunnglin in #1419
• Add LongMemEval benchmark by @Yunnglin in #1420
• Add LoCoMo QA benchmark by @Yunnglin in #1422
• Add BrowseComp benchmark support by @haoruilee in #1421
• Add SWE-bench Multilingual agentic benchmark by @Yunnglin in #1426
• feat: add BigCodeBench and BigCodeBench-Hard benchmark support by @Yunnglin in #1425
• refactor: move run_agent_loop to evalscope.api.agent as public API by @Yunnglin in #1427
• refactor: restructure adapter architecture - AudioLanguageAdapter, FunctionCallAdapter, merge AgentLoopAdapter by @Yunnglin in #1428
• Fix ASR filters before WER scoring by @haoruilee in #1431
• fix(perf): tolerate missing completion_tokens in OpenAI usage block by @angelynaye in #1432
• feat(agent): add OpenCode and OpenHands runners with Dockerfiles by @Yunnglin in #1429
• fix: support stdin buffer in LiveCodeBench by @Yunnglin in #1438
• fix: adapt thinkbench to new ReviewResult format by @Yunnglin in #1439
• fix(perf): absolute-time rate scheduling for multi-turn + close event loop by @Yunnglin in #1442
• Add GDPval benchmark integration by @Yunnglin in #1441
• (feat) Parallelize CPU-bound perf request generation by @haoruilee in #1440
• fix(gpqa): remove bracket-stripping regex that corrupts answer choices by @Yunnglin in #1448
• Add MCP-Atlas benchmark integration by @Yunnglin in #1445
• feat(perf): unify dataset loading with --data-source parameter by @Yunnglin in #1449
• feat: add ERQA and WorldVQA benchmarks by @Yunnglin in #1453
• Fix OpenAI-compatible streaming aggregation by @kaede316 in #1450
• feat(benchmarks): add CharXiv and BabyVision benchmarks by @Yunnglin in #1454
• feat(benchmarks): add arxivmath, cmath, hmmt26 and imo_answerbench benchmarks by @Yunnglin in #1455
• feat(benchmarks): add officeqa, agieval, arc_agi_2 benchmarks by @Yunnglin in #1458
• fix(agent): use login shell for SWE-bench sandboxes by @haoruilee in #1457
• feat(benchmark): add DeepSWE adapter by @Yunnglin in #1459
• feat(benchmarks): add EmbSpatial-Bench, KINA, and MeasureBench benchmarks by @Yunnglin in #1460
• Fix tau3_bench crash when completion usage is missing by @danielliu99 in #1462
• Add native SkillsBench runner support by @Yunnglin in #1451

New Contributors

• @angelynaye made their first contribution in #1432
• @kaede316 made their first contribution in #1450
• @danielliu99 made their first contribution in #1462

Full Changelog: v1.8.1...v1.9.0