Skip to content

v1.13.0

Latest
Latest

Choose a tag to compare

View all tags
@Yunnglin Yunnglin released this 13 Oct 10:27
v1.13.0
4980276

中文版

基准测试数据集

  • 多轮对话评测：新增 MT-Bench，支持两轮对话和可配置的 Judge，遵循官方评分流程。
  • Agent 评测：新增 Terminal-Bench 4.0（terminal_bench_v4），固定官方数据版本 4.0.0，支持任务筛选、重复试验及独立验证环境。
  • 决策能力评测：新增 ANLI、BoolQ、BANKING77、ContractNLI、RewardBench v1，默认从 ModelScope 加载，支持普通 LLM 和 System One Choice 评测。

功能增强

  • System One Choice：新增 eval_type='systemone_api'，通过现有模型接口支持文本单选决策模型，并保留选项概率、置信度和请求信息。
  • 数学评分：统一使用固定版本 Math-Verify 进行答案提取、解析与等价判断，覆盖 29 个基准，保留各基准的容差、部分得分及多答案规则。
  • 基准加载：适配器改为按需加载，减少 CLI 和库导入时加载整个基准目录的开销。
  • 数据集缓存：Native 加载支持复用 ModelScope SDK 本地快照；优化临时 Arrow 缓存、并发写入和中断恢复。
  • 多模态性能测试：新增真实 MMMU 多图数据模式，保留提示中的图像占位顺序。
  • 多轮性能测试：支持会话 ID 与逐轮请求 ID，SWE-Smith 数据构建支持副本生成，并减少延迟指标收集开销。
  • SLA 自动调优：支持显式 s / ms 阈值单位，校验指标有效性，使用准确请求计数，并改进约束与优化搜索。
  • Dashboard：展示已安装的 EvalScope 版本，并提供本地化说明。

文档优化

  • 更新 System One、Terminal-Bench 4.0、SLA 调优、多轮性能测试和数据缓存使用指南。
  • 新增 DeepSearchQA 的 Parallel Search MCP 示例与 Qwen-Image 文生图评测示例。
  • 修复性能结果分析、函数调用 API key、Python 3.10 示例以及多处外部文档链接。

问题修复

  • 修复 AI2D 图像版本按图而非逐题选择的问题，依据官方 abcLabel 标注选择原图或字母标注图。
  • 修复 IFBench 关键词/姓名子串匹配、部分校验器与上游不一致及无单词回复的除零问题。
  • 修复 AlpacaEval 胜负被计为 0.75/0.25、HallusionBench 分组、OCRBench-v2 类别宏平均、PubMedQA 未识别答案分母及 collection 宏平均问题。
  • 修复 BFCL-v3、ACEBench 和 OCRBench-v2 汇总行重复计入总分/样本数，以及 vote@k 将空答案纳入投票的问题。
  • 修复 C-Eval 答案格式解析、中文多选分隔符、MMLU-Redux 多答案标注和 BBH 自由答案括号保留问题。
  • 修复 NER 重复实体与 BIO 边界、LiveCodeBench sandbox 输出比较和视频压缩包文件匹配问题。
  • 修复流式性能测试中的工具调用 TTFT、推理文本 token 估算、Responses ITL/失败识别/缓存用量、单 token TPOT、分位数精度与 NaN、推测解码估算及 embedding/rerank 响应体延迟统计。
  • 修复 Anthropic 图片 URL 和 thinking/signature 历史保留、Responses 中断重试及自定义 finish reason 处理。
  • 修复 Agent 提交数值 0、临时 HOME 清理、工具 Schema 字面量、缺省样本 metadata、本地音视频路径，以及未测量 PD 指标导致性能详情 API 返回 500 等问题。
  • 修复 Terminal-Bench 的 Codex max_turns 兼容问题，并更新 Full CI 的文本/视觉模型配置。

升级说明

  • 数学评分涉及的 29 个基准、22 个 NER 基准及 AI2D、IFBench 等评分修复均更新了相应评测版本。比较结果时请核对 evaluation_version；旧预测可通过 rerun_review=True / --rerun-review 重新评分，旧 reviews 不能直接跨评测版本复用。
  • AI2D v1.1 使用修正后的逐题图像，结果与 v1.0 不直接可比。Terminal-Bench 4.0 结果也不直接与 2.0/2.1 比较。
  • Math-Verify 使用 ANTLR 4.13.2；当前 ms-vlmeval==0.0.20 要求 ANTLR 4.11.1，因此 vlmeval / all extras 的联合依赖安装存在冲突，请使用独立环境。数学解析与验证不提供硬超时。
  • System One 仅支持文本单选；RewardBench v1 的整体 accuracy 按样本加权，ContractNLI 当前评估文档分类 accuracy，请勿将它们视为完整复现官方排行榜的全部指标。

English Version

Benchmark Datasets

  • Multi-turn Evaluation: Added MT-Bench with two-turn conversations, configurable judges, and the official judging flow.
  • Agent Evaluation: Added Terminal-Bench 4.0 (terminal_bench_v4), pinned to official revision 4.0.0, with task filtering, repeated trials, and separate verifier environments.
  • Decision Evaluation: Added ANLI, BoolQ, BANKING77, ContractNLI, and RewardBench v1, defaulting to ModelScope and supporting both standard LLM and System One Choice evaluation.

Feature Enhancements

  • System One Choice: Added eval_type='systemone_api' for text single-choice decision models through the existing model interface, retaining option probabilities, confidence, and request metadata.
  • Mathematical Scoring: Unified extraction, parsing, and equivalence checking with pinned Math-Verify across 29 benchmarks, preserving benchmark-specific tolerances, partial credit, and multiple-answer policies.
  • Benchmark Loading: Load adapters on demand to reduce full-catalog import overhead for CLI and library use.
  • Dataset Caching: Reuse local ModelScope SDK snapshots in Native loading and improve temporary Arrow storage, concurrent cache publication, and interruption recovery.
  • Multimodal Performance Testing: Added a real MMMU multi-image mode with correct prompt/image placeholder ordering.
  • Multi-turn Performance Testing: Added conversation and per-turn request IDs, SWE-Smith dataset replication, and lower latency-metric collection overhead.
  • SLA Auto-tuning: Added explicit s / ms threshold units, metric-validity checks, exact request-count handling, and improved constraint/optimization searches.
  • Dashboard: Display the installed EvalScope version with localized guidance.

Documentation

  • Updated System One, Terminal-Bench 4.0, SLA auto-tuning, multi-turn performance testing, and dataset-cache guides.
  • Added Parallel Search MCP for DeepSearchQA and a Qwen-Image text-to-image evaluation example.
  • Fixed performance-result analysis, function-call API-key configuration, Python 3.10 examples, and external documentation links.

Bug Fixes

  • Fixed AI2D image selection to follow each question's official abcLabel annotation instead of choosing one image variant per diagram.
  • Fixed IFBench keyword/name substring matching, verifier deviations from upstream, and division by zero for word-less responses.
  • Fixed AlpacaEval win/loss scores of 0.75/0.25, HallusionBench grouping, OCRBench-v2 category averaging, PubMedQA denominators for unrecognized answers, and collection macro averaging.
  • Fixed summary-row double counting in BFCL-v3, ACEBench, and OCRBench-v2, and empty answers participating in vote@k.
  • Fixed C-Eval answer parsing, Chinese multi-select separators, MMLU-Redux multi-answer annotations, and bracket preservation in BBH free-form targets.
  • Fixed repeated NER entities and BIO boundaries, LiveCodeBench sandbox stdout comparison, and exact video archive-member matching.
  • Fixed streamed tool-call TTFT, reasoning-token estimates, Responses ITL/failure/cached-usage handling, one-token TPOT, percentile precision/NaN handling, speculative-decoding estimates, and embedding/rerank response-body latency.
  • Fixed Anthropic image URL and thinking/signature preservation, interrupted Responses retries, and provider-specific finish reasons.
  • Fixed Agent numeric-zero submissions, temporary HOME cleanup, literal tool-schema values, missing sample metadata, local audio/video paths, and performance-detail API errors for unmeasured PD metrics.
  • Fixed Codex max_turns compatibility in Terminal-Bench and updated Full CI's text/vision model configuration.

Upgrade Notes

  • Evaluation versions have changed for the 29 mathematical benchmarks, 22 NER benchmarks, and scoring fixes such as AI2D and IFBench. Check evaluation_version when comparing results. Use rerun_review=True / --rerun-review to rescore saved predictions; old reviews cannot be reused across evaluation versions.
  • AI2D v1.1 uses corrected per-question images and is not directly comparable with v1.0. Terminal-Bench 4.0 results are also not directly comparable with 2.0/2.1.
  • Math-Verify uses ANTLR 4.13.2, while ms-vlmeval==0.0.20 requires 4.11.1. Joint dependency installation with the vlmeval / all extras conflicts; use separate environments. Mathematical parsing and verification have no hard timeout.
  • System One supports text single-choice tasks only. RewardBench v1 reports sample-weighted overall accuracy, and ContractNLI currently evaluates document-classification accuracy; these are not full reproductions of every official leaderboard metric.

What's Changed

  • feat(perf): add real multi-image MMMU dataset mode by @Bruce-Yii in #1729
  • refactor(benchmarks): load adapters on demand instead of importing all 238 at package import by @Yunnglin in #1749
  • feat(bench): add MT-Bench by @Yunnglin in #1750
  • fix(metrics): parse Chinese multi-select answers separated by 、 or / by @feizhuzheng in #1751
  • fix(perf): preserve MMMU image placeholder order by @git-jxj in #1754
  • fix(agent): prevent zero answers from falling back to assistant text by @git-jxj in #1753
  • fix(ifbench): count keywords and person names on word boundaries by @feizhuzheng in #1755
  • fix(ifbench): align remaining upstream verifiers by @Yunnglin in #1756
  • fix(agent): remove the temporary HOME after codex/hermes/gemini-cli runs by @Linxiushen in #1758
  • fix: give logger calls a placeholder for their extra positional arguments by @Linxiushen in #1757
  • fix(alpaca_eval): count a win as 1 and a loss as 0 by @jiangLLM in #1771
  • fix(benchmarks): match archive members exactly in mvbench/videomme_v2 video lookup by @feizhuzheng in #1752
  • feat(math): unify mathematical scoring with Math-Verify by @Yunnglin in #1785
  • fix(models): preserve provider finish reasons in streamed responses by @1fanwang in #1777
  • feat(web): display the installed EvalScope version by @git-jxj in #1784
  • docs: update moved Qwen and ms-swift readthedocs links (EN and ZH) by @pratikgx in #1782
  • docs(t2i): add Qwen-Image example to text-to-image best practice by @Bruce-Yii in #1768
  • fix(perf): stop perf/detail 500 for runs without PD metrics by @Yunnglin in #1790
  • fix(bbh): keep brackets in free-form targets such as dyck_languages by @Lostmanu in #1773
  • fix(ifbench): guard StopWordPercentageChecker against word-less responses by @feizhuzheng in #1761
  • fix(hallusion_bench): align fAcc/qAcc grouping with official evaluation by @feizhuzheng in #1769
  • fix(multi-choice): accept spaces between labels in Chinese multi-select answers by @RizgarOzan in #1789
  • fix(perf): keep a rerank request successful when the top score is not a number by @Linxiushen in #1760
  • fix(perf): tolerate non-JSON SSE terminators; route per-turn max_tokens per protocol by @Yunnglin in #1791
  • fix(perf): include response body time in embedding and rerank latency by @git-jxj in #1763
  • fix(tools): preserve literal values in JSON schemas by @git-jxj in #1765
  • fix(models): preserve Anthropic image URL sources by @git-jxj in #1764
  • fix(web): localize version badge tooltip by @git-jxj in #1796
  • fix(perf): support explicit units in SLA thresholds by @David-Wu1119 in #1793
  • test(perf): assert p99 siblings of unmeasured optional metrics are pruned by @RizgarOzan in #1797
  • fix(perf): leave NaN samples out of percentile columns by @David-Wu1119 in #1798
  • docs: fix API key lookup in function calling examples by @git-jxj in #1801
  • docs: fix benchmark examples for Python 3.10 by @git-jxj in #1802
  • docs: fix moved links in full-stack, RAG and OpenCompass guides by @pratikgx in #1799
  • docs: fix performance result analysis examples by @git-jxj in #1803
  • fix(halueval): read Yes/No verdicts as whole words, not substrings by @Zhuoxi2000 in #1800
  • fix(perf): stop treating rerank search units as tokens by @git-jxj in #1811
  • fix(perf): fail the SLA success gate on any failed request by @David-Wu1119 in #1810
  • fix(perf): keep runs with failed requests out of SLA optimization by @David-Wu1119 in #1812
  • fix(ocr_bench_v2): write text spotting files as UTF-8 by @RizgarOzan in #1804
  • fix(dataset): default missing sample metadata to an empty dict by @March-7 in #1806
  • chore(perf): enforce F401 and remove unused imports by @loyce-cheng in #1805
  • docs: require benchmark registry index regeneration by @git-jxj in #1807
  • fix(models): retry interrupted Responses API streams by @git-jxj in #1808
  • fix(perf): count Responses protocol failures as failed requests by @git-jxj in #1809
  • fix(benchmarks): read Yes/No verdicts as whole words in pope, hallusion_bench, drivel_binary by @Yunnglin in #1816
  • fix(perf): leave one-token replies out of TPOT by @David-Wu1119 in #1814
  • fix(perf): keep millisecond resolution in latency percentiles by @David-Wu1119 in #1815
  • fix(perf): count streamed tool calls as output for TTFT by @David-Wu1119 in #1813
  • fix(perf): harden SLA auto-tuning while preserving summary format by @Yunnglin in #1818
  • fix(perf): count reasoning text in the tokenizer fallback by @David-Wu1119 in #1819
  • fix(perf): count empty-delta chunks in the speculative decoding estimate by @David-Wu1119 in #1820
  • fix(report): flag BFCL-v3, ACEBench and OCRBench-v2 summary rows as aggregates by @David-Wu1119 in #1824
  • fix(collection): average macro_avg. over (dataset, subset) members by @David-Wu1119 in #1823
  • fix(ocr_bench_v2): average EN, CN and OVERALL over categories, not samples by @David-Wu1119 in #1825
  • fix(dataset): reuse the default ModelScope cache for Native loading by @sfeng1996 in #1795
  • feat: support System One Choice models and decision benchmarks by @Yunnglin in #1829
  • fix(perf): measure Responses ITL between output deltas by @git-jxj in #1826
  • fix(messages): absolutise local audio and video paths in markdown output by @D41910 in #1833
  • fix(pubmedqa): include unrecognized answers in recall and F1 denominators by @mavericksea-ai in #1828
  • fix(multi-choice): accept a full-width comma between unbracketed answer labels by @D41910 in #1832
  • fix(perf): preserve cached usage in incomplete Responses streams by @heranliu in #1831
  • feat: add Terminal-Bench 4.0 support by @Yunnglin in #1835
  • fix(ner): preserve entity occurrences and BIO boundaries by @git-jxj in #1834
  • fix(models): preserve Anthropic thinking in streams and history by @git-jxj in #1840
  • fix(live_code_bench): compare sandbox stdio output like the local grader by @David-Wu1119 in #1843
  • fix(ceval): parse the answer line with the shared Chinese parser by @David-Wu1119 in #1842
  • fix(mmlu_redux): read space-separated multiple-answer annotations by @David-Wu1119 in #1844
  • perf: add conversation IDs and replicas, optimize latency metrics for multi-turn benchmark by @syuoni in #1845
  • fix(metrics): ignore failed extractions in vote@k majority vote by @li872 in #1839
  • fix(terminal_bench): skip max_turns for Codex agent (Harbor 0.24.0 compatibility) by @zzx04-cool in #1846
  • Add a Parallel Search MCP example for DeepSearchQA by @georgeatparallel in #1841
  • fix(ai2d): use per-question images per official abcLabel semantics (v1.1) by @Yunnglin in #1848

New Contributors

Full Changelog: v1.12.0...v1.13.0

Contributors

pratikgx, syuoni, and 20 other contributors
Assets 2
Loading