Skip to content

v1.12.0

Choose a tag to compare

View all tags
@Yunnglin Yunnglin released this 16 Sep 08:15
v1.12.0
f09e55d

中文版

基准测试数据集

  • 语音评测：新增 THCHS-30 国际音标（IPA）音素识别评测，支持 Phone Error Rate（PER）指标。
  • 语音评测：LibriSpeech 新增 test-other 子集支持。

功能增强

  • Agent 评测：新增 DeepSeek Harness Bridge Runner，支持通过 OpenAI Chat Completions Bridge 运行 deepseek-harness。
  • Agent 评测：新增可选的工具调用参数 Schema 校验，帮助区分模型参数生成错误与工具执行错误。
  • Agent 基准：Tau3-Bench 支持配置 Agent 最大执行步数 max_steps。
  • 性能测试：新增基于 AIPerf 的 AgentX 回放场景，支持数据版本校验、运行有效性记录和敏感信息脱敏。
  • 性能测试：新增可选 PD 分离式推理指标，包括 Steady ITL、PD Handoff Latency 和 PD Handoff Overhead。
  • 性能测试：AgentX 支持 tokenizer_trust_remote_code 与 benchmark_grace_period 配置。
  • Dashboard：报告列表支持按模型展示同一模型的多个报告，保持原始评测报告不变。
  • Dashboard：聚合结果表首屏按批次展示，优化大规模模型与基准组合下的页面性能。

文档优化

  • 更新 Agent Bridge、AgentX 性能测试、Tau3-Bench、评测架构和基准贡献说明。
  • 修复基准 smoke evaluation 示例命令，并补充正确的 mock 评测配置。
  • 更新产品网站入口及相关使用指引。

问题修复

  • 修复多参考答案在 inclusion-based 评分中被拼接为单个错误目标的问题，影响 TriviaQA、MMLU-Redux 等评测。
  • 修复 Judge 模板渲染、指标异常处理、包含匹配规则及 GeneralArena bootstrap 随机数处理，提升评测结果的正确性与可复现性。
  • 修复 GeneralArena 导入历史评测结果时按行错配、将已生成回复带入 Judge 输入的问题；评测版本升级至 v1.2。
  • 修复 Agent Bridge 丢失模型原始输出、性能指标和推理内容，以及上游生成失败未记录到 Agent Trace 的问题。
  • 修复 Agent sandbox stdin 隔离和显式 stdin 传递问题。
  • 修复 MCP 初始化错误被隐藏、service worker 无法运行 MCP stdio 客户端的问题。
  • 修复空 ASR 转写被错误计为 WER=0 的问题；Seed-TTS-Eval 评测版本升级至 v1.1。
  • 修复流式推理输出的 TTFT/ITL 统计、多轮 warmup 交接，以及带 token usage 的 Agent Trace 导致 Predictions API 返回 500 的问题。
  • 修复 few-shot 配置未在数据加载前校验、CLI benchmark-info 导入错误、Dashboard 空分数行模型名异常换行等问题。
  • 修复可用模块的导入期异常被误报为缺失依赖的问题，并移除会直接报错的 CLIP Benchmark 模板入口。

English Version

Benchmark Datasets

  • Speech Evaluation: Added THCHS-30 IPA phoneme recognition with Phone Error Rate (PER) support.
  • Speech Evaluation: Added the LibriSpeech test-other subset.

Feature Enhancements

  • Agent Evaluation: Added the DeepSeek Harness Bridge Runner for running deepseek-harness through the OpenAI Chat Completions Bridge.
  • Agent Evaluation: Added optional tool-call argument Schema validation to distinguish invalid model arguments from tool execution failures.
  • Agent Benchmarks: Added configurable Agent max_steps support for Tau3-Bench.
  • Performance Testing: Added an AIPerf-based AgentX replay scenario with dataset-version validation, run-validity records, and secret redaction.
  • Performance Testing: Added optional PD-disaggregated serving metrics, including Steady ITL, PD Handoff Latency, and PD Handoff Overhead.
  • Performance Testing: Added tokenizer_trust_remote_code and benchmark_grace_period options for AgentX.
  • Dashboard: Added display-only grouping of reports from the same model without mutating source evaluation reports.
  • Dashboard: Added incremental rendering for aggregated results to improve large-scale dashboard performance.

Documentation

  • Updated documentation for Agent Bridge, AgentX performance testing, Tau3-Bench, evaluation architecture, and benchmark contributions.
  • Fixed the benchmark smoke-evaluation example and documented the correct mock-evaluation configuration.
  • Updated product website entry points and related guidance.

Bug Fixes

  • Fixed multi-reference answers being flattened into an invalid target in inclusion-based scoring, affecting benchmarks such as TriviaQA and MMLU-Redux.
  • Fixed Judge template rendering, metric-failure handling, inclusion matching, and GeneralArena bootstrap RNG behavior for more correct and reproducible evaluation results.
  • Fixed mispaired imported reviews and generated responses leaking into Judge inputs in GeneralArena; advanced its evaluation version to v1.2.
  • Fixed Agent Bridge loss of original model outputs, performance metrics, reasoning content, and upstream generation failures in Agent Traces.
  • Fixed Agent sandbox stdin isolation and explicit stdin forwarding.
  • Fixed hidden MCP initialization errors and MCP stdio clients failing in service workers.
  • Fixed empty ASR transcripts being scored as WER=0; advanced Seed-TTS-Eval to evaluation version v1.1.
  • Fixed TTFT/ITL timing for streamed reasoning output, multi-turn warmup handoff, and Predictions API 500 errors caused by Agent Traces with token usage.
  • Fixed late few-shot configuration validation, CLI benchmark-info imports, and model-name wrapping for empty Dashboard score rows.
  • Fixed available modules' import-time failures being misreported as missing dependencies, and removed the CLIP Benchmark template entry point that always raised an error.

What's Changed

  • chore: tighten ruff rule baseline by @Yunnglin in #1686
  • fix(perf): hand off multi-turn warmup without draining by @Yunnglin in #1684
  • fix(clip_benchmark): remove main block that always raises TypeError by @Anai-Guo in #1687
  • chore: enforce F401 in evalscope/utils by @Moenupa in #1689
  • fix(dep): remove dependency overrides; type check instead of runtime by @Moenupa in #1692
  • feat(tau3_bench): support configurable max_steps parameter by @ZTJiu in #1690
  • docs(agent): point prediction docstrings at the hook that resolves it by @ChenCJ-io in #1696
  • fix(web): stop score-matrix model name wrapping character-by-character by @Dhru1001 in #1704
  • fix(perf): measure TTFT from reasoning output by @git-jxj in #1700
  • fix(bridge): record upstream generate failures on the agent trace by @ChenCJ-io in #1698
  • fix(mcp): preserve initialization errors and support service stdio by @git-jxj in #1701
  • refactor(metrics): give metric semantics one source per decision by @Yunnglin in #1705
  • fix(eval): correct judge rendering, metric scoring, and bootstrap RNG by @git-jxj in #1702
  • fix(judge): import canonicalize_producer_identity from its new module by @ChenCJ-io in #1711
  • fix(bridge): keep the model output intact on reconstructed assistant messages by @ChenCJ-io in #1710
  • fix(benchmark): validate few-shot capabilities by @Yunnglin in #1714
  • feat(agent): validate tool-call arguments against the advertised schema by @ChenCJ-io in #1712
  • fix(cli): enforce F401 in cli and fix benchmark info import errors by @Moenupa in #1691
  • docs: fix the benchmark smoke evaluation command by @Excelius-Wang in #1721
  • fix(audio): preserve empty ASR transcripts when scoring WER by @Excelius-Wang in #1722
  • fix(agent): stop sandboxed commands from inheriting the evaluator's stdin by @ChenCJ-io in #1718
  • feat(web): group same-model reports in the report list, display-only by @Dhru1001 in #1703
  • fix(service): stop 500ing on predictions whose agent trace carries token usage by @Dhru1001 in #1725
  • perf(web): cap dashboard's aggregated results table to 100 rows up front by @Dhru1001 in #1724
  • feat(perf): add AgentX AIPerf scenario by @Yunnglin in #1716
  • fix(utils): narrow check_import exception handling by @Bruce-Yii in #1720
  • feat: add LibriSpeech test-other subset by @Yunnglin in #1733
  • docs: align benchmark contribution examples with metadata requirements by @BingH225 in #1730
  • fix(agent): forward exec stdin into the sandbox instead of dropping it by @ChenCJ-io in #1719
  • fix(arena): validate review identities before pairing by @elandesberg in #1731
  • fix(perf): forward tokenizer trust option for AgentX by @Xiangyi1996 in #1745
  • feat(agent): add DeepSeek Harness bridge runner by @Yunnglin in #1746
  • feat(benchmarks): add THCHS-30 phoneme recognition by @chenminupup in #1732
  • feat(perf): add optional PD handoff metrics by @gbdjxgp in #1744
  • fix(perf): forward AgentX benchmark grace period by @Xiangyi1996 in #1747
  • fix(evaluator): preserve multi-alias targets for inclusion-based scoring by @Dhru1001 in #1727

New Contributors

Full Changelog: v1.11.1...v1.12.0

Contributors

ZTJiu, elandesberg, and 12 other contributors
Assets 2
Loading