Skip to content

v1.14.0

Latest
Latest

Choose a tag to compare

View all tags
@Yunnglin Yunnglin released this 20 Oct 10:31
v1.14.0
1c9d729

中文版

基准测试数据集

  • 决策评测：新增 JevBench、OpenDecisions、STS-B 和自定义 JSONL 数据集 general_decision，覆盖 Choice、Noul 和 Score 任务。
  • 代码与数据库评测：新增 DS-1000 数据科学代码生成和 BIRD-SQL Mini-Dev 文本转 SQL 评测，使用执行结果评分。
  • 表格与金融评测：新增 TableBench、FinQA 和 TAT-QA，分别支持官方 Mix_Metric、程序执行/等价性评分及考虑数值尺度的 EM/F1。
  • 空间推理评测：新增 SPaRC，评估二维网格中的路径规划和规则约束满足能力。
  • 多模态评测：新增 VisualOverload，保留官方提示与原始图像，并导出 visualoverload_submission.json 供官方服务器评分。

功能增强

  • 决策模型：扩展 System One 并新增 AutoTrust API，统一校验结构化决策请求、答案类型与概率分布；支持 BoolQ 的 Noul 转换和 BANKING77 的固定语义类别。
  • 概率指标：新增分类、概率校准、相关性、序数评分及风险/覆盖率与 AURC 分析；保留原始概率，并单独记录用于评分的分布。
  • 决策可视化与压测：Web UI 支持查看逐样本选项概率、预测与参考答案；新增 perf --api systemone / autotrust 及 open_decisions 压测数据集。
  • 数据集加载：通过共享 DatasetSource 自动解析已声明的 Hugging Face 数据集 ID、版本与文件路径；覆盖基准加载及相关压测插件，保留本地路径优先级。
  • 共享前缀池：新增 --prefix-count，按请求索引轮询分配随机 token 前缀；固定配置与种子时，串行和多进程生成保持一致。
  • 报告展示：CLI、HTML 和 Web 指标标签显示 Context、Depth、Threshold、Min/Max tokens 等数值维度名称。

文档优化

  • 新增中英文决策模型评测实践、自定义决策数据、API 协议及各新增基准说明。
  • 更新共享前缀池、跨数据源加载和指标维度的用法及示例。
  • 修正离线模型下载命令、mini-swe-agent 和 ms-swift 链接，并同步 AI2D 中文说明。

问题修复

  • 修复 Needle Haystack 动态指标名导致的语义缺失、主指标缺失与热力图生成失败，按 context/depth 维度保留分格评分。
  • 修复 MMBench 的 nan 提示及提示与问题拼接；修复 MMMU 开放题数值与多参考答案评分。
  • 修复 BigCodeBench / BigCodeBench-Hard 仅定义而未执行 TestCases，导致错误代码被判为通过的问题。
  • 修复 ZebraLogicBench 未解析答案未计入部分分母及多行 JSON 解析；修复 ToolBench 不适用指标被按零分参与平均的问题。
  • 修复 OlympiadBench 全角逗号答案、数学标记包裹的选择题标签、加粗或不同大小写的 ANSWER: 行，以及 ANLS 对 JSON 字面量参考答案的处理。
  • 修复 OmniDocBench 小样本未同时覆盖中英文时的缺失指标访问，以及 general_mcq 自定义提示模板被默认值覆盖的问题。
  • 修复 Seed-TTS-Eval 混合语言样本共用首个样本的 WER 语言配置，以及无法评分的音频被计为完美 WER 0 的问题。
  • 修复 CLIP 零样本分类在新 NumPy 下的标量转换、limit 截断和 skip_existing 提前终止整个任务的问题。
  • 修复一条记录生成多个样本时，repeats 将不同问题分到同一重复组，影响 pass@k / vote@k 的问题。
  • 修复预测缓存恢复时同一样本重复评分，保留每个样本最后一条有效预测；支持带 UTF-8 BOM 的本地 CSV、TSV 和 JSONL。
  • 修复并发数、超时、流式开关及重试等传输配置变化导致缓存评测标识不必要变化的问题。
  • 修复逐次调用的 GenerateConfig 默认值覆盖用户生成配置，以及 LiteLLM 流式延迟未包含流读取时间的问题。
  • 修复 HTTP 200 的 SSE 流中错误事件被计为成功请求，以及批量评分子集全部失败时聚合崩溃的问题。
  • 修复 Web 页面刷新后丢失活动任务监控，恢复进度、日志与停止操作；仅在会话存储中保留任务 ID。
  • 修复工具 Schema 对 dataclass 字符串类型注解的解析，排除伪字段，并在类型注解无法解析时保留字段及回退行为。

升级说明

  • 多项基准的评测语义已更新，请核对 evaluation_version。MMBench v1.1 改变了输入提示，需重新生成预测；评分类修复可使用 rerun_review=True / --rerun-review 重新评分。
  • 缓存标识对生成配置做了规范化。升级后旧缓存可能需要一次 rerun_review=True；之后仅调整并发或传输参数不会使相同评测请求失配。
  • VisualOverload 不提供本地答案准确率；response_present 仅表示响应是否非空。请使用 repeats=1，并将完整提交文件上传至官方评分服务。
  • TableBench 当前包含 836 道文本答案题，不包含需要执行绘图代码的 Visualization 题。
  • DS-1000 默认在本地子进程执行代码，不构成安全沙箱。运行不可信代码请启用 sandbox，并使用对应依赖环境；TensorFlow 子集的预置容器要求 Linux/amd64。
  • 决策 API 仅支持文本任务；AutoTrust 的 Score 协议固定为六级（0-5），不会隐式重映射其他评分尺度。

English Version

Benchmark Datasets

  • Decision Evaluation: Added JevBench, OpenDecisions, STS-B and custom JSONL datasets through general_decision, covering Choice, Noul and Score tasks.
  • Code and Database Evaluation: Added DS-1000 data science code generation and BIRD-SQL Mini-Dev text-to-SQL evaluation with execution-based scoring.
  • Table and Financial Evaluation: Added TableBench, FinQA and TAT-QA, using the official Mix_Metric, program execution/equivalence metrics, and scale-aware EM/F1 respectively.
  • Spatial Reasoning: Added SPaRC for two-dimensional grid pathfinding under rule constraints.
  • Multimodal Evaluation: Added VisualOverload with official prompts, original images and visualoverload_submission.json export for official server-side grading.

Feature Enhancements

  • Decision Models: Extended System One and added AutoTrust APIs with shared validation of typed requests, answer kinds and probability distributions. Added BoolQ Noul conversion and stable semantic class identities for BANKING77.
  • Probability Metrics: Added classification, calibration, correlation, ordinal scoring, risk/coverage and AURC analysis. Raw probabilities remain available separately from scoring distributions.
  • Decision UI and Performance Testing: Inspect per-sample probabilities, predictions and references in the Web UI. Added perf --api systemone / autotrust and the open_decisions performance dataset.
  • Dataset Loading: Shared DatasetSource resolves declared Hugging Face dataset IDs, revisions and file paths for benchmarks and related performance plugins, while preserving local-path precedence.
  • Shared Prefix Pools: Added --prefix-count to assign random token prefixes round-robin by request index. Serial and spawned-process generation produce the same ordered inputs for a fixed configuration and seed.
  • Report Labels: CLI, HTML and Web labels now name numeric dimensions such as Context, Depth, Threshold and Min/Max tokens.

Documentation

  • Added bilingual decision evaluation practices, custom-data and protocol guides, and documentation for the new benchmarks.
  • Updated shared-prefix, cross-hub loading and metric-dimension examples.
  • Corrected offline model download commands and mini-swe-agent/ms-swift links, and synchronized the Chinese AI2D description.

Bug Fixes

  • Fixed Needle Haystack metric semantics, primary metric selection and heatmaps by retaining context/depth cells as structured dimensions.
  • Fixed MMBench nan hints and hint/question formatting, and MMMU open-question numerical and multi-reference scoring.
  • Fixed BigCodeBench and BigCodeBench-Hard defining but not executing TestCases, which incorrectly passed wrong solutions.
  • Fixed ZebraLogicBench unanswered-sample denominators and multi-line JSON parsing, and ToolBench averaging inapplicable metrics as zeros.
  • Fixed OlympiadBench full-width comma answers, math-wrapped choice labels, bold/case-varied ANSWER: lines, and ANLS JSON-literal reference handling.
  • Fixed missing-language metric access in small OmniDocBench runs and custom general_mcq templates being replaced by defaults.
  • Fixed shared WER language settings across Seed-TTS-Eval samples and unscorable audio receiving a perfect WER of zero.
  • Fixed CLIP zero-shot classification scalar conversion with newer NumPy, exact limit handling and skip_existing ending the entire run.
  • Fixed repeats grouping different questions together when one record produces multiple samples, affecting pass@k / vote@k.
  • Fixed duplicate prediction-cache restoration by keeping the last valid prediction per sample. Added UTF-8 BOM support for local CSV, TSV and JSONL files.
  • Fixed unnecessary cache identity changes caused by concurrency, timeout, streaming and retry settings.
  • Fixed per-call GenerateConfig defaults overriding user settings and LiteLLM streaming latency excluding stream consumption.
  • Fixed SSE error events inside HTTP 200 responses counting as success and aggregation crashing when every review in a batch-scored subset fails.
  • Fixed active task monitoring after a Web page reload, restoring progress, logs and Stop controls while storing only task IDs in session storage.
  • Fixed dataclass string annotations in tool schemas, excluded pseudo-fields and retained graceful fallbacks for unresolvable annotations.

Upgrade Notes

  • Evaluation semantics changed for several benchmarks; check evaluation_version. MMBench v1.1 changes input prompts and requires new predictions. Scoring-only fixes can use rerun_review=True / --rerun-review to rescore saved predictions.
  • Generation settings are now canonicalized for cache identity. Older caches may require a one-time rerun_review=True after upgrading; subsequent concurrency/transport-only changes do not invalidate otherwise identical evaluation requests.
  • VisualOverload has no local answer-accuracy metric: response_present only checks non-empty responses. Use repeats=1 and submit the complete export to the official grader.
  • TableBench currently covers 836 text-answer questions, excluding Visualization questions that require executing chart-generation code.
  • DS-1000 runs code in local subprocesses by default, not a security sandbox. Enable sandbox execution for untrusted code and use the required dependency environment. The bundled TensorFlow environment requires Linux/amd64.
  • Decision APIs support text tasks only. AutoTrust Score uses exactly six levels (0-5), without implicit remapping of other scales.

What's Changed

  • fix(mm_bench): drop 'nan' hints and separate hint from question (v1.1) by @surgan12 in #1852
  • fix(needle_haystack): keep context length and depth as metric dimensions by @Thanatos9404 in #1851
  • feat(benchmark): add SPaRC spatial pathfinding benchmark by @lkaesberg in #1853
  • fix(multi_choices): read labels wrapped in math markup after the answer marker by @David-Wu1119 in #1857
  • fix(mmmu): score open questions like the official MMMU evaluation by @David-Wu1119 in #1858
  • feat(benchmark): add VisualOverload submission export by @paulgavrikov in #1855
  • fix(bigcodebench): run the TestCases suite when scoring by @David-Wu1119 in #1859
  • fix(metrics): preserve literal JSON references in ANLS by @git-jxj in #1864
  • fix(omnidoc_bench): handle missing language metrics in small samples by @pumpkin12135 in #1865
  • fix(olympiad_bench): read full-width commas in answers as separators by @David-Wu1119 in #1860
  • fix(metrics): name numeric dimensions in metric labels by @Thanatos9404 in #1856
  • feat: auto-map benchmark dataset ids when loading from the Hugging Face hub by @Yunnglin in #1867
  • fix(benchmarks): read bold and lower-case ANSWER lines in open-answer benchmarks by @David-Wu1119 in #1870
  • fix(general_mcq): keep a prompt_template passed in dataset_args by @David-Wu1119 in #1871
  • fix(evaluator): skip a batch-scored subset whose reviews all failed by @David-Wu1119 in #1874
  • fix(models): time litellm streaming requests after reading the stream by @David-Wu1119 in #1875
  • fix(tool_bench): score each step only with the metrics that apply to it by @David-Wu1119 in #1872
  • Read local dataset files that start with a UTF-8 BOM by @PerryLink in #1877
  • fix(zebralogicbench): count unparsed answers and read multi-line JSON by @David-Wu1119 in #1869
  • fix(cache): deduplicate restored predictions by sample ID by @ningyixi in #1876
  • fix(dataset): repeat samples, not records, so a repeat group is one question by @David-Wu1119 in #1885
  • feat: add decision benchmarks, probability metrics and native APIs by @Yunnglin in #1866
  • fix(versioning): keep eval_batch_size out of the evaluation identity by @David-Wu1119 in #1884
  • feat(benchmark): add BIRD-SQL Mini-Dev text-to-SQL benchmark with execution accuracy by @flyingdoubleG in #1879
  • feat(perf): support round-robin shared prefix pools for random workloads by @changan593 in #1873
  • feat(benchmark): add TableBench table QA benchmark with the official Mix_Metric by @flyingdoubleG in #1881
  • fix(model): merge only the fields a per-call GenerateConfig sets by @David-Wu1119 in #1886
  • fix(perf): count an error sent inside a 200 SSE stream as a failed request by @David-Wu1119 in #1887
  • fix(seed_tts_eval): score each sample in its own language by @David-Wu1119 in #1888
  • fix(web): restore active task monitoring after reload by @git-jxj in #1889
  • feat(benchmark): add DS-1000 data science code generation benchmark by @flyingdoubleG in #1883
  • fix(seed_tts_eval): score unscorable audio as WER 1.0, not a perfect 0 by @David-Wu1119 in #1892
  • docs: fix mini-swe-agent and ms-swift links in third-party docs by @pratikgx in #1900
  • docs: fix offline model download commands by @git-jxj in #1901
  • fix(clip_benchmark): three zero-shot classification and skip_existing bugs by @David-Wu1119 in #1891
  • feat(benchmark): add FinQA financial numerical reasoning benchmark by @flyingdoubleG in #1896
  • fix(tool): resolve dataclass annotations in tool schemas by @xlb3055 in #1898
  • feat(benchmark): add TAT-QA hybrid table + text financial QA benchmark by @flyingdoubleG in #1895

New Contributors

Full Changelog: v1.13.0...release/1.14

Contributors

pratikgx, paulgavrikov, and 12 other contributors
Assets 2
Loading