Skip to content

v1.11.0

Latest
Latest

Choose a tag to compare

View all tags
@Yunnglin Yunnglin released this 24 Aug 12:24
v1.11.0
f291a80

中文版

基准测试数据集

  • 多模态评测：新增 PerceptionBench、ScreenSpot-Pro、PMC-VQA、LogicVista、CC-OCR-V2、SLAKE、olmOCR-Bench、CountQA 等视觉、多模态与文档理解基准。
  • 推理与专业能力评测：新增 HiPhO 高中物理奥赛、PhyX 物理推理（选择题与开放题）、PLawBench 法律实践能力评测。
  • 多语言评测：新增 Milu、ARC-Indic、GSM8K-Indic、IndicBoolQ、TriviaQA-Indic、IndicPara、Sanskriti、Hindi HellaSwag，以及 BhashaBench / BhashaBench-Multi 的金融、法律、农业和阿育吠陀子集。

功能增强

  • 评测版本管理：新增原生评测版本与缓存身份校验，并支持确定性的选择题选项打乱，避免语义变更后复用不兼容缓存。
  • Judge 评测：统一 LLM Judge 的 JSON 输出契约；格式不合法或 [ERROR] 回复将从指标统计中排除。
  • 自定义数据集：general_vmcq 统一支持图片、视频、音频输入，并支持 Parquet 和二进制媒体字段；general_vqa 新增媒体占位符支持。
  • 指标与报告：统一指标语义和评测报告展示，优化 Agent Trace 的步骤分组及工具调用与结果关联。
  • 服务与 Web：评测界面支持 Sandbox 配置；优化报告列表元数据读取和条件请求。

文档优化

  • 更新文本生成图像任务的指标选择说明。
  • 修复 API 消息、工具调用和 Tau-bench pass^k 等文档说明问题。

问题修复

  • 修复 IFBench NLTK 英文词性标注模型下载、ACEBench 官方评分协议对齐、IFEval 与 GPQA 结果可复现性等问题。
  • 修复多选题括号答案和多答案场景下的标签提取问题。
  • 修复 TaskConfig 未知字段被静默忽略的问题，提供相近字段建议；同时修复 reasoning_effort 参数透传。
  • 修复 CMMMU 等视觉基准的媒体输入归一化、OmniDocBench 重复加载、VQA 信息读取等问题。
  • 修复终端评测非法 reward 与未完成运行的结果报告问题，以及 BFCL 空工具调用丢失问题。
  • 修复性能测试中的流式 usage 统计、空 content、请求构建失败、缺失 chat template 和插件返回空数据集等问题。
  • 修复 Web 报告中的本地媒体渲染、空 reasoning_tokens 与后端空值处理问题。

English Version

Benchmark Datasets

  • Multimodal Evaluation: Added PerceptionBench, ScreenSpot-Pro, PMC-VQA, LogicVista, CC-OCR-V2, SLAKE, olmOCR-Bench, CountQA, and other vision, multimodal, and document-understanding benchmarks.
  • Reasoning and Domain Evaluation: Added HiPhO for high-school physics Olympiad problems, PhyX for multiple-choice and open-ended physical reasoning, and PLawBench for legal practice evaluation.
  • Multilingual Evaluation: Added Milu, ARC-Indic, GSM8K-Indic, IndicBoolQ, TriviaQA-Indic, IndicPara, Sanskriti, Hindi HellaSwag, and BhashaBench / BhashaBench-Multi finance, legal, agriculture, and Ayurveda subsets.

Feature Enhancements

  • Evaluation Versioning: Added native evaluation versioning, cache identity checks, and deterministic choice shuffling to prevent incompatible cached predictions from being reused.
  • Judge Evaluation: Unified LLM-judge JSON output contracts; malformed and [ERROR] responses are excluded from metric aggregation.
  • Custom Datasets: Unified image, video, and audio inputs in general_vmcq, with Parquet and binary media-field support; added media placeholders to general_vqa.
  • Metrics and Reports: Unified metric semantics and evaluation reports, with improved Agent Trace step grouping and tool-call/result linking.
  • Service and Web: Added Sandbox configuration to the evaluation UI and improved report-list metadata retrieval with conditional requests.

Documentation

  • Clarified metric selection for text-to-image tasks.
  • Fixed documentation issues for API messages, tool calls, and Tau-bench pass^k.

Bug Fixes

  • Fixed IFBench NLTK English tagger downloads, ACEBench official-protocol alignment, and reproducibility issues in IFEval and GPQA.
  • Fixed answer-label extraction for bracketed and multiple-choice responses.
  • Fixed silently ignored unknown TaskConfig keys with close-match suggestions, and fixed reasoning_effort parameter forwarding.
  • Fixed media-input normalization across CMMMU and other vision benchmarks, repeated OmniDocBench loading, and VQA information handling.
  • Fixed invalid-reward and incomplete-run reporting for terminal benchmarks, and BFCL empty tool-call handling.
  • Fixed streaming usage accounting, empty content, request-building failures, missing chat templates, and empty plugin datasets in performance testing.
  • Fixed local media rendering, null reasoning_tokens, and backend null handling in Web reports.

What's Changed

  • fix(ifbench): download correct NLTK English tagger by @git-jxj in #1539
  • fix(acebench): align with the official protocol and fix milestone scoring by @Yunnglin in #1544
  • fix: multi-choice answer extraction, ifeval determinism, LiveCodeBench system prompt, tau-bench pass^k docs by @Yunnglin in #1549
  • feat(perception_bench): add PerceptionBench atomic visual perception benchmark by @Yunnglin in #1551
  • feat(screenspot_pro): add ScreenSpot-Pro GUI grounding benchmark by @Yunnglin in #1550
  • feat(plawbench): add PLawBench rubric-based legal practice benchmark by @Yunnglin in #1553
  • feat(pmc_vqa): add PMC-VQA medical visual question answering benchmark by @Yunnglin in #1557
  • feat(hipho): add HiPhO high school physics Olympiad benchmark by @Yunnglin in #1558
  • fix(config): pass reasoning_effort through instead of whitelisting it by @Yunnglin in #1561
  • fix(multi_choices): parse answer labels the model wrapped in brackets by @Yunnglin in #1560
  • feat(logic_vista): add LogicVista visual logical reasoning benchmark by @Yunnglin in #1556
  • fix(web): render local-path media in both dashboard chains by @Yunnglin in #1562
  • feat(cc_ocr_v2): add CC-OCR-V2 real-world document OCR benchmark by @Yunnglin in #1559
  • feat(web): add sandbox configuration to evaluation UI by @Dhru1001 in #1545
  • fix: validate service output directory by @heliubj18 in #1554
  • fix(perf): explain how to proceed when a tokenizer has no chat template by @Yunnglin in #1564
  • Fix async loop shutdown timeout cascade by @Yunnglin in #1566
  • fix(web): allow null reasoning_tokens in report content blocks by @Dhru1001 in #1563
  • fix(web): normalize backend null to undefined at the API validation boundary by @Yunnglin in #1567
  • feat(metrics): unify metric semantics and evaluation reporting by @Yunnglin in #1552
  • refactor(web): isolate agent trace grouping from rendering by @Yunnglin in #1568
  • fix(perf): surface fatal request-building errors instead of swallowing or retrying them by @Yunnglin in #1571
  • fix(perf tests): repair stale local-endpoint perf tests (hang + outdated return contract) by @Yunnglin in #1573
  • fix(perf): abort request generation when a plugin returns None for the whole dataset by @Yunnglin in #1574
  • fix(perf): parse streaming usage independently of choices to fix 0 ca… by @OctoberGitHub in #1572
  • fix(judge): fail closed on [ERROR] judge responses and surface silent extraction paths by @YuhaoLin2005 in #1576
  • fix(gpqa): seed choice shuffle from the question so rerun-review is reproducible by @Yunnglin in #1583
  • fix(agent): limit SWE-bench toolcall nudge to once by @Roovelrz in #1581
  • feat(io): support parquet and binary image features in general_vmcq by @Moenupa in #1584
  • fix(agent): make AgentLoop own the nudge count and give models an honest reminder by @Yunnglin in #1585
  • fix: enforce strict judge output parsing by @atirna in #1588
  • feat(benchmark): add CountQA object counting benchmark by @Yunnglin in #1590
  • fix: correct self.dataset typo in VQA.info by @Ricardo-M-L in #1589
  • feat(benchmark): add PhyX physical reasoning benchmark (phyx_mc, phyx_oe) by @Yunnglin in #1593
  • feat(benchmark): add SLAKE bilingual medical VQA benchmark by @Yunnglin in #1592
  • fix(perf): guard None content in openai_api token accounting by @tianba-sh in #1591
  • fix(bfcl): patch bfcl_eval FC handler to stop dropping empty-tool_cal… by @Dhru1001 in #1596
  • fix(multi-choice): parse the last valid answer label instead of the first by @juzihan0459 in #1597
  • feat(benchmarks): add native Indic-language benchmark adapters (milu,… by @Dhru1001 in #1569
  • docs(api): fix message and tool documentation typos by @BingH225 in #1600
  • feat(io): unify loading image/video/audio for general_vmcq by @Moenupa in #1595
  • feat(benchmark): add olmOCR-Bench document transcription benchmark by @ClaireXi99 in #1598
  • perf(service): memoize report-list metadata and support conditional GET by @Yunnglin in #1607
  • feat(benchmarks): Add Native Indic Benchmarks Phase 2 by @Dhru1001 in #1603
  • refactor(judge): unify LLM judge scoring on a single JSON output contract by @Yunnglin in #1601
  • docs: clarify text-to-image metric selection by @MrChenfafafa in #1612
  • feat(eval): add native evaluation versioning and deterministic choices by @Yunnglin in #1615
  • fix(terminal_bench): reject invalid trial rewards by @Excelius-Wang in #1610
  • fix(eval): report incomplete terminal-bench runs by @Yunnglin in #1616
  • feat(io): support media placeholders in general_vqa by @Moenupa in #1609
  • fix(cmmmu): normalize image media inputs by @Excelius-Wang in #1618
  • fix(benchmark): normalize remaining vision media by @Yunnglin in #1623
  • fix(config): reject unknown task-config keys and consolidate deprecation/serialization by @Yunnglin in #1624
  • fix(benchmark): load OmniDocBench v1.6 from snapshot by @Yunnglin in #1625

New Contributors

Full Changelog: v1.10.0...v1.11.0

Contributors

Yunnglin, tianba-sh, and 14 other contributors
Assets 2
Loading