Skip to content

v1.15.0

Latest
Latest

Choose a tag to compare

View all tags
@Yunnglin Yunnglin released this 13 Nov 15:46
v1.15.0
c4f6021

中文版

基准测试数据集

  • 安全与诚实性：新增 SALAD-Bench（开放问答与多选）、XSTest（过度拒绝）、LongPIBench（长上下文提示注入）、MASK（受压场景下的诚实性）和 SLEIGHT-Bench（编码 Agent 操作记录中的攻击检测）。
  • 多模态安全：新增 MM-SafetyBench++，分别评估安全与不安全场景中的回答、拒绝行为和回答质量。
  • Agent 评测：新增 AgentDojo 提示注入、AgentHarm 恶意与良性任务，以及 Cybench 无引导 CTF 任务，支持工具轨迹记录及缓存重新评分。
  • 代码理解：新增 SWE-Atlas QnA，评测对真实代码仓库的理解，分别报告任务解决率和 rubric 通过率。
  • 法律与语言理解：新增 LawBench 原生零样本评测，覆盖 19 项任务、9,500 个样本（不含文书校对任务 2-1）；新增 FewCLUE CLUEWSC 中文指代消解。
  • 数据更新：VLMs Are Biased 默认使用固定 revision 的 v2 数据，同时保留显式 v1-legacy 入口及独立缓存标识。

功能增强

  • 远端沙箱：支持通过官方 E2B 和阿里云 Agent Sandbox 运行受支持的代码及原生 Agent 基准；Docker 仍为默认引擎。阿里云普通 Python 评测可自动使用内置 base 模板，显式配置优先。参考沙箱指南。
  • 文生图 API：支持 OpenAI 兼容的图像生成服务，解析 base64 或 URL 图像结果；本地 Diffusers 依赖检查仅用于本地推理路径。
  • Web 看板：支持勾选指定历史运行进行对比，包括同一模型的多次运行；新增推理强度、思考预算、思考开关，以及用于顶层任务配置的 Extra Config。
  • 报告：为二值、越高越好的指标增加 Correct 列，与 Num 一起展示正确样本数量。
  • 压测：新增 KV 缓存命中率的请求与输入 token 覆盖率，缺失 cached_tokens 时给出提示，不将未知值当作未命中。
  • 预热控制：新增单轮闭环压测选项 --exclude-warmup-overlap，将预热仍在运行期间发出的过渡请求单独记录并排除出测量统计；默认行为不变。
  • 模型与工具：完善本地模型文本中的工具调用解析，支持 parameters 和 [TOOL_CALLS] 数组；为 OpenAI 官方推理模型使用兼容的请求默认参数。
  • 请求重试：OpenAI 兼容 Chat 请求支持有上限的指数退避，并将数值型 Retry-After 作为最短等待时间。

文档优化

  • 重整双语沙箱与 System One 使用指南，补充引擎选择、模板、安装及结果说明。
  • 补充新增基准、Web 历史运行对比和文生图服务用法，刷新 CC-OCR-V2 完整数据统计。
  • 说明自定义裁判 JSON 输出协议，以及可选的 judge.models[*].generation_config JSON 模式配置。

问题修复

  • 裁判输出：统一提示词与 schema，严格校验一个完整 JSON 对象，拒绝夹带正文、Markdown 围栏、多对象、重复键和非标准常量；无效裁判结果按现有失败策略保留记录并排除，显式规则回退仍保留其规则结果。
  • 答案提取：修复加粗答案标记、标点及 答案：[AD]、[A D] 多选格式；避免异常括号输入导致正则回溯卡住；完善 ZebraLogicBench、LongPIBench 和 Maritime OCR 的 JSON 答案提取。
  • 评分准确性：修复 RefCOCO 的语料级 CIDEr/BLEU、累计 BLEU n-gram 权重、中英混合 MER、CrossNER 子集标签、BFCL-v3 汇总权重、WideSearch 主键对齐及 ToolBench Plan.EM 指标名。
  • 评分边界：修复 IFBench 四类校验器的官方协议与空文本处理、MRCR 越界样本的上下文分桶、数学数字间距与货币符号，以及无效或缺失函数调用被误判为 schema 合法的问题。
  • 重复评测：隔离 BFCL-v4 模拟器状态及 VisFactor、HallusionBench、ACEBench 的重复生成评分组，避免不同试验互相影响。
  • 缓存与数据：collection 缓存标识纳入成员基准版本及有效配置；支持直接加载 collection JSONL 文件；修复本地文件随机种子转发、内存数据原地打乱和 repeats=0 除零；按配置隔离缓存的 metric 实例。
  • 聚合：pass@k / pass^k 只接受二值结果；不兼容指标保留普通均值并省略 pass 估计，避免将部分得分当作成功次数。
  • 模型协议：保留流式工具名、无 index 的并行工具调用及 token logprobs；处理缺失 token 计数、Responses 内嵌失败和 Anthropic 不完整流；修复 Anthropic SDK 2.x 参数构造、结束原因、扩展思考参数及自定义请求头/查询参数。
  • 代码与 Agent：修复 LiveCodeBench JSON 输入与引号代码、MultiPL-E Java 代码组装、DeepSWE 端点与 provider 转发、SWE-Atlas rubric 提示和 SWE-bench Pro 测试补丁；超时或取消时清理本地命令进程组。
  • 输入与产物：补齐 ArxivRollBench 完形题材料，遵循 AA-LCR 文档清单顺序；缩小 NeedleHaystack 裁判输入；隔离重复生成的图像与音频文件，避免覆盖已缓存产物。
  • 压测与展示：拒绝无响应事件的 SSE 和 HTTP 200 内嵌错误；修复闭环时限后排队请求继续发送、连接探测超时、embedding 吞吐、Responses 工具调用计时、trace 请求头优先级及 AgentX 本地镜像回放；修复无分数报告、开放环运行归档和历史运行时间展示。
  • 其他：统一使用设备检测选择本地推理设备，保留 Pydantic 单值 Literal 工具参数约束；翻译模块导入不再修改 BUILD_DOC。

升级说明

  • 最低 Python 版本升至 3.11。 沙箱相关 extras 使用 ms-enclave[docker,e2b]>=0.0.9；E2B 仅覆盖文档列出的基准，外部 Agent 和 Harbor/Pier 执行不在本次集成范围内。
  • 共享评分版本现为 v1.3。裁判输出须满足声明的 JSON schema；格式无效会减少实际计分样本数，请结合覆盖情况解读结果。可选 JSON 模式不替代校验，也不保证裁判判断正确。
  • collection 成员版本或配置变化会阻止复用不匹配的评分；旧 collection 缓存需重新评分。其他评分修复请核对各基准的 evaluation_version，需要时使用 rerun_review=True。
  • ArxivRollBench、AA-LCR、CrossNER 等输入或提示修复需要重新生成预测。VLMs Are Biased 切换 v2 与旧版时应使用新输出目录；SWE-bench Pro 旧分数应与修复后的 v1.1 分开。
  • LawBench 总分仅覆盖已支持的 19 项任务，不等同于完整 20 项官方分数。远端沙箱和部分 Agent 基准仍需各自的运行环境与可选依赖。

English Version

Benchmark Datasets

  • Safety and Honesty: Added SALAD-Bench (open-ended and multiple-choice), XSTest (over-refusal), LongPIBench (long-context prompt injection), MASK (honesty under pressure), and SLEIGHT-Bench (attack detection in coding-agent transcripts).
  • Multimodal Safety: Added MM-SafetyBench++ to evaluate answering, refusal behavior and response quality in safe and unsafe contexts.
  • Agent Evaluation: Added AgentDojo prompt injection, AgentHarm harmful and benign tasks, and unguided Cybench CTF tasks, with tool traces and cached regrading.
  • Code Understanding: Added SWE-Atlas QnA for real-repository comprehension, reporting task resolve rate and rubric pass rate separately.
  • Legal and Language Understanding: Added native zero-shot LawBench for 19 tasks and 9,500 samples, excluding document proofreading task 2-1, and FewCLUE CLUEWSC for Chinese coreference resolution.
  • Dataset Update: VLMs Are Biased now defaults to revision-pinned v2 data, with an explicit v1-legacy entry and separate cache identity.

Feature Enhancements

  • Remote Sandboxes: Run supported code and native-agent benchmarks on official E2B or Aliyun Agent Sandbox; Docker remains the default. Ordinary Python evaluations on Aliyun can use the built-in base template automatically, while explicit settings take precedence. See the sandbox guide.
  • Text-to-Image APIs: Support OpenAI-compatible image generation services with base64 or URL results. Diffusers dependency checks apply only to local inference.
  • Web Dashboard: Select specific historical runs for comparison, including several runs of one model. Added reasoning effort, thinking budget, thinking controls and Extra Config for top-level task settings.
  • Reports: Added a Correct column beside Num for binary, higher-is-better metrics.
  • Performance Metrics: Added request and input-token coverage for KV cache hit rates, with warnings for missing cached_tokens; unknown values are not treated as cache misses.
  • Warmup Control: Added opt-in --exclude-warmup-overlap for single-turn closed-loop tests. Requests sent while warmup remains in flight are recorded separately as transition work and excluded from measured statistics; defaults are unchanged.
  • Models and Tools: Extended tool-call extraction from local model text to parameters and [TOOL_CALLS] arrays, and use compatible request defaults for official OpenAI reasoning models.
  • Retries: OpenAI-compatible Chat requests use capped exponential backoff and honor numeric Retry-After as a minimum wait.

Documentation

  • Reorganized the bilingual sandbox and System One guides around engine selection, templates, installation and results.
  • Documented new benchmarks, historical-run comparison and text-to-image services; refreshed complete CC-OCR-V2 statistics.
  • Documented custom judge JSON contracts and optional JSON mode through judge.models[*].generation_config.

Bug Fixes

  • Judge Outputs: Aligned prompts with schemas and validate one complete JSON object, rejecting surrounding prose, Markdown fences, multiple objects, duplicate keys and nonstandard constants. Invalid observations remain recorded and excluded under the existing failure policy; explicit rule fallbacks retain their rule result.
  • Answer Extraction: Fixed bold answer markers, punctuation and multi-select forms such as 答案：[AD] and [A D]; prevent pathological bracket input from stalling regex matching. Improved JSON answer extraction for ZebraLogicBench, LongPIBench and Maritime OCR.
  • Scoring Accuracy: Fixed corpus CIDEr/BLEU for RefCOCO, cumulative BLEU n-gram weights, mixed Chinese-English MER, CrossNER subset labels, BFCL-v3 summary weights, WideSearch key alignment and ToolBench's Plan.EM metric name.
  • Scoring Boundaries: Aligned four IFBench verifiers and wordless-response handling with upstream, corrected MRCR context bins for out-of-range samples, normalized mathematical digit spacing and currency symbols, and reject invalid or missing function calls during schema scoring.
  • Repeated Evaluation: Isolated BFCL-v4 simulator state and repeat-specific scoring groups for VisFactor, HallusionBench and ACEBench.
  • Caching and Data: Include member versions and effective settings in collection cache identities, accept direct collection JSONL paths, honor local-file seeds, preserve caller-owned record order, handle repeats=0, and isolate cached metric instances by configuration.
  • Aggregation: Require binary outcomes for pass@k / pass^k; retain ordinary means and omit incompatible pass estimates instead of treating partial credit as successful generations.
  • Model Protocols: Preserve streamed tool names, parallel calls without indices and token logprobs. Handle missing usage counts, in-band Responses failures and incomplete Anthropic streams. Fixed Anthropic SDK 2.x parameter construction, stop reasons, extended-thinking options and custom headers/query parameters.
  • Code and Agents: Fixed LiveCodeBench JSON inputs and quoted code, MultiPL-E Java assembly, DeepSWE endpoint/provider forwarding, SWE-Atlas rubric prompts and SWE-bench Pro test-patch staging. Clean up local command process groups on timeout or cancellation.
  • Inputs and Artifacts: Include ArxivRollBench cloze material, honor AA-LCR document order, reduce NeedleHaystack judge input, and isolate generated image/audio files across repeats to avoid overwriting cached artifacts.
  • Performance and Display: Reject eventless SSE and in-band HTTP 200 errors. Fixed queued dispatch after closed-loop deadlines, connection-probe timeouts, embedding throughput, Responses tool-call timing, trace header precedence and AgentX local-mirror replay. Fixed no-score reports, open-loop archives and historical-run timestamps.
  • Other: Use shared device detection for local serving, retain singleton Literal constraints in tool schemas, and stop translator imports from changing BUILD_DOC.

Upgrade Notes

  • Python 3.11 or newer is required. Sandbox extras use ms-enclave[docker,e2b]>=0.0.9. E2B supports only the documented benchmarks; external agents and Harbor/Pier execution are outside this integration.
  • The shared scoring version is now v1.3. Judge replies must satisfy their declared JSON schema; invalid formats reduce scored coverage. Optional JSON mode does not replace validation or guarantee correct judgments.
  • Collection member version/configuration changes reject incompatible cached reviews; older collection caches require regrading. Check each benchmark's evaluation_version for other scoring changes and use rerun_review=True when needed.
  • Input/prompt fixes such as ArxivRollBench, AA-LCR and CrossNER require new predictions. Use a new output directory when switching VLMs Are Biased data versions, and keep older SWE-bench Pro scores separate from corrected v1.1 results.
  • LawBench's overall score covers only the supported 19 tasks and is not the complete official 20-task score. Remote sandboxes and some agent benchmarks still require their documented environments and optional dependencies.

What's Changed

  • fix(refcoco): score caption metrics as one corpus by @David-Wu1119 in #1931
  • feat(benchmark): add XSTest by @Yunnglin in #1944
  • feat(benchmark): add SALAD-Bench safety benchmark by @Yunnglin in #1943
  • feat(benchmark): add MM-SafetyBench++ contextual safety benchmark by @Yunnglin in #1946
  • feat(benchmark): add LongPIBench prompt injection benchmark by @Yunnglin in #1945
  • feat(benchmark): add MASK honesty benchmark by @Yunnglin in #1948
  • feat(benchmark): add SLEIGHT-Bench by @Yunnglin in #1947
  • fix(modelscope): read tool calls out of a local model's text output by @PerryLink in #1937
  • fix: treat remaining in-band failures as errors (follow-up to #1918) by @lemonzoey in #1942
  • fix(openai): concatenate streamed tool function names by @Ricardo-M-L in #1950
  • fix(anthropic): build message params as plain dicts so anthropic 2.x works by @JOhnsonKC201 in #1955
  • fix(cross_ner): prompt and score each subset with its own entity labels by @David-Wu1119 in #1952
  • fix(needle_haystack): ask the judge only the retrieval question by @David-Wu1119 in #1954
  • fix(bfcl_v3): compute summary rows with the official leaderboard weights by @David-Wu1119 in #1953
  • fix(arxivrollbench): include the passage and candidates in cloze prompts by @David-Wu1119 in #1951
  • feat(web): compare selected dashboard runs by @JianhaoYu-AI in #1958
  • fix(web): honor custom roots on initial comparison load by @Yunnglin in #1960
  • feat(report): show the number of correct samples next to Num by @hyunsooda in #1956
  • fix(dataset): preserve DictDataLoader input when shuffling by @xlb3055 in #1972
  • fix(chartqa, real_world_qa): fall back to prediction when ANSWER: marker is missing by @lsyx6399 in #1966
  • fix(perf): embedding input token throughput has 10x deviation at high concurrency by @hjh288 in #1969
  • fix(perf): count embedding/rerank in-band 200 errors as failures by @lsyx6399 in #1964
  • fix(models): keep parallel tool calls apart when a stream omits their index by @pcbeingused333 in #1974
  • fix(xstest): judge with the official XSTest classifier definitions by @jaeminbird in #1967
  • feat(benchmark): add AgentDojo prompt-injection benchmark by @Yunnglin in #1973
  • fix(bridge): report tool_use when a stopped Anthropic message carries tool calls by @pcbeingused333 in #1975
  • fix(perf): bound the connection test with --connection-timeout, not --total-timeout by @lsyx6399 in #1971
  • [Bugfix] Use get_device() instead of hardcoded CUDA in local serving and perf by @lsyx6399 in #1962
  • feat: support text-to-image API services by @Excelius-Wang in #1617
  • fix(deep_swe): pass model endpoint credentials to the Pier agent by @AliceSama1 in #1979
  • feat(web): add reasoning controls to the evaluation config form by @AliceSama1 in #1981
  • feat(benchmark): add native AgentHarm evaluation by @Yunnglin in #1977
  • fix(models): accept parameters key, [TOOL_CALLS] arrays, and warn on unparseable tool-call output by @bjptw in #1976
  • fix(perf): preserve CLI header overrides during trace replay by @Ricardo-M-L in #1985
  • fix(perf): honor duration after closed-loop dispatch waits by @git-jxj in #1984
  • fix(dataset): keep loaders from dividing by zero when repeats is 0 by @abishek-ctrl in #1987
  • fix(bfcl): isolate v4 rollout and scoring state by @JiahaoTanXX in #1989
  • fix(service): use a run's recorded start time for the report list by @abishek-ctrl in #1983
  • docs(cc_ocr_v2): refresh complete dataset statistics by @Yunnglin in #1990
  • fix: parse multi-argument LiveCodeBench sandbox inputs as JSON by @damienen in #1994
  • Fix LiveCodeBench sandbox grading for quoted candidate code by @gugugu-b in #2005
  • fix(anthropic): omit temperature and top_k when extended thinking is enabled by @pcbeingused333 in #2002
  • fix(perf): send speed_benchmark requests to the completions endpoint by @David-Wu1119 in #2001
  • fix(multipl_e): grade complete Java solutions independently of formatting by @JiahaoTanXX in #2006
  • fix(anthropic): retry a stream that ends without a stop reason by @David-Wu1119 in #2000
  • fix(anthropic): report a max_tokens stop as max_tokens by @David-Wu1119 in #1999
  • fix(models): accept usage blocks that omit a token count by @David-Wu1119 in #1998
  • fix(perf): count streamed tool calls as output in the Responses plugin by @David-Wu1119 in #1997
  • fix(service): serve open-loop perf runs without a fixed rate by @David-Wu1119 in #1996
  • fix(report): render no-score reports as empty in the dashboard views by @David-Wu1119 in #1995
  • fix(anthropic): forward extra_headers and extra_query to the Anthropic client by @pcbeingused333 in #2008
  • fix(agent): clean up local command process trees on timeout and cancellation by @Cookie0103 in #2007
  • feat(deep_swe): let pier_agent_kwargs set the agent LiteLLM provider by @AliceSama1 in #2010
  • feat(benchmark): add native Cybench unguided evaluation by @Yunnglin in #1991
  • fix(aa_lcr): honor the ordered document manifest by @JiahaoTanXX in #2012
  • fix(schema): preserve Pydantic singleton Literal constraints by @yunyv in #1957
  • fix: honor singleton Gemini tool allowlists in ANY mode by @yunyv in #1868
  • perf: report KV cache hit-rate coverage and warn when cached_tokens is missing by @Bakaraju in #1939
  • feat(web): add an Extra Config field for top-level TaskConfig keys by @AliceSama1 in #2014
  • feat(benchmark): add SWE-Atlas QnA codebase question-answering benchmark by @AliceSama1 in #2009
  • fix(metrics): preserve Chinese token boundaries in mixed-script MER by @yunyv in #1959
  • fix(models): honor Retry-After with bounded exponential backoff by @bjptw in #2017
  • fix(metrics): use cumulative weights for BLEU-2 through BLEU-4 by @feiiiiii5 in #2018
  • fix(models): map Anthropic model_context_window_exceeded to model_length by @pcbeingused333 in #2020
  • fix(models): retain logprobs in streamed OpenAI completions by @feiiiiii5 in #2019
  • fix(agentx): replay local Weka mirrors through the native loader by @simin-yuan in #2022
  • feat(perf): add opt-in exclusion of warmup overlap in closed-loop benchmarks by @OpenAI-Insights in #2015
  • fix(tool_bench): report Plan.EM under its declared metric name by @shlokbam in #2028
  • fix(multi_choices): drop punctuation after a closing bold answer line by @shlokbam in #2029
  • fix(benchmarks): isolate grouped scoring across repeats by @JiahaoTanXX in #2023
  • fix(docs): do not set BUILD_DOC when importing the translator by @GPT-AGI in #2039
  • fix(zebralogicbench): keep braces inside strings out of the object scan by @simin-yuan in #2030
  • refactor(benchmarks): share the last-JSON-object extractor by @simin-yuan in #2035
  • fix(longpi_bench): keep the last parseable JSON object by @simin-yuan in #2031
  • feat(benchmark): add native LawBench evaluation for 19 tasks by @JiahaoTanXX in #2025
  • fix(swe_atlas_qna): send the judge the rubric statement the official grader sends by @David-Wu1119 in #2044
  • fix(perf): do not send a queued closed-loop request after --duration by @David-Wu1119 in #2045
  • fix(perf): leave turns without cached_tokens out of per-trace cache rates by @David-Wu1119 in #2046
  • feat(sandbox): integrate E2B for SWE-bench and native agents by @Yunnglin in #2036
  • test(sandbox): scope the scoring version guard to Pro by @Yunnglin in #2048
  • fix(math): normalize thin-space digit groups and currency symbols before Math-Verify parsing by @shlokbam in #2049
  • feat: use Aliyun base for Python evaluations and clarify guides by @Yunnglin in #2054
  • fix(scoring): reject invalid function-call payloads by @JiahaoTanXX in #2042
  • fix(dataset): honor run seeds in local-file benchmark loading by @anticipate218 in #2050
  • fix(metrics): isolate cached scorers by configuration by @JiahaoTanXX in #2041
  • fix(vlms_are_biased): pin v2 data and separate legacy runs by @Yunnglin in #2068
  • fix(multi_choices): read an answer marker whose bold closes before the colon by @David-Wu1119 in #2056
  • fix(media): isolate generated artifacts across repeated evaluations by @JiahaoTanXX in #2058
  • fix(aggregation): require binary outcomes for pass estimates by @JiahaoTanXX in #2059
  • fix(wide_search): let the primary-key judge align raw values by @2362461686 in #2060
  • fix(models): use reasoning-safe request defaults on api.openai.com by @simin-yuan in #2064
  • fix(mrcr): exclude out-of-range samples from context bins by @JiahaoTanXX in #2066
  • fix(ifbench): align four verifiers with upstream and guard wordless responses by @Y9Y123 in #2071
  • feat(benchmark): add native CLUEWSC (FewCLUE) adapter by @JerryChaox in #2062
  • fix(perf): reject SSE streams without response events by @JiahaoTanXX in #2076
  • fix(collection): support direct JSONL file paths by @JiahaoTanXX in #2078
  • fix(collection): include member semantics in cache identities by @JiahaoTanXX in #2077
  • fix(judge): enforce a strict single-JSON output contract by @Yunnglin in #2070
  • fix(multi_choices): read bracketed multi-select labels that run together by @simin-yuan in #2075

New Contributors

Full Changelog: v1.14.1...v1.15.0

Contributors

hyunsooda, JerryChaox, and 31 other contributors
Assets 2
Loading