v1.14.1

中文版

基准测试数据集

• 新增 SafetyBench 零样本安全评测，支持 zh、en、zh_subset，分别报告整体及七类准确率；无法解析或缺失的预测计错，不随机回退到某个答案。

功能增强

• Web 任务工作区：评测与压测共用模型连接，支持独立草稿、配置模板、导入导出及活动任务发现；刷新页面后可恢复监控，无需重复提交。
• 任务辨识：通过可选别名和脱敏后的端点区分同名模型的运行，在监控、报告、对比和 HTML 中展示；别名不改变评测标识。
• Bayes@N：新增二值均匀先验聚合，报告后验均值和单独的后验标准差，适用于完整固定重复次数的 accuracy / exact_match。
• CLI：新增 --dry-run，解析并打印配置后退出，不执行推理。

文档优化

• 补充音频基准所需的服务端 input_audio 支持及 SGLang 版本要求。
• 新增 SafetyBench、Bayes@N 和 Web 任务工作区说明，并更新缓存重新评分失败后的恢复方式。

问题修复

• 依赖：移除核心安装中的 LiteLLM 依赖；显式选择 eval_type='litellm' 时仍可单独安装使用。
• 缓存一致性：修复 rerun_review=True 失败后普通续跑静默复用旧分数的问题，原子保存配置并持久化待完成评分状态，保留原始预测来源。
• 数学评分：使用进程隔离为支持 fork 的非 daemon 进程中的数学评分调用增加计算与结果传输期限，超时样本按执行失败排除，避免退化输入长时间占用所有评分线程。
• 模型响应：拒绝缺少完整结束状态的 OpenAI / LiteLLM 流，识别 HTTP 200 内嵌的 SSE 错误和错误响应体；流读取失败纳入重试，避免保存部分答案或把失败计为正常输出。
• 代码评测：修复 MBPP+ 未执行扩展测试及 test_imports、HumanEval 本地超时过早开始计时，以及 SciCode 将代码块外文本送入执行的问题。
• 多模态评测：按官方 leave-one-out 方式计算 VQAv2 分数；ScienceQA 加入题目 hint；ARC-AGI-2 评测每个 test 输入并按任务汇总；MMAU 使用有公开答案的 test_mini，取消缺省选择 A。
• ASR：LibriSpeech、FLEURS、CommonVoice15 按全语料累计编辑数与参考词数计算 WER；TORGO 同时修复 corpus WER 和 CER。
• 视频描述：MSR-VTT、MSVD 按完整子集计算 CIDEr 和 BLEU，避免指标随评分批大小变化。
• 问答与分类：修复 DocMath 裁判问题及带符号数值答案、FRAMES 答案行、TriviaQA 官方规范化、Drivel 多标签、PubMedQA 单标签、HaluEval 与 CoinFlip 评分和汇总一致性。
• 评分协议：修复 EQ-Bench v2 的排除与报告尺度；修正 SPaRC 的星形、正负多格骨牌规则提示。
• 数据导出：修复 JSONL 转 CSV 时值与字段名未对齐的问题。

升级说明

• 需要 LiteLLM 后端的用户请显式执行 pip install litellm；基础 OpenAI、System One 等路径不再通过核心依赖安装它。第三方后端仍可能自行依赖 LiteLLM。
• 失败或中断的重新评分会阻止普通缓存续跑；请使用 rerun_review=True / --rerun-review 完成恢复。预测文件仍可复用，不会把旧分数当作新评测结果。
• 多项基准的输入或评分发生变化，请核对 evaluation_version。ScienceQA、SPaRC 等提示变化需要重新生成预测；评分修复可对旧预测重新评分。
• 数学评分的 30 秒期限从子进程启动后开始，覆盖计算和完整结果传输，不覆盖预热、等待 fork 锁、启动、父进程解码和清理。Windows 等无 fork 环境及 daemon 进程仍为无期限的进程内路径。本次执行期限变化不会自动使旧数学 reviews 失效，需要验证新策略时请显式重新评分。
• Bayes@N 当前仅支持一个选定子集，不支持 collection、多子集或缺失/重复试验；默认聚合方式不变。
• Web 连接与模板默认不保存或导出 API key，携带密钥需显式选择并确认。任务注册表仍是进程内的，服务重启不提供持久调度或孤儿子进程托管。

---

English Version

Benchmark Datasets

• Added zero-shot SafetyBench evaluation for zh, en and zh_subset, with separate overall and seven-category accuracies. Missing and unparsed predictions count as incorrect, without random answer fallback.

Feature Enhancements

• Task Workspace: Evaluation and performance testing share model connections, with independent drafts, reusable templates, import/export and active-task discovery. Reloading reconnects monitoring without resubmission.
• Run Identity Display: Optional aliases and sanitized endpoints distinguish runs using the same model across monitors, reports, comparisons and HTML. Aliases do not change evaluation identity.
• Bayes@N: Added binary uniform-prior aggregation with a posterior mean and a separate posterior standard deviation for complete fixed-repeat accuracy / exact_match trials.
• CLI: Added --dry-run to parse and print configuration without running inference.

Documentation

• Documented server-side input_audio support and SGLang version requirements for audio benchmarks.
• Added SafetyBench, Bayes@N and task-workspace guides, and documented recovery from failed review reruns.

Bug Fixes

• Dependencies: Removed LiteLLM from the core installation. Users selecting eval_type='litellm' can install it separately.
• Cache Consistency: Fixed stale review reuse after a failed rerun_review=True by atomically publishing configuration and persisting pending review state while preserving prediction provenance.
• Mathematical Scoring: Added process isolation and a computation/result-transfer deadline in fork-capable, non-daemonic processes. Timed-out samples are excluded as execution failures rather than freezing scoring threads on degenerate inputs.
• Model Responses: Reject incomplete OpenAI/LiteLLM streams and detect in-band SSE errors and error response bodies. Stream-consumption failures participate in retries instead of persisting partial answers or treating failures as valid output.
• Code Evaluation: Fixed MBPP+ extended-suite and test_imports execution, the start time of HumanEval local deadlines, and SciCode execution of text outside code blocks.
• Multimodal Evaluation: Aligned VQAv2 with official leave-one-out accuracy, included ScienceQA hints, evaluated all ARC-AGI-2 test inputs with per-task aggregation, and restricted MMAU to labeled test_mini without defaulting to A.
• ASR: LibriSpeech, FLEURS and CommonVoice15 now pool edits and reference words for corpus WER. TORGO now reports corpus WER and CER.
• Video Captioning: Compute MSR-VTT/MSVD CIDEr and BLEU over the complete subset, independently of review batch size.
• QA and Classification: Fixed DocMath judge questions and signed numerical answers, FRAMES answer lines, official TriviaQA normalization, Drivel multi-label parsing, PubMedQA single-label verdicts, and HaluEval/CoinFlip scoring/report consistency.
• Evaluation Protocols: Fixed EQ-Bench v2 exclusions and report scale, and corrected SPaRC star and positive/negative polyshape instructions.
• Data Export: Fixed CSV values being misaligned with field names when exporting JSONL.

Upgrade Notes

• Install litellm explicitly when using that backend. Core OpenAI, System One and related paths no longer install it transitively; third-party backends may still depend on it.
• Interrupted or failed review reruns block ordinary cache reuse. Resume with rerun_review=True / --rerun-review; predictions remain reusable without silently restoring old scores.
• Input or scoring changes affect several benchmarks; check evaluation_version. Prompt changes such as ScienceQA/SPaRC require new predictions. Scoring-only fixes can rescore saved predictions.
• The 30-second math deadline starts after child-process startup and covers computation plus complete result transfer. Warm-up, fork-lock waiting, process startup, parent decoding and cleanup are outside it. Windows/non-fork and daemon environments retain the unbounded in-process path. This execution-policy change does not automatically invalidate older math reviews; explicitly rerun scoring to apply it.
• Bayes@N currently supports one selected subset, not collections, multiple subsets or incomplete/duplicate trials. Default aggregation is unchanged.
• Web connections/templates exclude API keys from automatic storage and default exports; including secrets requires explicit opt-in and confirmation. The task registry remains process-local, without durable scheduling or orphan-process supervision after service restart.

---

What's Changed

• docs(benchmarks): note SGLang input_audio version requirement for audio benchmarks by @lemonzoey in #1902
• feat: add binary Bayes@N aggregation for repeated evaluations by @mohsenhariri in #1903
• fix(metrics): bound Math-Verify parsing with process isolation by @Zhuwenbopro in #1897
• fix(models): reject incomplete OpenAI and LiteLLM streams by @Ricardo-M-L in #1910
• fix(halu_eval): aggregate parsed verdicts, not the opposite of a miss by @David-Wu1119 in #1912
• fix(mbppplus): run the MBPP+ test suite and the task's test_imports by @David-Wu1119 in #1913
• fix(vqav2): use the official leave-one-out VQA accuracy by @David-Wu1119 in #1916
• fix(docmath): preserve the judge question and numeric answer signs by @David-Wu1119 in #1914
• fix(eq_bench): align v2 scoring exclusions and report scale by @David-Wu1119 in #1911
• fix(science_qa): include the question context (hint) in the prompt by @David-Wu1119 in #1917
• feat(benchmark): add SafetyBench zero-shot evaluation by @Yunnglin in #1920
• feat(web): add reusable task workspace and run monitoring by @Yunnglin in #1904
• fix(mmau): score test_mini only and stop defaulting answers to A by @David-Wu1119 in #1915
• fix(msr_vtt, msvd): report corpus-level CIDEr and BLEU, independent of eval_batch_size by @David-Wu1119 in #1919
• fix(humaneval): start the time limit when the sandbox child starts by @David-Wu1119 in #1922
• feat(cli): implement --dry-run to parse and print the config without inference by @lemonzoey in #1921
• fix(models): detect in-band SSE errors and error payloads in streaming + LiteLLM paths by @lemonzoey in #1918
• fix(pubmed_qa): credit only a reply that gives exactly one label by @David-Wu1119 in #1925
• fix(drivel_multilabel): read every listed label on the answer line by @David-Wu1119 in #1926
• fix(arc_agi_2): ask and score every test input of a task by @David-Wu1119 in #1927
• fix(frames): read the answer line in any case and stop at its end by @David-Wu1119 in #1924
• fix(asr): report corpus WER for LibriSpeech, FLEURS and CommonVoice15 by @David-Wu1119 in #1923
• SPaRC: state the star, polyshape and negative polyshape rules as the puzzles use them by @lkaesberg in #1928
• fix: align JSONL values with CSV field names by @git-jxj in #1930
• fix(scicode): run only the code inside the response's code block by @David-Wu1119 in #1932
• fix(trivia_qa): normalize answers as the official TriviaQA evaluation does by @David-Wu1119 in #1933
• fix(coin_flip): score samples and report accuracy with the same verdict parser by @David-Wu1119 in #1934
• fix(deps): remove LiteLLM from core dependencies by @Yunnglin in #1940
• fix(torgo): report corpus WER and CER by @David-Wu1119 in #1935
• fix(cache): keep review identities consistent after failed rescoring by @xlb3055 in #1936

New Contributors

• @lemonzoey made their first contribution in #1902
• @mohsenhariri made their first contribution in #1903
• @Zhuwenbopro made their first contribution in #1897

Full Changelog: v1.14.0...v1.14.1