v1.11.1

中文版

基准测试数据集

• 多模态评测：新增 SURDS、VLMs Are Biased、Ref-Adv-s、VisFactor、MedXpertQA、VTCBench 等视觉、多模态与长视频理解基准。
• 推理与专业能力评测：新增 $OneMillion-Bench、PRBench、HMMT-Nov-2025 等 Agent、深度推理与数学评测基准。

功能增强

• API 契约：Web API 响应契约改由 Pydantic 模型自动生成，提升前后端类型一致性。
• 缓存与数据集：隔离数据集缓存键，增强视频解码与 limit 参数校验。
• 模型服务：支持识别 HTTP 200 响应中的网关错误载荷，并优化音频预处理、流式响应、用量统计与重试行为。
• 性能测试：优化指标精度、关闭流程、闭环 warmup 交接，以及 SSE Unicode 内容处理。

文档优化

• 更新 README 与 Dashboard 使用说明。
• 修复 Model API 文档中的过期说明。

问题修复

• 修复 ProcessBench 等运行只产生部分指标时的报告异常：主指标不可用将被明确标记，不再错误地替换为辅助分数。
• 修复 General QA 在错误样本下的指标身份保留、指令级指标聚合、答案解析和 ROUGE 评分问题。
• 修复 IFBench 重复评测版本及唯一词约束校验问题。
• 修复 OmniDocBench 空页面指标、VTCBench HTML 标签解析、Toolathlon 任务生命周期与结果校验问题。
• 修复 Judge 延迟初始化并发、NLTK 镜像文件校验、缓存恢复不完整记录等问题。

---

English Version

Benchmark Datasets

• Multimodal Evaluation: Added SURDS, VLMs Are Biased, Ref-Adv-s, VisFactor, MedXpertQA, VTCBench, and other vision, multimodal, and long-video benchmarks.
• Reasoning and Domain Evaluation: Added $OneMillion-Bench, PRBench, and HMMT-Nov-2025 for agent, deep-reasoning, and mathematical evaluation.

Feature Enhancements

• API Contracts: Web API response contracts are now generated from Pydantic models for stronger frontend/backend type consistency.
• Cache and Datasets: Isolated dataset cache keys and improved video decoding and limit validation.
• Model Services: Detects gateway-error payloads returned with HTTP 200, and improves audio preprocessing, streaming responses, usage accounting, and retry behavior.
• Performance Testing: Improved metric accuracy, shutdown handling, closed-loop warmup handoff, and Unicode handling in SSE streams.

Documentation

• Updated README and Dashboard guidance.
• Fixed stale Model API documentation.

Bug Fixes

• Fixed report failures when ProcessBench and similar runs emit only part of their metrics: unavailable primary metrics are now reported explicitly without substituting an auxiliary score.
• Fixed metric-identity preservation for failed General QA samples, instruction-level aggregation, answer parsing, and ROUGE scoring.
• Fixed duplicate evaluation versions and unique-word validation in IFBench.
• Fixed empty-page metrics in OmniDocBench, HTML tag parsing in VTCBench, and Toolathlon job lifecycle and result validation.
• Fixed concurrent lazy Judge initialization, NLTK mirror archive verification, and incomplete cache-resume records.

What's Changed

• fix(rag): raise a clear error when a LogitScore reranker loads on an old sentence-transformers by @AmirF194 in #1620
• feat(io): unify undecoding behavior and support overlong images list by @Moenupa in #1626
• cicd(isort): migrate to ruff isort by @Moenupa in #1629
• fix(typing): add type hints to logger by @Moenupa in #1632
• docs: fix stale ModelAPI docstrings (nonexistent api_key_vars / ChatUserMessage) by @BingH225 in #1634
• fix(score): prune rouge scoring function and fix en version of rouge by @Moenupa in #1633
• feat(benchmarks): add hmmt_nov25 benchmark by @haoruilee in #1636
• feat(benchmark): support VTCBench by @Moenupa in #1635
• fix(perf): preserve Unicode line separators in SSE payloads by @Yunnglin in #1642
• fix(benchmark): VTCBench wrongly parsed html tags by @Moenupa in #1643
• test(perf): stop asserting log text in workload_trace tests by @Yunnglin in #1647
• fix(perf): hand closed-loop warmup over without draining the server by @Yunnglin in #1641
• fix(report): sync HTML reports with console theme by @Yunnglin in #1646
• chore: unify linting and formatting with Ruff by @Yunnglin in #1644
• feat: support MedXpertQA benchmark by @Yunnglin in #1655
• fix(web): derive API contracts from Pydantic by @Yunnglin in #1658
• feat(benchmarks): add PRBench by @Yunnglin in #1665
• feat: add OneMillion-Bench benchmark by @Yunnglin in #1667
• fix(benchmark): restore F821 and fix jobbench mis-ignored linting issue by @Moenupa in #1652
• feat: add VisFactor benchmark by @Yunnglin in #1661
• feat(benchmarks): add Ref-Adv-s by @Yunnglin in #1668
• feat(benchmarks): add VLMs Are Biased by @Yunnglin in #1669
• fix(metrics): correct answer parsing and text scoring by @git-jxj in #1649
• fix(models): correct retry semantics and Anthropic streaming by @git-jxj in #1651
• fix(models): correct cache, usage, streaming, and image outputs by @git-jxj in #1653
• fix(perf): improve metric accuracy, validation, and shutdown by @git-jxj in #1654
• feat(benchmarks): add SURDS by @Yunnglin in #1670
• fix(judge): serialize lazy judge initialization by @git-jxj in #1664
• fix(models): avoid blocking event loop during async audio preprocessing by @git-jxj in #1662
• fix(resources): verify NLTK mirror archives with pinned digests by @git-jxj in #1663
• fix(cache): deduplicate review state and tolerate torn resume rows by @git-jxj in #1657
• fix(benchmark): handle empty OmniDocBench page metrics by @git-jxj in #1666
• refactor(models): move litellm imports to module level by @seroze in #1678
• fix(metrics): make inst_level_* a micro-average over instructions by @arkrolin in #1672
• fix(ifbench): enforce unique words in sentence checker by @linhongyu510 in #1676
• fix(ifbench): remove duplicate evaluation version by @Excelius-Wang in #1681
• fix(perf): ignore metadata-only chunks in TTFT and ITL by @Excelius-Wang in #1645
• fix(benchmark): harden Toolathlon job lifecycle and result validation by @git-jxj in #1656
• fix(benchmark): preserve metric identities on General QA/VQA scoring errors by @git-jxj in #1660
• fix(dataset): isolate cache keys, undecode video, and validate limits consistently by @git-jxj in #1659
• fix(models): retry 200 responses that carry a gateway error payload by @seroze in #1673
• refactor(io): remove unused type cast and deduplicate code in media io by @Moenupa in #1683

New Contributors

• @AmirF194 made their first contribution in #1620
• @seroze made their first contribution in #1678
• @arkrolin made their first contribution in #1672
• @linhongyu510 made their first contribution in #1676

Full Changelog: v1.11.0...v1.11.1