v3.1-20260912-master

🌟 Release Note

Key Features

1. Agent测评完全对接harbor框架，并提供一键部署的离线测评解决方案：Agent测评完全支持Harbor适配的agent以及harbor格式的数据集，同时为常用agent（包括claude-code，terminus-2，mini-swe-agent等）以及当前常用的agent数据集（SWE-Bench，terminal-bench，DeepSWE）提供了一键部署的离线测评解决方案。使用说明见 👉 基于Harbor框架的agent测评。🔥🔥🔥🚀🚀🚀
2. 支持 Prefix Cache 场景化压测与指标分析：提供数据生成、请求调度、缓存预热、指标采集与结果分析全链路能力，支持单 Prefix / 多 Prefix Group、目标命中率反算、cold/warmup 双模式与多 DP 定向路由，新增 prefix_hit_rate 等指标查询。详见 Prefix Cache 场景化压测。🚀🚀🚀
3. 新增 api_model_args CLI 命令组：支持在命令行显式传入 path、model、request_rate、host_ip、host_port、url、max_out_len、generation_kwargs 等 API 模型通用参数，并在配置加载阶段覆盖所有模型配置中的对应字段，免去临时修改配置文件。🔥🔥🔥
4. 新增推理响应异常检测功能：在推理评测的同时自动检测大模型响应中的生成异常，覆盖重复、乱码、生僻字、NaN 四类异常；命令行增加 --response-anomaly 即可零配置开启，检测结果按数据集落盘为 JSONL 供独立审计，不影响原有精度与性能指标。🔥🔥🔥 配置与使用详见 推理响应异常检测。
5. 支持投机推理（Speculative Decoding）性能评测：在既有性能评测流程中新增投机推理指标采集、计算与展示，详见 投机推理性能评测。🔥🔥🔥

👉 Click Here For Details

📦 Docker Images For This Release

Agent Runtime Images For This Release

name	python version	offline resources	image size	offline resource size
v3.1-20260912-master-ubuntu24.04-py312	3.12	aarch64 x86_64	aarch64: 3.12 GB x86_64: 3.31 GB	aarch64: 873 MB x86_64: 957.92 MB

👉 Click Here For Agent Runtime Docker Images Usage Guidance

Base Images For This Release

name	python version	offline resources	image size	offline resource size
v3.1-20260912-master-openeuler22.03-py310	3.10	aarch64 x86_64	aarch64: 3.01 GB x86_64: 3.17 GB	aarch64: 822 MB x86_64: 902 MB
v3.1-20260912-master-ubuntu22.04-py310	3.10	aarch64 x86_64	aarch64: 2.38 GB x86_64: 2.59 GB	aarch64: 653 MB x86_64: 723 MB
v3.1-20260912-master-openeuler24.03-py311	3.11	aarch64 x86_64	aarch64: 2.75 GB x86_64: 2.95 GB	aarch64: 759 MB x86_64: 849 MB
v3.1-20260912-master-ubuntu24.04-py312	3.12	aarch64 x86_64	aarch64: 2.37 GB x86_64: 2.75 GB	aarch64: 701 MB x86_64: 777 MB

👉 Click Here For Base Docker Images Usage Guidance

📄 Document For This Release

https://ais-bench-benchmark-rf.readthedocs.io/zh-cn/v3.1-20260912-master/

What's Changed

• [Bugfix] Add .gitignore rules to exclude downloaded dataset data files in ais_bench/datasets/ by @yejj710 in #386
• limit max_out_len in (0, 131072], remove it by @wenba0 in #388
• refactor(swebp): 重构swebp镜像操作逻辑，优化镜像管理粒度 by @ivanbao9783 in #385
• [feature] Support docker in docker by @SJTUyh in #392
• refractor(swebp): Optimize customer-facing text display of swebp eval by @ivanbao9783 in #400
• docs: add AA-LCR dataset README and datasets.md entry by @jschen069 in #415
• Aa lcr 0723 by @jschen069 in #411
• feat: add IFBench dataset and config by @jschen069 in #417
• docs: add IFBench dataset README and datasets.md entry by @jschen069 in #416
• 【feature】Support Multi Lora Infer by @SJTUyh in #408
• ifbench添加数据集评测任务说明 by @jschen069 in #423
• 0729 fixprefill disp by @jschen069 in #427
• 解决硬编码问题 by @jschen069 in #436
• Modify GPQA postprocess to return last answer match by @Keithwwa in #442
• feat(perf): add sepc-decode benchmark by @ivanbao9783 in #433
• fix(doc): add agent dataset in en and zh_cn docs by @ivanbao9783 in #448
• 修改nltk加载包的路径 by @jschen069 in #439
• 修改ifbench/aa_lcr描述 by @jschen069 in #440
• [CICD] add PR checker workflow by @SJTUyh in #451
• 非流式不展示ttft和tpot指标 by @jschen069 in #444
• feat(spec-decode): add spec-decode docs by @ivanbao9783 in #445
• [feature]geo3k需求交付 by @jschen069 in #453
• [doc]添加geo3k资料 by @jschen069 in #459
• fix the logic for displaying steady-state test results by @jschen069 in #464
• delete redundent pkg by @jschen069 in #461
• [Feature]AISBench支持大模型响应异常检测 by @Libotry in #468
• [DOCS] Add docs of terminal-bench 2.1 by @SJTUyh in #476
• [Feature] Added herding_coreset_selector, a tool for compressing large model quantization evaluation sets. by @flame-hu in #474
• feat(output): add origin_top_logprobs field in vllm custom api by @ivanbao9783 in #458
• feat(simulator): add /metrics endpoint for spec-decode exception testing by @ivanbao9783 in #472
• [Docs] Add recommand instruction for custom configs by @SJTUyh in #349
• docs: remove response anomaly design document from docs tree by @Libotry in #486
• fix: response anomaly model_name fallback and CLI-only enable switch by @Libotry in #487
• [feature] Support api models useful cmd by @SJTUyh in #492
• 【feature】Add corpusQA 1M dataset by @Keithwwa in #495
• generate/analysis prefix cache by @jschen069 in #489
• docs: restructure response anomaly detection docs; remove empty payload_staging shell by @Libotry in #502
• docs: announce response anomaly detection feature in README news by @Libotry in #506
• 【feature】Add mrcr 1M dataset by @Keithwwa in #512
• 【feature】【docs】补充mrcr数据集資料 by @Keithwwa in #515
• [AISBench] New harbor adapter by @SJTUyh in #503
• Prefix cache performance testing and performance metrics query by @jschen069 in #509
• [feature] harbor args support --extra-docker-compose by @SJTUyh in #519
• feat(deepswe): 新增DeepSWE数据集接入 by @ivanbao9783 in #522
• feat(cfg): 统一harbor task模版和summary by @ivanbao9783 in #528
• [bugfix] Fix Debug level not take effect in infer & eval task by @SJTUyh in #529
• [docs] fix some docs stupid mistakes by @SJTUyh in #530
• feat: 新增 agent-runtime 镜像（harbor-only）与双模式拉起脚本 by @Huafubing in #531
• [bugfix] add docker buildx in base dockerfile & ak ae append bugfix by @SJTUyh in #534
• [CI/CD][Docs] Add scripts for agent runtime image release & some docs for pre-release version by @SJTUyh in #537

New Contributors

• @jschen069 made their first contribution in #415
• @flame-hu made their first contribution in #474
• @Huafubing made their first contribution in #531

Full Changelog: v3.1-20260630-master...v3.1-20260912-master