Repository navigation
OpenCompass v0.5.5 Release Notes
🌟 Highlights
✨ Integrated More Benchmarks: Integrated new benchmarks spanning physics, reasoning, long-context, alignment, hallucination, maritime, biomedical, translation and multimodal evaluation, including CritPt, General365, AA-OmniScience v1.0 and v1.1, AA-LCR v1.1, HLE-Diamond, arena-hard-v2, ReasonZoo, LG-Bench, CHHallu-Src, GrandPhysics, PromptCBLUE Life Science, MaritimeBench, WMT19, LogiQA and ethics, plus ChartQAPro, MicroVQA, SFE and SciDocBench from VLMEvalKit (#2610, #2626, #2658, #2713, #2362, #2256, #2690, #2525, #2118, #2073, #2018, #1503, #516, #495, #2665).
✨ Documentation Refactor: Restructured the Chinese and English documentation into a new layout (get started, user guides, evaluation, execution, extension, tools and FAQ), rewrote the README, installation and quick start guides, and added configs for the latest models (#2666).
✨ Refactored CI: Refactored the mock API tests, fixed the PR stage CI, and extended the PR stage CI to cover Python 3.9 and 3.12 (#2698, #2670, #2715).
✨ Dropped Python 3.8 and Fixed Python 3.9 Compatibility: Removed Python 3.8 support and fixed the PEP 604 annotations that broke imports on Python 3.9 (#2715, #2674).
🚀 New Features
🔧 Added new benchmark configurations: CritPt, General365, AA-OmniScience, AA-LCR v1.1, HLE-Diamond, arena-hard-v2, ReasonZoo, LG-Bench, CHHallu-Src v1, GrandPhysics, PromptCBLUE Life Science, MaritimeBench, WMT19, LogiQA and ethics, with AA-LCR, AA-OmniScience (v1.1) and HLE-Diamond configs and the prompt template updated (#2610, #2626, #2658, #2713, #2362, #2256, #2690, #2525, #2118, #2073, #2018, #1503, #516, #495).
🔧 Added ChartQAPro, MicroVQA, SFE and SciDocBench datasets through VLMEvalKit (#2665).
🔧 Added retry and verbose options to the TurboMind API for get_ppl and get_ll (#2650, #2652).
🐛 Bug Fixes
🔧 Fixed reasoning parsing in the OpenAI, OpenAISDK and OpenAISDKStreaming models, and surfaced inference failures in OpenICLInferConcurrentTask (#2653, #2651).
🔧 Fixed dataset loading and downloads for AA-LCR, MBPP-CN, C-Eval (ModelScope and HF generation for causal LMs) and SciReasoner 1.5 (with a new llm_extract config), and fixed MATH JudgeLLM config imports (#2630, #2695, #2416, #2640, #2662).
🔧 Fixed VLMEvalKit empty predictions and JSONL resume with Unicode separators (#2691).
⚙ Enhancements and Refactors
⚙ Refactored the OpenCompass documentation (#2666).
⚙ Refactored the mock API tests and fixed the PR stage CI (#2698, #2670).
⚙ Dropped Python 3.8 support, added Python 3.9 and 3.12 to the PR stage CI, and fixed PEP 604 annotations that broke imports on Python 3.9 (#2715, #2674).
⚙ Fixed VLM installation dependencies and requirement declarations (#2679, #2676).
🎉 Welcome New Contributors
A warm welcome and special thanks to our newest contributors who made this release possible:
- @alleyxd made their first contribution in #2684
- @DeliWang made their first contribution in #2416
- @avalyset made their first contribution in #2618
- @BeingIsA made their first contribution in #2253
- @K-zhy made their first contribution in #2018
- @Xiao-Youth made their first contribution in #2118
- @JimmyWang0417 made their first contribution in #2669
- @ktwu01 made their first contribution in #2177
- @mkzung made their first contribution in #2594
- @Owen-Qin made their first contribution in #2216
- @SashaMIT made their first contribution in #2661
- @ShikangPang made their first contribution in #2187
- @DewidPig made their first contribution in #1677
- @WatchTree-19 made their first contribution in #2523
- @YL1N made their first contribution in #2585
- @JensenDong made their first contribution in #1811
- @YueshenZ made their first contribution in #2662
- @HYZ17 made their first contribution in #2218
- @ZhangXiangyi0523 made their first contribution in #2690
- @zhangzefeng92 made their first contribution in #1819
- @lizhuojunx86 made their first contribution in #2525
- @bajiu-bajiu made their first contribution in #2599
- @bjptw made their first contribution in #2674
- @freemedom made their first contribution in #2374
- @greedylearnerG made their first contribution in #2694
- @linhongyu510 made their first contribution in #2656
- @moonturbo made their first contribution in #2612
- @cnlnpjhsy made their first contribution in #2340
Full Changelog: 0.5.4...0.5.5
Thank you to all contributors who made OpenCompass v0.5.5 possible!
