OpenCompass v0.5.4 Release Notes
🌟 Highlights
✨ Multimodal Evaluation with VLMEvalKit: OpenCompass now integrates VLMEvalKit for native multimodal dataset loading, OpenAI-compatible inference, and official VLMEvalKit evaluation metrics (#2586).
✨ Multi-Round Evaluation: GenInferencer now supports multi-round inference, accompanied by the new Multi-IF benchmark for evaluating multi-turn instruction following (#2557).
✨ Expanded Model API Ecosystem: Added LiteLLM as an AI gateway backend and extended TurboMind API capabilities with PPL evaluation support (#2441, #2510).
✨ Broader Benchmark Coverage: Added new evaluations spanning finance, long-context understanding, alignment, information extraction, and code reasoning, including BuySideFinBench, ELBench, Inverse IFEval, AA-LCR, CRUXEval, and updated LongBench/MRCR V2 configurations (#2446, #2495, #2514, #2558).
🚀 New Features
🔧 Integrated VLMEvalKit datasets and evaluators into the OpenCompass workflow, with MMBench and MMMU-Pro evaluation examples (#2586).
🔧 Added multi-round inference to GenInferencer and introduced Multi-IF dataset support (#2557).
🔧 Added LiteLLM AI Gateway model support and the TurboMind API /get_ppl endpoint (#2441, #2510).
🔧 Added and expanded benchmark configurations for CRUXEval, LongBench v1, Qasper, MRCR v2, Fake-Alignment, and more (#2572, #2592, #2549, #2553).
🐛 Bug Fixes
🔧 Strengthened evaluation correctness by rejecting deceptive return values in code evaluators and preventing invalid SIQA generations from being scored as correct (#2565, #2450).
🔧 Fixed generation behavior across HuggingFace and vLLM, including max_out_len=None, default torch dtype loading, accelerator generation config conversion, and chat-template BOS handling (#2459, #1969, #2519, #2554).
🔧 Improved evaluator reliability for BBH free-form answers, LiveCodeBench memory limits, Wikitext PPL without references, and official Fake-Alignment prompts (#2548, #2538, #2532, #2567).
🔧 Fixed dataset loading, local path resolution, download mappings, and prompts across AGIEval, BoolQ, GSM8K, C-Eval, SVAMP, TACO, and other benchmark configurations (#2497, #2498, #2540, #2546).
⚙ Enhancements and Refactors
⚙ Improved multi-round observability by fixing per-turn statistics in dump-res-length and repeat detection for multi-round conversations (#2569).
⚙ Improved model API robustness for Qwen response extraction, DeepSeek retry timeouts, Claude custom headers, and TurboMind proxy compatibility (#2521, #2530, #2445, #2529).
⚙ Improved data and task handling with safer SizePartitioner cost calculation and corrected TopK retriever metadata collation (#2438, #2556).
⚙ Refactored CI workflows, pinned third-party CI actions, and removed misleading error logs from successful RJob submissions (#2570, #2533, #2588).
📦 Compatibility and Packaging
📦 Made OpenCC optional to restore Windows installation compatibility (#2512).
📦 Pinned the mmengine-lite runtime dependency for more reproducible installations (#2531).
📦 Resolved the Dingo/Pillow dependency conflict and fixed Dingo evaluation behavior (#2547, #2607).
🎉 Welcome New Contributors
A warm welcome and special thanks to our newest contributors who made this release possible:
- @Amber0614 made their first contribution in (#2495).
- @ziyangwang007 made their first contribution in (#2459).
- @cindy90 made their first contribution in (#2446).
- @irexyc made their first contribution in (#2510).
- @Hibbert133 made their first contribution in (#2426).
- @RheagalFire made their first contribution in (#2441).
- @octo-patch made their first contribution in (#2438).
- @HXC-Cloud-lab made their first contribution in (#2549).
- @kobihikri made their first contribution in (#2533).
- @thecaptain789 made their first contribution in (#2397).
- @SongYippee made their first contribution in (#2445).
- @hongleng made their first contribution in (#2556).
- @thinszx made their first contribution in (#2586).
- @Danielxu0208 made their first contribution in (#2578).
Full Changelog: 0.5.3...0.5.4
Thank you to all contributors who made OpenCompass v0.5.4 possible!
