v0.2.0.post1

What's Changed

• Include VCR by @tianyu-z in #105
• [Small Update] Update the version of LMMs-Eval by @pufanyi in #109
• add II-Bench by @XinrunDu in #111
• Q-Bench, Q-Bench2, A-Bench by @teowu in #113
• LongVideoBench for LMMs-Eval by @teowu in #117
• Fix the potential risk by PR #117 by @teowu in #118
• add tinyllava by @zjysteven in #114
• Add docs for datasets upload to HF by @pufanyi in #120
• [Model] aligned llava-interleave model results on video tasks by @Luodian in #125

New Contributors

• @tianyu-z made their first contribution in #105
• @XinrunDu made their first contribution in #111
• @teowu made their first contribution in #113
• @zjysteven made their first contribution in #114

Full Changelog: v0.2.0...v0.2.0.post1

v0.2.0

What's Changed

• pip package by @pufanyi in #1
• Fix mmbench dataset submission format by @pufanyi in #7
• [Feat] add correct tensor parallelism for larger size model. by @Luodian in #4
• update version to 0.1.1 by @pufanyi in #9
• [Tasks] Fix MMBench by @pufanyi in #13
• [Fix] Fix llava reproduce error by @kcz358 in #24
• add_ocrbench by @echo840 in #28
• Joshua/olympiadbench by @JvThunder in #37
• [WIP] adding mmbench dev evaluation (#75) by @Luodian in #46
• Add llava model for 🤗 Transformers by @lewtun in #47
• Fix types to allow nullables in llava_hf.py by @lewtun in #55
• Add REC tasks for testing model ability to locally ground objects, given a description. This adds REC for all RefCOCO datasets. by @hunterheiden in #52
• [Benchmarks] RealWorldQA by @pufanyi in #57
• add Llava-SGlang by @jzhang38 in #54
• Add MathVerse by @CaraJ7 in #60
• Fix typo in Qwen-VL that was causing "reference before assignment" by @tupini07 in #61
• New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens by @hunterheiden in #63
• [New Task] WebSRC (multimodal Q&A on web screenshots) by @hunterheiden in #69
• Bugfix: WebSRC should be token-level F1 NOT character-level by @hunterheiden in #70
• Multilingual LLava bench by @gagan3012 in #56
• [Fix] repr llava doc by @cocoshe in #36
• add idefics2 by @jzhang38 in #59
• [Feat] Add qwen vl api by @kcz358 in #73
• Adding microsoft/Phi-3-vision-128k-instruct model. by @vfragoso in #87
• Add MathVerse in README.md by @CaraJ7 in #97
• add MM-UPD by @AtsuMiyai in #95
• add Conbench by @Gumpest in #100
• Update conbench in README by @Gumpest in #101
• update gpt-3.5-turbo version by @AtsuMiyai in #107
• [Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval by @Luodian in #108

New Contributors

• @pufanyi made their first contribution in #1
• @Luodian made their first contribution in #4
• @kcz358 made their first contribution in #24
• @echo840 made their first contribution in #28
• @JvThunder made their first contribution in #37
• @lewtun made their first contribution in #47
• @hunterheiden made their first contribution in #52
• @jzhang38 made their first contribution in #54
• @CaraJ7 made their first contribution in #60
• @tupini07 made their first contribution in #61
• @gagan3012 made their first contribution in #56
• @cocoshe made their first contribution in #36
• @vfragoso made their first contribution in #87
• @AtsuMiyai made their first contribution in #95
• @Gumpest made their first contribution in #100

Full Changelog: v0.1.0...v0.2.0

LMMs-Eval 0.1.0.dev

[Enhancement & Fix] Add Tensor Parallelism and Fix LLaVA-W/MMBench's issue.

LMMs-Eval 0.1.0 Release

Currently support 40+ evaluation datasets with 60+ subsets/variants, and 5 commonly used LMMs.