Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

61 Open 30 Closed
61 Open 30 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

ConnectionError: Couldn't reach 'lmms-lab/ScienceQA' on the Hub (ProxyError)
#154 opened Jul 19, 2024 by xmu-xiaoma666
BUG: when I test mme on internvl2, an error will occur：'InternVL2' object has no attribute 'accelerator'
#152 opened Jul 18, 2024 by daixiangzi
1
Results on video
#151 opened Jul 18, 2024 by liguopeng0923
TypeError: 'NoneType' object is not subscriptable
#144 opened Jul 12, 2024 by rixejzvdl649
1
Reproduction results of videollava in VideoMMe
#141 opened Jul 11, 2024 by adrianwestmoon
Reproduction results of llava v1.5 in MMBench CN
#139 opened Jul 11, 2024 by xzhouzeng
Error when evaluate llava-v1.5-7b[importing reka,flashattention error]
#135 opened Jul 5, 2024 by zhizhou57
1
The evaluation of videoMME is very slow
#134 opened Jul 2, 2024 by xiaofei05
2
The score is zero for seedbench/seed_video [Bug]
#132 opened Jul 1, 2024 by jungle-gym-ac
3
latency and throughput (tokens/sec) in evaluation enhancement New feature or request
#131 opened Jun 30, 2024 by hasanar1f
3
Adding MobileCaptureVQA to the benchmark
#127 opened Jun 20, 2024 by arnaudstiegler
1
[FAQ] about caching the zipped file from HF to local folder faq Questions and Answers
#124 opened Jun 20, 2024 by Luodian
Propose to print error information when import models
#123 opened Jun 20, 2024 by SDaoer
1
[Reproduce] Unable to reproduce AI2D, ChartQA and InfoVQA results for llava-1.6-mistral-7b
#122 opened Jun 20, 2024 by GoGoJoestar
Cannot find any code / link to worldqa dataset worldqa-related Any issue that is related to worldqa
#121 opened Jun 19, 2024 by Hasnat79
1
Can not load llava_hf models since the new updates!
#119 opened Jun 18, 2024 by hasanar1f
4
There is an inconsistency in the number of mmbench images help wanted Extra attention is needed
#116 opened Jun 15, 2024 by MuskAI
1
Unable to reproduce SQA results for llava-1.5 help wanted Extra attention is needed
#115 opened Jun 15, 2024 by clairez-cerebras
3
Error during videomme help wanted Extra attention is needed
#112 opened Jun 13, 2024 by yukang2017
2
building context takes too long question Further information is requested
#110 opened Jun 13, 2024 by simplelifetime
1
AssertionError: No tasks specified, or no tasks found. Please verify the task names when using local dataset path question Further information is requested
#106 opened Jun 11, 2024 by MuskAI
Why cot not supported for Mathverse
#104 opened Jun 9, 2024 by huiyeruzhou
Unreasonable data in AI2D dataset used for evaluation bug Something isn't working
#103 opened Jun 6, 2024 by yqy2001
3
Allow loading models and tasks from external packages
#102 opened Jun 6, 2024 by lorenzomammana
1
How to contribute a new dataset?
#99 opened Jun 5, 2024 by JohnTang93
1
ProTip! Updated in the last three days: updated:>2024-07-19.