Skip to content

Issues: open-compass/opencompass

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

104 Open 269 Closed
104 Open 269 Closed
Author
Filter by author
Label
Filter by label
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Milestones
Filter by milestone
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

[Bug] When testing on gen datasets, even if the output is empty or incorrect, unexpected scores can be obtained
#1232 opened Jun 7, 2024 by chairmanQi
2 tasks done
@tonysy
opencompass公开榜单更新[Feature]
#1231 opened Jun 7, 2024 by cobraheleah
1 task
@tonysy
1
大海捞针数据集初始化报错（ Failed to get opencompass.datasets.needlebench.origin.NeedleBenchOriginDataset）
#1229 opened Jun 6, 2024 by macheng6
2 tasks done
@kennymckormick
[Bug] run pytorch Qwen-7B-Chat with ARC-c ppl under CPU ,and result is not good
#1226 opened Jun 5, 2024 by FlexLaughing
2 tasks done
@liushz
1
[Bug] which version of the dataset should be selected When evaluating the Llama3 model,
#1223 opened Jun 3, 2024 by bullw
2 tasks done
@Leymore
[Bug] llm-compression task faild at eval stage with latest version
#1222 opened Jun 1, 2024 by mqy004
2 tasks done
@tonysy
3
[Bug] configs/datasets/agieval/agieval_mixed_713d14.py not found
#1220 opened May 31, 2024 by Galaxy-Husky
2 tasks done
@tonysy
[Bug] hf_chatglm3_6b评测AFQMC数据集时，自测结果与官方不一致。且自测结果不稳定。
#1202 opened May 28, 2024 by jieheroli
2 tasks done
@bittersweet1999
1
[Feature] config的bug，提示下载configs，然后下载了又出现以下bug
#1197 opened May 28, 2024 by 1518630367
1 task
@bittersweet1999
[Feature] 如何在needlebench 中使用api model？
#1196 opened May 28, 2024 by 1518630367
1 task
@DseidLi @bittersweet1999
1
[Feature] 支持openai/GPT4-o的评测seting
#1187 opened May 23, 2024 by Ezra-Yu
1 task
@tonysy
1
[Bug] 使用api测评时mode参数不起作用，超出max_seq_len并没有按mode切分输入
#1181 opened May 22, 2024 by wz0424
2 tasks done
@liushz
[Feature] Leval数据集少了两个config：codeU和sci_fi
#1179 opened May 21, 2024 by liuxize-dhu
1 task
@bittersweet1999
[Bug] CLUEWSC的测试结果全是 50%
#1174 opened May 21, 2024 by SefaZeng
2 tasks done
@bittersweet1999
[Feature] Add -f for evaluation stage
#1172 opened May 17, 2024 by liushz
1 task
@tonysy
[Bug] 根据官方的Quick Start运行，输出结果为空
#1168 opened May 16, 2024 by MansonHua
2 tasks done
@kennymckormick
10
[Bug] 评测 Hellaswag 失败
#1166 opened May 16, 2024 by SefaZeng
2 tasks done
2
自定义的模型API都要配置步骤有哪些，除了写config文件外
#1165 opened May 15, 2024 by zhe123tc
1 task
@liushz
3
[Bug] RACE数据集在hf-llama-7b上的评测结果异常
#1164 opened May 15, 2024 by Li-jiaxian
2 tasks done
@liushz
[Feature] MMLU-PRO
#1161 opened May 15, 2024 by Ezra-Yu
1 task
@liushz
1
the prompt for triviaqa dataset
#1158 opened May 15, 2024 by mengrusun
1 task done
@liushz
[Bug] run with transformers==4.40.2, error "HuggingFacewithChatTemplate does not support ppl-based evaluation".
#1157 opened May 15, 2024 by zhulinJulia24
2 tasks done
@Leymore
2
[Feature] AGIEval数据集是否可以支持few-shot评测设置？
#1150 opened May 14, 2024 by cabbagecabbage
1 task
@bittersweet1999
[Bug] math_gen数据集评估随机失败
#1148 opened May 14, 2024 by berton820
2 tasks done
@bittersweet1999
4
[Feature] Typos in the official document
#1147 opened May 13, 2024 by Galaxy-Husky
1 task done
@Leymore
1
ProTip! Exclude everything labeled bug with -label:bug.