Skip to content

Issues: bigcode-project/bigcode-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

28 Open 95 Closed
28 Open 95 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

MBPP Llama3-8B-Instruct lower pass@1 score expected
#246 opened Jun 16, 2024 by YangZhou08
1
Using custom prompts and postprocessing
#245 opened Jun 14, 2024 by anil-gurbuz
1
API-based evaluation support (humanevalpack_openai.py is too old)
#234 opened May 10, 2024 by s-natsubori
If I want to add my own designed prompts before each question, how should I modify the code
#230 opened Apr 27, 2024 by ALLISWELL8
1
The results of Llama3-8b pass@1 is worse than report
#228 opened Apr 22, 2024 by shuaiwang2022
6
ignore --use_auth_token if model doesn't require it
#227 opened Apr 21, 2024 by Vipitis
[FR] include "config" data in generations_only
#226 opened Apr 21, 2024 by Vipitis
Multiple-E Go test file name suffix does not contain _test.go
#224 opened Apr 20, 2024 by hitesh-1997
1
Support for vLLM
#221 opened Apr 18, 2024 by noforit
Please add flag to log score for each sample (akin to Eleuther's LM Evaluation Harness)
#215 opened Apr 8, 2024 by RylanSchaeffer
3
Finetune starcoderbase-1b
#214 opened Apr 6, 2024 by SummCoder
MultiPL-E generations step is hung
#213 opened Apr 3, 2024 by Santhoshkumar-p
why change n_copies from 1 to 2?
#209 opened Mar 25, 2024 by Reeleon
Support for StudentEval Dataset (Again)
#198 opened Feb 12, 2024 by guanqun-yang
1
AATK process_results is missing
#197 opened Feb 9, 2024 by adiprasad
@terryyz
4
To evaluate Github copilot?
#195 opened Jan 31, 2024 by liw8hz
[FEATURE REQUEST] Support HumanEval+ tests for MultiPL-E
#193 opened Jan 27, 2024 by Randl
Potentially extra slow inference when using LoRA adapter
#192 opened Jan 25, 2024 by sadaisystems
1
Evaluation speed with multi-gpu
#169 opened Nov 25, 2023 by Cheungki
2
exact commandline to reproduce leaderboard
#162 opened Nov 13, 2023 by Derekglk
2
A common interface for APIs and Models.
#161 opened Nov 12, 2023 by Anindyadeep
9
Is it possible to run the harness against API hosted models?
#148 opened Oct 16, 2023 by pnewhook
3
Update MultiPL-E docker image
#113 opened Jul 5, 2023 by loubnabnl
investigate discrepancy in odex implementation good first issue Good for newcomers
#68 opened Apr 30, 2023 by loubnabnl
Would be better to save generations on the fly enhancement New feature or request
#58 opened Apr 9, 2023 by Muennighoff
ProTip! Find all open issues with in progress development work with linked:pr.