Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
149 Open 1,581 Closed
149 Open 1,581 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

Fix add_bos_token not updated for Gemma tokenizer
#3206 opened Aug 4, 2025 by DarkLight1337 Loading…
Support for DDP+MP with native torch and no accelerate
#3205 opened Aug 3, 2025 by xgal Loading…
1
feat: COT trace response handling in evaluator and model classes
#3204 opened Aug 3, 2025 by hhh2210 Loading…
1
Add new task: kmmlu_pro, kmmlu_redux
#3198 opened Aug 1, 2025 by jeonghodot Loading…
2
Add xnli_va dataset
#3194 opened Jul 30, 2025 by FranValero97 Loading…
1
Fixed #2552: Improve answer extraction for hendrycks_math
#3192 opened Jul 30, 2025 by JoonYong-Park Loading…
1
1
refactor registry
#3189 opened Jul 28, 2025 by baberabb Loading…
Leverage vllm's tokenizer_info endpoint to avoid manual duplication
#3185 opened Jul 25, 2025 by m-misiura Loading…
1
Add LM-SynEval Benchmark
#3184 opened Jul 24, 2025 by jmichaelov Loading…
1
Update MMLU-ProX task
#3174 opened Jul 22, 2025 by weihao1115 Loading…
3 of 6 tasks
1
feat: Add CLIcK task
#3173 opened Jul 22, 2025 by shing100 Loading…
1
Remove generate_until (multiple_target and doc_to_choice indexing) logic in ConfigurableTask.process_results
#3169 opened Jul 21, 2025 by baberabb Loading…
Add eqbench tasks in Spanish and Catalan
#3168 opened Jul 21, 2025 by priverabsc Loading…
4
Add EsBBQ and CaBBQ tasks
#3167 opened Jul 21, 2025 by valleruizf Loading…
1
Bugfix: set default SamplingParams based on generation_config
#3160 opened Jul 18, 2025 by cuttle-fish-my Loading…
2
Bugfix: update hellaswag ds path
#3158 opened Jul 18, 2025 by marawangamal Loading…
2
Feat/add permutation benchmark/task to lm-evaluation-harness
#3157 opened Jul 18, 2025 by BeeGass Loading…
5
Add task dynamic_ifeval
#3149 opened Jul 15, 2025 by davideguidobene Loading…
6
Add DETAILS.md for improved documentation
#3141 opened Jul 12, 2025 by ginylil-tech Loading…
1
Add tasklist
#3133 opened Jul 11, 2025 by baberabb Loading…
set repeat metrics from config
#3109 opened Jul 5, 2025 by baberabb Loading…
add metric configs
#3105 opened Jul 4, 2025 by baberabb Loading…
Add support for OpenVINO text2text generation models
#3101 opened Jul 3, 2025 by nikita-savelyevv • Draft
2
Adds Anthropic/discrim-eval to lm-evaluation-harness
#3091 opened Jun 27, 2025 by Helw150 Loading…
1
7
feat(api_models): add enable_thinking param in chat_template_kwargs
#3088 opened Jun 27, 2025 by johnsonafool Loading…
1
ProTip! Mix and match filters to narrow down what you’re looking for.