-
Notifications
You must be signed in to change notification settings - Fork 396
add a task dump in registry for better documentation of tasks #1052
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR adds a task dump functionality to the registry that exports all task names, metadata, and docstrings as JSON for better documentation. The changes also reduce the number of Global MMLU task variations.
- Adds
get_tasks_dump()method to the Registry class to serialize task configurations and metadata - Adds a new
dumpCLI command to output task information as JSON - Reduces Global MMLU task variations by removing CF and Hybrid formulations and limiting to "ALL" sensitivity label
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/lighteval/tasks/registry.py | Adds instance variables to store initialization parameters and implements get_tasks_dump() method with docstring parsing and value serialization logic |
| src/lighteval/tasks/multilingual/tasks/global_mmlu.py | Removes unused imports (CFFormulation, HybridFormulation) and reduces task variations |
| src/lighteval/main_tasks.py | Adds new dump CLI command that uses the registry's task dump functionality to output JSON |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
No description provided.