New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add context-based requests processing #1571
base: main
Are you sure you want to change the base?
Conversation
It is also important to define the behaviour of ConfigurableTask so that user could define context-based task with yaml file, isn't it? |
I have added support for yaml tasks format. Now context-based task can be passed via task.yaml file. ContextInstance is called only for context-based tasks, that are defined by specific flag inside the task. Old tasks are not affected and use original Instance class. |
@haileyschoelkopf is there anything I can do to enhance the PR and speed up the merging process? If there is anything I can add to make the PR look better, the code work more efficiently or avoid any confusion I will do it. I am open for ideas and comments :) |
@haileyschoelkopf merged recent updates from main so that it was easier to review the changes I'm proposing |
Left a comment about this PR and the feature in discord! |
is this primarily about multi-step / multi-round problems? |
@StellaAthena hi! Yes, when first introducing these changes I was thinking about multi-step prompting things. Like described here: https://arxiv.org/pdf/2305.14279.pdf |
@haileyschoelkopf can you give more information so that I could develope this PR? :) |
This PR is designed to accomodate primarily different variants of multi-step and multi-round tasks. This way it suggests flexible functions to update the internal storage (keeps info about previous requests of the dataset) and to update the currect request (takes info from storage). With minor changes can close #1816 issue (need to envisage managing the amount of requests so that user can add more requests following some condition for it without affecting other context-based tasks). I see it working this way. Eample of yaml task:
|
Examples of funcs:
|
@haileyschoelkopf i would gladly incorporate my ideas in existing lm-harness features! |
@artemorloff I like the general idea but I'm missing some context here. Could describe a few types of tasks that would benefit from this type of evaluation (I've seen a recent issue that provided an example, but I'm wondering if there are other types of tasks that would benefit from this pattern) |
@lintangsutawika thank you for reply! there emerging a lot of works and tasks that use multi-step/multi-round strategy. The goal is to allow users right now start creating these tasks and find surely existing ways to enhance it. Some examples of tasks and ideas using this strategy:
all this works refer to other works, so I think multi-step (round) reasoning is quite important and may allow for new tasks in LM-harness. |
my PR introduces a quite straightforward approach to the issue. All such tasks are processed one by one to avoid possible problems with batches failing GPU memory, multiple batch size computations. |
The PR add support for new type of tasks - context-based tasks.
Motivation. Some tasks and CoT strategies may require knowing the answer of the model for the previous question to form the current request. Till now it is impossible to implement such tasks without changing
evaluator.py
(or models/*.py file) which leads to inability to use lm-evaluation-harness as an external library (user cannot directly pass new evaluator.py instead of the default one while running the task). This PR changes it.How it works. All requests are split into two meta-groups: regular tasks and context-based tasks. Each group is processed separately. No changes in processing regular tasks. For context-based tasks after preparing requests each request is updated, processed through the model, the external storage is updated. If no context-based tasks claimed, the loop for processing them is not accessed. So, the workflow for all existing tasks has not been changed.
Also, to encompass new functionality a new instance class is added: ContextInstance. It inherits from regular Instance and adds only two new methods: update_request that takes storage and request and does something to the request right before passing it into the model so that the changes are available with
--log_samples
flag. Old tasks that use Instance are not affected. New class is meant to avoid confusion between regular and context-based tasks instances.To indicate that task is context-based the new attr is used. It shouldn't be False by defalut, as while running all tasks the presence of this attr and its value is checked. So, no changes needed to run existing tasks, no way old tasks will be run through a new loop.
All tests are passed successfully. No need in changes for different models. The only problem that may happen: each time while calling the model a new progress bar apears. This can be solved by merging #1569
Closes: #1432 #1537