Filter docs not offset by doc_id

[baberabb]

closes #1293. The full dataset was passed to the filter object and not offset by doc_id when using accelerate. I've made a tentative fix by using only the docs from the instances but I think a better fix would be to remove this arg altogether and keep a clear distinction between it and process_results. Is there a backward compatibility reason not to do that? Do any other tasks other than super_glue-wsc-t5-prompt compare resps to docs in theirfilter function?

[haileyschoelkopf]

Thanks for catching this!

Hm, so we do want the filters to be able to have access to the gold standard answer in some cases, if people will use them for model-based evaluations. The idea was that if you want to do something like run a model over all the documents, the filter apply() should permit you to load that model a single time and run over all the docs, something not currently possible with process_results().

I think that your fix is probably the right way to currently handle this as a hotfix but sympathetic to what you're saying re: keeping a distinction between process_docs(). I guess the broader change would be to: - move this WSC logic to a custom process_results() - put a deprecation warning if people try to pass docs into filters, and later fully phase it out?

(cc @dwadden who was interested in model-based evaluation in lm-eval, to make sure this change would not break anything for him, and @pminervini who this maybe also affects due to an interest in stateful process_results())

[baberabb]

Yes, that makes sense. I guess there isn't a really a reason not to give the option other than to keep things simple. As a compromise, I've kept the Filter class as is but modified the input docs in FilterEnsemble to args.docs (this is the main entrypoint to the filter pipeline from Task). The only major change I can think of is that the gold standard answers will be list[dict] (one for the respective row) rather than a Dataset object.

Removed some unnecessary and unused looping in task.py while parsing the filters from the config. The results remain the same for TinyLLama on gsm8k, before and after this change.

Also modified the super_glue-wsc-t5-prompt answer processing to use process_results rather than Filter. Now accuracy is the same with and w/o accelerate (0.6058 on TinyLLama)

cc: @lintangsutawika

[haileyschoelkopf]

As a compromise, I've kept the Filter class as is but modified the input docs in FilterEnsemble to args.docs (this is the main entrypoint to the filter pipeline from Task). The only major change I can think of is that the gold standard answers will be list[dict] (one for the respective row) rather than a Dataset object.

Thanks, I think this sounds great in terms of keeping backward compatibility!

This looks good to me, thanks again for chasing this down!

[lintangsutawika]

Thanks @baberabb @haileyschoelkopf