Implement the Natural Questions evaluation

[StellaAthena]

From the GPT-3 paper

In this section we measure GPT-3’s ability to answer questions about broad factual knowledge. Due to the immense amount of possible queries, this task has normally been approached by using an information retrieval system to find relevant text in combination with a model which learns to generate an answer given the question and the retrieved text. Since this setting allows a system to search for and condition on text which potentially contains the answer it is denoted “open-book”. [RRS20] recently demonstrated that a large language model can perform surprisingly well directly answering the questions without conditioning on auxilliary information. They denote this more restrictive evaluation setting as “closed-book”. Their work suggests that even higher-capacity models could perform even better and we test this hypothesis with GPT-3. We evaluate GPT-3 on the 3 datasets in [RRS20]: Natural Questions [KPR+19], WebQuestions [BCFL13], and TriviaQA [JCWZ17], using the same splits. Note that in addition to all results being in the closed-book setting, our use of few-shot, one-shot, and zero-shot evaluations represent an even stricter setting than previous closed-book QA work: in addition to external content not being allowed, fine-tuning on the Q&A dataset itself is also not permitted.

• Data processing code implemented
• Evaluation implemented

The evaluation code should be modeled after the interface in lm_eval/base.py and the example of the BoolQ task in lm_eval/tasks/suerglue.py

[cfoster0]

Note: HuggingFace includes this in its datasets package.

https://huggingface.co/datasets/natural_questions

[cfoster0]

Warning: This dataset is super big.

[StellaAthena]

Warning: This dataset is super big.

How big is “super big”?

[cfoster0]

97G.

[sdtblck]

what the fuck. Why are we not training on this.

[sdtblck]

Ah, dev set is only 1G. But we should add train set to the pile.

[cfoster0]

We would need to dedupe this with Wikipedia, since the bulk of it is just the HTML of Wikipedia pages.

[moirage]

I can claim this

[StellaAthena]

I can claim this

Assigned!

[cr458]

would love to take this on if help on implementing the evaluation is still needed?

[StellaAthena]

would love to take this on if help on implementing the evaluation is still needed?

Yes this would be quite helpful. Thanks!

[haileyschoelkopf]

I think Natural Questions is implemented already? https://github.com/EleutherAI/lm-evaluation-harness/blob/master/lm_eval/tasks/naturalqs.py

[juletx]

@haileyschoelkopf Some methods are not implemented, they raise NotImplementedError

[haileyschoelkopf]

Ah you're right sorry!--I'm not sure why this was originally merged then. It's not in the task registry though so it should be alright to keep in the repo until the refactor is done, at which point we can decide what to do with it

[memray]

I wonder what the progress of NQ eval is and if any help is needed?

[StellaAthena]

@memray I am under the impression that is hasn't been implemented and help is need.

[wwngh1233]

+1

[Sea-Snell]

+1

[haileyschoelkopf]

Closed by #789 which implements the NaturalQs dataset split used by Llama and (possibly, unconfirmed) used by PaLM and more!