Error upon running coqa

[robinsongh381]

While running coqa task, i faced an error as follows

If I modify primary_until, _ into primary_until this generates another issue

Any solutions ?
Thanks

[igor0]

I hit this recently as well. I believe that the CoQA dataset only works with GPT-3, not with GPT-2. The problem is that:

1. CoQA uses a multi-token EOS marker. In coqa.py:
rf.greedy_until(ctx, ['\nQ:'])

2. The harness only supports greedy_until(..) with a single token. All other callers only provide a single token, like '.' or a newline.

    def greedy_until(self, requests):
        # TODO: implement fully general `until` that handles untils that are 
        #       multiple tokens or that span multiple tokens correctly

If you pass in a multi-token string, you get "ValueError: too many values to unpack (expected 1)". That's because only a single token is expected by the implementation.

3. GPT-3 uses a different implementation of greedy_until(..) that calls into the OpenAI APIs, and that seems to support multi-token greedy_until(..)

To confirm this, you can modify the call to rf.greedy_until(ctx, ['\nQ:']) in coqa.py to only pass in a single token, which will avoid the crashes, but presumably won't do the right thing as far as the benchmark is concerned.

So, to get CoQA working on GPT-2, a multi-token greedy_until will need to be implemented in base.py.