Local API, Getting "No support for logits". How to run command line with generate_until only?

[lxe]

I'm trying to run this against a local instance of oobabooga exposing openai-like API:

OPENAI_API_KEY=x lm_eval --model local-chat-completions --model_args model=mymodel,base_url=http://127.0.0.1:5000/v1 --tasks lambada_openai

I'm getting

  File "/home/lxe/lm-evaluation-harness/lm_eval/models/openai_completions.py", line 499, in loglikelihood
    raise NotImplementedError("No support for logits.")
NotImplementedError: No support for logits.

I see in the docs that loglikelihood request type isn't supported, but I can't find anywhere in the docs about how to configure the harness to only do generate_until using the command line interface.

[Svjatoslav]

Did you manage to solve the problem?

[0-hero]

#1174 (comment)

[yaronr]

Is it now possible to run tests on self-hosted llm instances that either do or don't return logits?
I'm getting the same error and can't figure out how to advance.