Add local-completions support using OpenAI interface

[mgoin]

Since ChatCompletions doesn't allow for loglikelihood support, adding a locally hosted OAI-like server interface for Completions will allow for an easier interface to get loglikelihood for open-source models

[haileyschoelkopf]

Thanks very much for the PR!

Would you be willing to make the following changes:

• Add a tokenizer_backend kwarg, as existed in https://github.com/EleutherAI/lm-evaluation-harness/pull/1186/files for ChatCompletions ? This would allow for HF tokenizers to be used for this model.
• Run the linters

If so, then I think we're all set!

[mgoin]

Thanks for the fast review @haileyschoelkopf ! I also added a tokenizer input to specify the transformers model_id to use for the tokenizer if the model name doesn't match a transformers model_id exactly. Maybe we could remove tokenizer_backend and just use transformers if a tokenizer is defined? I'll share an example of how I tested this with deepsparse

Server:

deepsparse.server --integration openai --task text-generation --model_path hf:mgoin/llama-2-7b-gsm8k-pruned60-quant-ds

Client lm-eval:

lm_eval --model local-completions --model_args base_url=http://localhost:5543/v1,model=hf:mgoin/llama-2-7b-gsm8k-pruned60-quant-ds,tokenizer_backend=huggingface,tokenizer=mgoin/llama-2-7b-gsm8k-pruned60-quant-ds --tasks gsm8k --num_fewshot 0

[haileyschoelkopf]

Maybe we could remove tokenizer_backend and just use transformers if a tokenizer is defined?

Sounds good to me! We may want to add a check or log message warning people in the event they pass base_url but not tokenizer to initialization though, or a log message saying what tokenizer name + backend type are being used.

You may need to copy this handling in for the HF tokenizer case though:

https://github.com/EleutherAI/lm-evaluation-harness/blob/89618bf8421d27c8cf28004d616b33fc5b305ceb/lm_eval/models/huggingface.py#L228C9-L239C76

As some models don't have an EOS token set (and some, like Qwen, have custom tokenizer code that may not allow the addition of new special tokens.)

[haileyschoelkopf]

Also cc @veekaybee @gmottajr this PR updates local-completions

@mgoin does Deepsparse's OpenAI server support echo=True for completions logits? If so, will it continue to do so?

[mgoin]

Hey @haileyschoelkopf I'll add in the tokenizer messaging today. I don't think we've implemented echo yet, and I now see it's required, but I think we should definitely do it. Let me get back to you on time to implement

[haileyschoelkopf]

Thanks again for the contribution @mgoin !

Let me know if you get a chance to check out echo=True soon.

[mgoin]

Hey @haileyschoelkopf sorry for this falling off my radar, I had the changes but forgot to push 😅 thanks for finishing it!

I think for the server, rather than using the openai-style interface we'll actually just submit native local model backends for deepsparse and sparseml. Trying to stabilize the interface for sparseml to submit my fork next week

[haileyschoelkopf]

No problem!

I think for the server, rather than using the openai-style interface we'll actually just submit native local model backends for deepsparse and sparseml. Trying to stabilize the interface for sparseml to submit my fork next week

fantastic! Hopefully these can be used as templates for other local server LMs, in the event that using the OpenAI style interface blocks echo=True.