[Feature Request] Add support for logprobs to the mlx_lm server

[chimezie]

I have begun a PR request to lm-evaluation-harness to support MLX models, but have become bogged down by details regarding many things that are already implemented in mlx_lm. As I have become more familiar with that framework, it seems all we really need to be able to produce evaluations of MLX models over OpenAI is to add support for logprobs to the current server infrastructure.

This would greatly benefit the feedback loop of using MLX to train and evaluating them in a semi-standard way against other models.

I would be happy to contribute a PR, but just need some pointers regarding the relationship between Log probabilities of output tokens (i.e,. the "likelihood of each token occurring in the sequence given the context. To simplify, a logprob is log(p), where p = probability of a token occurring at a specific position based on the previous tokens in the context.") and the token probabilities we already return from mlx_lm.utils.generate_step

[awni]

The probabilities that gets returned is the probability of the given token at that time step. To get the log probabilities you would just take the log of it mx.log(p).

[awni]

I would be happy to contribute a PR,

Thanks! It would be great to ease the path to evaluating models in MLX.

[chimezie]

Sounds good. I will contribute a PR for the server. Thanks for the input.