AI feedback using logprobs

[tom-doerr]

Whenever I use a LM to evaluate something I ask a Yes/No question and do logprob_yes - logprob_no. This worked very well for me in the past. How can I access logprobs in my metric function?

[arnavsinghvi11]

Hi @tom-doerr , logprobs are indeed quite useful. Currently, DSPy doesn't support returning additional parameters besides the response content, but you can extract the logprobs from the response (subject to different LM configurations).

Feel free to push a PR that enables some flag for returning completions with logprobs while maintaining internal behavior!

[arthurgreef]

This no longer works with the LiteLLM client on 2.5.13.

output = gpt("Hello, world!", logprobs=1)
print (output)

output = gpt("Hello, world!", logprobs=True, top_logprobs=1)
print (output)

Expected: Logprobs after the text object.
Actual: Just the text: ['Hello! How can I assist you today?']

[tranhoangnguyen03]

from google.colab import userdata

lm = dspy.OpenAI(
    model='gpt-4o', 
    max_tokens=512, 
    api_key=userdata.get('openai_api_key')
)

dspy.settings.configure(lm=lm)

system_prompt = "answer in a single word.\n"
test_text = system_prompt + "The answer to the ultimate question to the meaning of life, the universe, and everything is ..."
test_output = lm(test_text, logprobs=True, top_logprobs=20)
print("test_output:", test_output)

The script above is working in terms of outputing the logprobs, but it triggers the deprecation warning.

from google.colab import userdata

lm = dspy.LM(
    'openai/gpt-4o', 
    max_tokens=512, 
    api_key=userdata.get('openai_api_key')
)
dspy.configure(lm=lm)

system_prompt = "answer in a single word.\n"
test_text = system_prompt + "The answer to the ultimate question to the meaning of life, the universe, and everything is ..."
test_output = lm(test_text, logprobs=True, top_logprobs=20)
print("test_output:", test_output)

This script doesn't produce the logprobs.

Are there plans to fix this? Is there a way for me to apply a patch locally on my machine to hotfix this?

[okhat]

Support for logprobs is merged now in dspy.LM and the adapters/modules.