LiteLLM proxy model fallback not reflected in LM.history

[itay1542]

I am using a LiteLLM proxy which has fallback enabled, this means that failed requests to the desired model provider will automatically reroute to the default that is set on the proxy. When such a fallback happens, I expect dspy.LM.history to reflect the actual model that returned the response, and not the original. See example:

import dspy
real_model = "openai/mistral/pixtral-12b-2409"
lm = dspy.LM(real_model, api_key=litellm_api_key, api_base=litellm_proxy_url)
print(lm("Hello, how are you?"))
print("actual model:", lm.history[-1]['response'].model)
print("dspy history model:", lm.history[-1]["model"])

prints:
["Hello! I'm functioning well, thank you. I'm here to help. How about you? How are you doing today?"]
actual model: mistral/pixtral-12b-2409
dspy history model: openai/mistral/pixtral-12b-2409

non_existant_model = "openai/bla_bla"
lm = dspy.LM(non_existant_model, api_key=litellm_api_key, api_base=litellm_proxy_url)
print(lm("Hello, how are you?"))
print("actual model:", lm.history[-1]['response'].model)
print("dspy history model:", lm.history[-1]["model"])

prints:
["Hello! I'm just a computer program, so I don't have feelings, but I'm here and ready to help you. How can I assist you today?"]
actual model: gpt-4o-mini-2024-07-18
dspy history model: openai/bla_bla

Ill be happy to open a PR

[okhat]

Thank you @itay1542 !

I think users would generally expect that the LM name they pass to DSPy is the LM name DSPy records. It might be strange to record a different LM name based on the response from the server.

Maybe the history should save model and response_model, both?

[itay1542]

Thanks for the quick response @okhat :)

Yeah I think that adding response_model to history could be helpful. This way it will be possible for users to assert whether there was some problem with the original model and escalate if necessary. Should I open a PR?

[okhat]

Yes, thanks a lot @itay1542 ! Looking forward to it.