Fallback Model

[sydney-runkle]

To mitigate issues with unpredictable LLM downtime: we'd like to add a new model that supports a list of models to use as fallbacks if the first doesn't work.

[samuelcolvin]

I think the API can be as Simple as

model = FallbackModel('openai:gpt-4o', 'claude-3-5-haiku-latest', GeminiModel(...))
agent = Agent(model, ...)

E.g. the signature of FallbackModel is something like

class FallbackModel:
    def __init__(self, default_model: models.Model | models.KnownModelName, *fallback_models: models.Model | models.KnownModelName):
        ...

[phil65]

Implemented a pydantic-ai FallbackModel some time ago here, in case there is interest:
https://github.com/phil65/LLMling-models
Cheers!

[samuelcolvin]

Another advantage of having a consistent way to detect 503 etc. responses, is we can use it to mark live tests as xfail when the underlying model fails.

[phil65]

It´s actually quite funny what kind of things all can be done at model level. I now added a model to
to stream from another remote pydantic-ai model using FastAPI . 😄 (code might be sub-optimal though, just learning FastAPI a bit)

[sydney-runkle]

Will be starting work on this later today. PR to come soon 👍

[sydney-runkle]

Implemented a pydantic-ai FallbackModel some time ago here, in case there is interest:
https://github.com/phil65/LLMling-models
Cheers!

Awesome, thanks so much @phil65!