Model proxy/gateway/router

[zachmayer]

Description

Braintrust and Cloudflare, among others, provide AI gateways or proxies that do things like caching your requests and routing them between providers.

I've had some difficulty getting both of these to work correctly with Pydantic, mostly because of small differences in abstractions and programming styles.

I'd really love to see an AI proxy or gateway as part of the Pydantic AI ecosystem. I know this is not a small ask, but it'd be very useful to me.

References

https://www.braintrust.dev/docs/guides/proxy

https://developers.cloudflare.com/ai-gateway/

[samuelcolvin]

I'm friends with cloudflare, let's just make theirs work well.

Cc @threepointone.

If we can't use Cloudflare's router, we'd just build it on cloudflare anyway, so let's try to use theirs first

[threepointone]

Both things are true.

• You're friends with Cloudflare. (And we're friends with you ♥️)

• We should make sure our gateway works great ootb with pydantic ai.

Bit swamped with the developer week stuff next week, but I'll get this issue on the internal chat asap.

[skye0402]

It would be great to have a better abstraction of the models. PydanticAI is a fantastic framework but I really struggle with how models are implemented through model/provider concept. If we could work through LiteLLM it would be much easier. Not everyone works with Anthropic/Google or OpenAI through API keys. E.g. I need to authenticate to a corporate LLM layer with OAuth2. With e.g. LiteLLM I can get this covered.

[samuelcolvin]

How does litellm help with this?

[skye0402]

How does litellm help with this?

In the same way like it helps for crew.ai - they first started with langchain as layer to provide LLM access and then changed the approach by summer 2024 to litellm for all that's not bread and butter.

[samuelcolvin]

I've heard a lot of bad things about LiteLLM, is much rather avoid them

[Jacse]

I've heard a lot of bad things about LiteLLM, is much rather avoid them

This doesn't feel like a very constructive argument for favoring any specific proxy provider.

While I think supporting multiple proxy providers would be ideal, I understand if you prefer to pick a single interface to support. When picking this interface, however, I'd argue towards open source.

For our use case (which is probably not super common although also not unique) we cannot use cloudflare's gateways and have to either self-host a proxy or do load balancing and fallbacks in code. LiteLLMs router works relatively well for load balancing and fallbacks (better than the fallbacks currently in PydanticAI), but the integration of the two is very unergonomic right now.

For us, the most helpful solution is to either expand PydanticAI's load balancing and fallback mechanisms so we can tear out LiteLLM or support LiteLLM directly in PydanticAI.

Happy to expand on any of this.