Feature Request: Realtime/Live API integration

[shun-liang]

Description

Both OpenAI and Gemini have provided realtime APIs for realtime voice/video interaction with their models.

Meanwhile, there are a couple of open-source or proprietary platforms for building realtime agents like

• LiveKit
• Daily
• ElevenLabs

It would be great if we can have a way to build realtime agent with Pydantic AI with minimal proprietary dependency (no more than just the model providers), allowing full onwership of the agent and tool definitions with Pydantic taste.

References

OpenAI realtime API:

• https://platform.openai.com/docs/guides/realtime

Gemini Live API

• https://platform.openai.com/docs/guides/realtime
• https://cloud.google.com/vertex-ai/generative-ai/docs/live-api

[aristideubertas]

Please add Nova Sonic to the list

(I had opened an older ticket here)