Feature Request: Extend Fallback Provider to support OpenAI Realtime API

[bml1g12]

If the OpenAI Realtime API is unresponsive, it would be great to have a fallback mechanism to a conventional pipeline or another realtime API

i.e. it would be great to extend https://docs.livekit.io/agents/build/events/#fallbackadapter to work for realtime models

[asdkjasdfla]

Is there any update on this?

[nestormh]

Hello, is there any update on this topic?

Now that the latest gpt-realtime is out, it would be nice to have this functionality.

[davidzhao]

it's not straight forward to "fallback" for a realtime API.. the main challenge is realtime APIs keeping state on the server side.

how do you imagine this would work? If OpenAI is down.. would you want to fallback to Gemini?

[bml1g12]

The most likely use case I can see for this is if a realtime provider went down for a short time, having an out-of-the-box failover mechanism. This may already be possible manually as I understand it by developers looking for an error marked as "unrecoverable", and then in the on_error event handler adding code to e.g. re-instantiate the realtime session with a cooldown, or switch provider, but it would be a lot less complex and boilerplate if there was an out of the box pattern for that.

I could see a common pattern might be to e.g. switch from OpenAI Realtime provider to Azure OpenAI realtime provider and vice versa (when one fails or is rate limited etc.), or even to switch between different realtime models like Gemini (although I can see that would increase complexity).

Even if this is not implemented as an "adapter" class, I can imagine a reference example of best practice for handling failures the realtime provider session may be almost as useful and/or documentation on what kind of edge cases are already handled by existing realtime model implementations in LiveKit. For example, when it comes to realtime sessions what situations specifically we would expect an unrecoverable error vs recoverable error (e.g. if rate limited, or if the provider was down for 5 seconds, I am unclear what to expect and whether some kind of retry or failover to a new realtime session would already be handled). It does seem livekit already has a fair amount of code for robustness of the OpenAI realtime model, e.g. I think I have seen situations where it automatically restarts a broken session.

Aside from session failover, I think LiveKit persists in the chat context essentially an approximation of the "state" of the realtime models (an approximation, in that it contains no speech token embeddings and the text is often not necissarily the same as what the model received if its come through 3rd party transcription like Whisper). The instructions and user interactions would all be recorded there in the chat context, so in principle I think it would be possible to dump that and throw it into a non-realtime LLM,STT,TTS combo to continue the conversation - but I suspect that increases complexity of implementation a lot!

[kikoncuo]

it's not straight forward to "fallback" for a realtime API.. the main challenge is realtime APIs keeping state on the server side.

how do you imagine this would work? If OpenAI is down.. would you want to fallback to Gemini?

Most of the time crashes don't happen in the middle of a conversation, it's the service not being available from the start of the call. In those cases, the state is not relevant, plus I think I rather fall over to an agent that has no context (Or manage it with tools and pass the context) rather than a silent crash.

Ideally we'd find a way to fit it into the existing fallback functionality and it would fall to any agent, to a pipeline based one or another realtime

[bml1g12]

it's not straight forward to "fallback" for a realtime API.. the main challenge is realtime APIs keeping state on the server side.
how do you imagine this would work? If OpenAI is down.. would you want to fallback to Gemini?

Most of the time crashes don't happen in the middle of a conversation, it's the service not being available from the start of the call. In those cases, the state is not relevant, plus I think I rather fall over to an agent that has no context (Or manage it with tools and pass the context) rather than a silent crash.

Ideally we'd find a way to fit it into the existing fallback functionality and it would fall to any agent, to a pipeline based one or another realtime

I think then the key thing is what would be a reasonable choice of "trigger" for failover? Would it simply be an error emited marked as "recoverable=False"?

I can imagine a simple solution might be to provide a RealtimeFalloverAdapter that tracks unrecoverable errors, if they occur, it restarts session, and if that fails, it transfers to a different (non-stateful) cascaded model set

We can implement this ourselves, but I suspect all users of all realtime models have the same needs, so I think its a valid feature request