Skip to content

fix(google): support gemini-3.1-flash-live-preview realtime API for g…#5262

Closed
acharan-tech-200037 wants to merge 4 commits intolivekit:mainfrom
acharan-tech-200037:fix-gemini3.1_generatereply_issue
Closed

fix(google): support gemini-3.1-flash-live-preview realtime API for g…#5262
acharan-tech-200037 wants to merge 4 commits intolivekit:mainfrom
acharan-tech-200037:fix-gemini3.1_generatereply_issue

Conversation

@acharan-tech-200037
Copy link
Copy Markdown

@acharan-tech-200037 acharan-tech-200037 commented Mar 28, 2026

This pr resolves #5260

Adds support for gemini-3.1-flash-live-preview in the Google Realtime plugin.

Problem

The gemini-3.1-flash-live-preview model (released March 26, 2026) rejects all send_client_content WebSocket calls with a 1007 error (Request contains an invalid argument). This causes:

  • generate_reply() to timeout waiting for generation_created event
  • Initial chat context delivery to fail on connection
  • Mid-session updates via update_instructions() / update_chat_ctx() to fail

The model only accepts send_realtime_input for all content delivery.

How it was confirmed

A minimal test script was used to isolate the issue:

  • send_client_content → ❌ 1007 error (even with minimal config)
  • send_realtime_input with text → ✅ Works perfectly

Changes

File: livekit-plugins-google/livekit/plugins/google/realtime/realtime_api.py

  1. Added gemini-3.1-flash-live-preview to KNOWN_GEMINI_API_MODELS
  2. Added REALTIME_INPUT_ONLY_MODELS constant to detect models that reject send_client_content
  3. generate_reply() — Routes instructions as text via LiveClientRealtimeInput instead of LiveClientContent
  4. _main_task() — Converts initial chat context to send_realtime_input with [role]: text prefixes
  5. _send_task() — Catch-all: converts any LiveClientContent to send_realtime_input with role prefixes (covers update_instructions and update_chat_ctx flows)

All changes are backward-compatible — older models continue using send_client_content as before.

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Mar 28, 2026

CLA assistant check
All committers have signed the CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@acharan-tech-200037
Copy link
Copy Markdown
Author

@livekit/maintainers This PR is ready for review!

Summary: Adds support for Gemini 3.1 Flash Live Preview by switching from send_client_content to send_realtime_input for models that don't support the former.

Changes:

  • Added REALTIME_INPUT_ONLY_MODELS constant
  • Patched generate_reply, _send_task, and _main_task to use send_realtime_input
  • Fixed update_instructions to reconnect for these models
  • Included Python 3.9 compatibility fix for cambia plugin (required for CI)

Testing:

  • ✅ Gemini 3.1 works
  • ✅ Gemini 2.5 unaffected
  • ✅ All CI checks passed

Ready for review when you have time. Thanks!

@acharan-tech-200037
Copy link
Copy Markdown
Author

@theomonnom @tinalenguyen can you please review this pr.
This pr solves the issue #5260 along with failing on update_chat_ctx and update_instructions by gemini 3.1 flash live preview.

@acharan-tech-200037
Copy link
Copy Markdown
Author

Closing this in favor of #5251 by @Hormold which takes a cleaner
approach to the same problem.

Key differences:

  • Uses history_config.initial_history_in_client_content=True for
    proper initial seeding instead of converting turns to plain text
  • Drops mid-session LiveClientContent cleanly instead of converting
    to degraded text blobs
  • Adds empty server_content guard to prevent dropped audio on 3.1
  • Adds reconnect_on_update opt-in parameter for developer control

Tested and Hormold's approach is architecturally correct.
Recommend merging #5251.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

generate_reply times out on first dialogue with gemini-3.1-flash-live-preview — works fine with gemini-2.5-flash-

2 participants