Gemini native-audio models require AUDIO modality - TEXT-only mode not supported

[jiangzhuo]

Summary

Gemini native-audio models (e.g., gemini-2.5-flash-native-audio-latest) do not support responseModalities: [TEXT] when receiving audio input. This affects the participant client feature which uses textOnly: true mode.

Error

When attempting to connect with responseModalities: [Modality.TEXT]:

{
  "type": "session.closed",
  "data": {
    "code": 1007,
    "reason": "Cannot extract voices from a non-audio request.",
    "wasClean": true
  }
}

Root Cause

Native-audio models are designed for bidirectional audio (audio in → audio out) and require responseModalities: [AUDIO]. Unlike non-native models (e.g., gemini-live-2.5-flash-preview), they cannot operate in text-only output mode.

Workaround Implemented

In GeminiClient.ts, we now:

1. Always use responseModalities: [Modality.AUDIO] regardless of textOnly flag
2. Track textOnlyMode flag to skip sending audio deltas when in text-only mode
3. Rely on inputAudioTranscription and outputAudioTranscription for text output

This allows participant clients to work with Gemini native-audio models while showing only transcription text (no audio playback).

Related GitHub Issues

The following issues in googleapis/js-genai report similar problems:

• #1212 - "gemini-2.5-flash-native-audio Live API: Transcription Fails with 'Invalid Argument' (1007)"

  • Reports that Modality.TEXT in responseModalities causes immediate 1007 close
  • Same error: "Request contains an invalid argument"

• #1189 - "Unexpected websocket close code 1007 after switching to gemini-2.5-flash-native-audio-preview-12-2025"

  • Reports websocket close after sending first audio chunk with native-audio model

Expected Behavior (Feature Request for Google)

Native-audio models should support responseModalities: [TEXT] with audio input for use cases like:

• Speech-to-text only (transcription without TTS response)
• Participant translation in video conferencing (text display only)
• Low-bandwidth scenarios where audio output is not needed

Current Status

• Workaround: ✅ Implemented (use AUDIO modality, skip audio delta in textOnly mode)
• Upstream fix: ⏳ Pending Google's update to support TEXT modality with native-audio models

References

• Gemini Live API Documentation
• googleapis/js-genai Issues