G2F - Multimodal Live - TEXT returned in foreign language

[OldManMeta]

• Programming language: Python, C++, JS
• OS: Windows 11

Version 0.7.0

I have a client app that is sending the Setup to the python websocket server

setup_client_message = {
                setup: {
                    generation_config: { response_modalities: ['TEXT'] },
                },
            };

Now I am an English speaker, and I just happen to be in Thailand at the moment.
My client application is not a webbrowser - so I am assuming that language locale is normally being set by the browser?

When I send an audio message in, gemini responds with Thai language

receiving from gemini
response: setup_complete=None server_content=LiveServerContent(model_turn=Content(parts=[Part(video_metadata=None, thought=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=None, inline_data=None, text='อาจ')], role=None), turn_complete=None, interrupted=None) tool_call=None tool_call_cancellation=None
part: video_metadata=None thought=None code_execution_result=None executable_code=None file_data=None function_call=None function_response=None inline_data=None text='อาจ'
response: setup_complete=None server_content=LiveServerContent(model_turn=Content(parts=[Part(video_metadata=None, thought=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=None, inline_data=None, text='ารย์บอกว่าทำไมคุณถึงพูดภาษาไทยกับฉัน? ฉันไม่')], role=None), turn_complete=None, interrupted=N        None) tool_call=None tool_call_cancellation=None
part: video_metadata=None thought=None code_execution_result=None executable_code=None file_data=None function_call=None function_response=None inline_data=None text='ารย์บอกว่าทำไมคุณถึงพูดภาษาไทยกับฉัน? ฉันไม่'
response: setup_complete=None server_content=LiveServerContent(model_turn=Content(parts=[Part(video_metadata=None, thought=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None, function_response=None, inline_data=None, text='เข้าใจ')], role=None), turn_complete=None, interrupted=None) tool_call=None tool_call_cancellaation=None
part: video_metadata=None thought=None code_execution_result=None executable_code=None file_data=None function_call=None function_response=None inline_data=None text='เข้าใจ'
response: setup_complete=None server_content=LiveServerContent(model_turn=None, turn_complete=True, interrupted=None) tool_call=None tool_call_cancellation=None

<Turn complete>

I have tried to set it in the generation_config to no avail.

How can I force it to return a response in only English text?

[sasha-gitg]

Are you using the Gemini Developer API or Vertex AI? Please provide a code sample to reproduce.

[OldManMeta]

Hi @sasha-gitg - thanks for the response.

This is the GDA.

Python package:

from google import genai

this is my server settings

os.environ['GOOGLE_API_KEY'] = 'my-key>>>>>>>>'
MODEL = "gemini-2.0-flash-exp"

client = genai.Client(
  http_options={
    'api_version': 'v1alpha'
  },
  location='us-central1'
)

A couple of things to note since creating this issue:

• The request for a code repro may not work if you are sat in a default English speaking country

• my initial assumption was that this was to do with the auto locale detection Google applies to everything (which really annoys people BTW)

• My intuition here tells me that it occurs if the input audio quality is extremely low and the model is attempting to decern the language, resulting in an output in whatever it believes is the input language

• I have since been able to generate responses with Korean and Chinese characters in the text field

• Perhaps a language force property will allow the model to respond appropriately i.e. your audio quality is too low, as opposed to responding inappropriately?

[OldManMeta]

Just as an update to this, I was able to modify my audio input to heavily reduced quality (created timing gaps in the samples, reduced gain, unintentional static/noise) which resulted in the model returning foreign languages on most occasions inside the text field.

When providing a strong audio quality stream (defined here as no timing gaps, average gain, no static) the model did return English every time.

I don't know if that helps?

I would like to see a property on the config that allows us to force a response language.

I see the location='us-central1' property is actually only supported in Vertex and I misunderstood its usage when I added it.

[sasha-gitg]

Thanks for the update. You can also try providing system instructions to interpret the audio in a specific language and/or respond in a specific language.