Live inputTranscription server messages are not sent

[jsalsman]

Environment details

• Programming language: JavaScript
• OS: all
• Package version: js-genai 0.9.0

Steps to reproduce

import { GoogleGenAI, Modality } from 'https://esm.run/@google/genai@0.9.0';
genAI = new GoogleGenAI({ apiKey: API_KEY });
liveSession = await genAI.live.connect({
    model: "gemini-2.0-flash-live-001",
    audioConfig: { targetSampleRate: 16000 },
    inputAudioTranscription: {}, // Enable audio transcription, not working
    config: {
        responseModalities: [Modality.TEXT],
    },
    callbacks: {
        onmessage: (message) => {
            if (message.inputTranscription) {
                    console.log("You said: " + message.inputTranscription);
...

This is a re-open of #454 for https://github.com/jsalsman/gemini-live

Thank you for your kind help!

[Annhiluc]

Thanks @jsalsman for raising this issue. Can you share context on what types of content are being sent to the liveSession?

[jsalsman]

@Annhiluc, most of the time just audio like this, although I have UI buttons to send images and text.

liveSession.sendRealtimeInput({
    media: {
        data: base64AudioData,
        mimeType: `audio/pcm;rate=${TARGET_SAMPLE_RATE}` // 16000
    }
});

See https://github.com/jsalsman/gemini-live/blob/14aaabd62930c8169170a10f93464b202311d754/gemini-live.html#L600 for more context.

[laishere]

+1 here.

Input audio transcription is useful for the chat applications. OpenAI realtime API supports this feature. But google live API does not. What's more, we would like to have VAD events about when the user speech start/end and a link between a transcript and the speech audio via something like itemId, which is like what the OpenAI offers.

Now we have to use an extra STT API to do the work.

Thanks!

[Wazbat]

I'm very confused. The typescript intellisense says this is supported, and the comments appear to document this feature properly, however after looking through the websocket api reference and other online docs, I cannot find any reference to input transcription anywhere.

https://ai.google.dev/api/live

Is this supposed to be supported? If not, is there a recommended way to detect when the user finished speaking? I'm planning to also use a STT API, but I'd like to accurately send it whatever gemini receives. My first thought was to wait for the first modelturn, but that might arrive late

[Wazbat]

After looking around, it seems like the vertexai version of gemini live supports it? At least, the docs say they do

However I'm unable to get live working with the vertexai version. The same code works fine normally, but after changing the constructor to use vertexAI and update to the appropriate model, I just quickly get a close event with code 1008 and the message

Publisher Model `projects/my-project-id/locations/us-east1/publishers/google/models/gemini-2.0-flash-live-preview-04-09`

and no further close event message text.
Looks like it's when sending it to the URL wss://us-east1-aiplatform.googleapis.com//ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent

I've also tried with apiVersion v1beta1 as I saw that in a few other issues, but that doesn't fix it

Ok!! So rubber ducking this worked out

So, it looks like input transcription only works with the vertexai version of the api, NOT with the gemini api key

However, you can't use the model name in the vertexai ui for gemini live, as that causes the above error. You need to keep using the same original model

So instead of gemini-2.0-flash-live-preview-04-09 you need to use gemini-2.0-flash-live-001

const genAI = new GoogleGenAI({
    vertexai: true,
    project: 'project-id-goes-here',
    location: 'us-east1',
    httpOptions: { }
});

// ...
const geminiSession = await genAI.live.connect({
    model: "gemini-2.0-flash-live-001",
    config: {
        responseModalities: [Modality.AUDIO],
        systemInstruction: SYSTEM_PROMPT,
        inputAudioTranscription: {},
        outputAudioTranscription: {}
    },
    callbacks: callbacks,
});

This works for me. I get both input transcription and output transcription. Though I hope the rest of it like tools work.

[laishere]

Yeah, we have to use vertex api.

[jsalsman]

My app at https://live.talknicer.com, asking the user to plug in their own API key, can't use the Vertex AI API because it can't run from just an API key; it needs a Cloud project.

[benbjurstrom]

This appears to be an issue with this package only. If you hardcode input_audio_transcription into the initial websocket setup message it works fine with the gemini api key.

In my local dev environment I just hardcoded it here for the time being:
node_modules/.pnpm/@google+genai@0.12.0/node_modules/@google/genai/dist/web/index.mjs

function liveConnectParametersToMldev(apiClient, fromObject) {
    const toObject = {};
    const fromModel = getValueByPath(fromObject, ['model']);
    if (fromModel != null) {
        setValueByPath(toObject, ['setup', 'model'], tModel(apiClient, fromModel));
    }
    const fromConfig = getValueByPath(fromObject, ['config']);
    if (fromConfig != null) {
        setValueByPath(toObject, ['config'], liveConnectConfigToMldev(apiClient, fromConfig, toObject));
    }

    // ADDED THESE LINES
    toObject.setup = {
        ...toObject.setup,
        input_audio_transcription: {}
    };

    return toObject;
}

[laishere]

@benbjurstrom I think they actually added this feature recently, I used to hardcode it via the live api web console demo which sends the config directly via the WebSocket, but the server will throw an error. Just tried it again today, it works now!

[laishere]

And I just see the latest changelog, it seems they added this feature 5 days ago! Without notifying us on this issue!
767c5d5

@jsalsman I think we can close this issue now that 0.13.0 already supports this feature.

[jsalsman]

Using import { GoogleGenAI, Modality } from 'https://esm.run/@google/genai@0.13.0'; I am still not getting input transcription messages. It looks like they fixed it for liveConnectConfigToMldev() and liveSendRealtimeInputParametersToVertex() (which was already working) but not the regular non-Vertex Gemini API key live access, which is not the same as Mldev.