Hi Google ADK Team,
Thanks for this wonderful ADK. I've some feature request for which I was not able to find any information on the Gemini Live APIs.
Is your feature request related to a problem? Please describe.
The current documentation and API offerings for the Gemini Live APIs (specifically for Speech-to-Text and Text-to-Speech) appear to lack detailed configuration options and granular control that are commonly available in other leading speech service providers (e.g., Azure, ElevenLabs). Developers building applications that require highly customized voice interactions, such as virtual assistants, educational tools, or accessibility features, face significant limitations.
The specific problems encountered include:
- Limited STT Configuration:
- Absence of explicit options for input language detection, which is crucial for multilingual applications.
- Lack of detailed control over the output transcript format (e.g., word-level timestamps, confidence scores, punctuation control).
- Limited TTS Configuration and Voice Options:
- Insufficient ability to fine-tune the output voice characteristics, such as tone of voice, speech rate (prosody), and explicit gender selection beyond a predefined set of voices.
- Currently, in the Google ADK and Gemini Live APIs, only a limited set of prebuilt voices are available for
prebuilt_voice_config, specifically: Puck, Charon, Kore, Fenrir, Aoede, Leda, Orus, and Zephyr.
- There is no readily apparent option or clear documentation for creating or using custom voices with the Gemini Live APIs. This is a significant limitation for brands or applications that require a unique, branded voice experience.
- Lack of programmatic control over emotional nuances or speaking styles.
- Insufficient Documentation: The existing documentation doesn't clearly outline if and how these advanced configurations can be achieved, leading to a trial-and-error approach for developers or a complete inability to implement desired features.
This lack of control hinders the creation of natural, highly personalized, and accessible voice experiences, forcing developers to compromise on quality or integrate third-party solutions for these specific functionalities, adding complexity and potential latency.
Describe the solution you'd like
We would like to request comprehensive documentation and, where currently lacking, the implementation of API parameters that allow for more granular control over both Speech-to-Text (STT) and Text-to-Speech (TTS) functionalities within the Gemini Live APIs.
Specifically, we propose:
- For Speech-to-Text (STT):
- Input Language Detection: Clear API parameters to enable and configure automatic input language detection for spoken audio.
- Output Transcript Configuration: Options to control the format and content of the output transcript, including:
- Word-level timestamps.
- Confidence scores for transcribed words/phrases.
- Detailed punctuation control.
- Speaker diarization (if multiple speakers are present).
- For Text-to-Speech (TTS):
- Voice Customization:
- Custom Voice Creation/Integration: Provide robust support and clear documentation for creating and integrating custom voices with the Gemini Live APIs, similar to Google Cloud Text-to-Speech's Custom Voice feature. This would allow developers to train a unique voice model using their own audio recordings and then use this voice within Gemini Live conversations.
- Expanded Prebuilt Voice Options: A larger and more diverse selection of prebuilt voices with varying tones, accents, and emotional ranges.
- Tone of Voice: Parameters to influence the emotional tone (e.g., happy, sad, formal, informal) of the synthesized speech.
- Speech Rate/Prosody: Granular control over speaking rate, pitch, and intonation beyond a basic "speed" setting.
- Gender Selection: Explicit options for selecting male, female, or gender-neutral voices, with a wider range of voice options.
- SSML Support (or equivalent): Robust support for Speech Synthesis Markup Language (SSML) to enable advanced control over speech output, including pauses, emphasis, pronunciation, and volume.
Along with these API enhancements, we request detailed, code-example-rich documentation demonstrating how to utilize these new configuration options, especially regarding custom voice creation and integration.
Describe alternatives you've considered
We have considered and, in some cases, currently utilize alternative speech service providers like Azure Speech Services and ElevenLabs. These platforms offer many of the granular STT and TTS configurations described above, including:
- Azure Speech Services: Provides extensive control over voices (neural voices with different styles), speech rate, pitch, SSML support, and robust STT capabilities with language detection and detailed transcript outputs (e.g., word-level timestamps, diarization). Azure also offers custom neural voice capabilities, allowing users to create unique voices.
- ElevenLabs: Known for its highly realistic voice synthesis, offering fine-tuning of voice tone, emotion, and prosody, along with robust voice cloning capabilities, enabling the creation of custom voices from limited audio samples.
While these alternatives provide the necessary functionality, integrating them alongside the Gemini Live API for the core conversational AI can lead to:
- Increased Latency: Additional network hops and processing between different services.
- Increased Complexity: Managing multiple API keys, authentication methods, and SDKs.
- Higher Costs: Separate billing and potential minimum usage fees for each service.
- Inconsistent Developer Experience: Juggling different documentation and development paradigms.
Our preference is to leverage the Google ADK and Gemini Live API as a unified solution, provided it can offer a comparable level of control for speech interactions, including the crucial ability to use and create custom voices.
Additional context
The ability to control the nuances of speech input and output, including having a unique and consistent brand voice, is paramount for building truly immersive and effective conversational AI experiences. For example:
- An educational application might need to dynamically adjust the speech rate for different learning levels or to emphasize specific words.
- A customer service bot might need to detect the user's input language automatically and respond in a specific, empathetic tone, or consistently use a branded custom voice.
- Accessibility features benefit greatly from detailed transcriptions (e.g., for deaf or hard-of-hearing users) and highly customizable voices for text-to-speech, potentially even a cloned voice for familiar interaction.
Providing these advanced configurations, especially the ability to create and use custom voices, within the Gemini Live APIs would significantly empower developers and expand the range of high-quality applications that can be built on the Google ADK platform.
Hi Google ADK Team,
Thanks for this wonderful ADK. I've some feature request for which I was not able to find any information on the Gemini Live APIs.
Is your feature request related to a problem? Please describe.
The current documentation and API offerings for the Gemini Live APIs (specifically for Speech-to-Text and Text-to-Speech) appear to lack detailed configuration options and granular control that are commonly available in other leading speech service providers (e.g., Azure, ElevenLabs). Developers building applications that require highly customized voice interactions, such as virtual assistants, educational tools, or accessibility features, face significant limitations.
The specific problems encountered include:
prebuilt_voice_config, specifically: Puck, Charon, Kore, Fenrir, Aoede, Leda, Orus, and Zephyr.This lack of control hinders the creation of natural, highly personalized, and accessible voice experiences, forcing developers to compromise on quality or integrate third-party solutions for these specific functionalities, adding complexity and potential latency.
Describe the solution you'd like
We would like to request comprehensive documentation and, where currently lacking, the implementation of API parameters that allow for more granular control over both Speech-to-Text (STT) and Text-to-Speech (TTS) functionalities within the Gemini Live APIs.
Specifically, we propose:
Along with these API enhancements, we request detailed, code-example-rich documentation demonstrating how to utilize these new configuration options, especially regarding custom voice creation and integration.
Describe alternatives you've considered
We have considered and, in some cases, currently utilize alternative speech service providers like Azure Speech Services and ElevenLabs. These platforms offer many of the granular STT and TTS configurations described above, including:
While these alternatives provide the necessary functionality, integrating them alongside the Gemini Live API for the core conversational AI can lead to:
Our preference is to leverage the Google ADK and Gemini Live API as a unified solution, provided it can offer a comparable level of control for speech interactions, including the crucial ability to use and create custom voices.
Additional context
The ability to control the nuances of speech input and output, including having a unique and consistent brand voice, is paramount for building truly immersive and effective conversational AI experiences. For example:
Providing these advanced configurations, especially the ability to create and use custom voices, within the Gemini Live APIs would significantly empower developers and expand the range of high-quality applications that can be built on the Google ADK platform.