TTS fixes and latency optimization

[Pento95]

This pull request significantly enhances the Text-to-Speech (TTS) service by introducing a streaming playback architecture. The primary goal is to reduce the perceived latency of TTS audio by allowing playback to start as soon as the initial audio segments are ready, rather than waiting for the entire text to be processed. The changes also centralize text chunking logic, improve audio resource management for better stability, and standardize playback speed control across different TTS providers.
Highlights

• Streaming Playback for TTS: Implemented a new streamAndPlay method in TTSProvider that allows audio playback to begin with the first generated chunk while subsequent chunks are processed in the background. This significantly reduces perceived latency for longer texts.
• Centralized Text Chunking: Refactored the TTS generation logic to centralize text splitting into chunks within the base TTSProvider class. Providers now define their maxChunkLength, and the generateSpeech method handles chunking, calling a new generateChunk method for individual chunk generation.
• Improved Audio Resource Management: Enhanced the audio element cleanup process in playSingleBlob and stopAudio to explicitly pause, remove the src attribute, and load the audio element. This ensures proper release of PulseAudio/PipeWire streams, addressing potential GStreamer crashes on Linux.
• Consistent Playback Speed Control: Standardized client-side control of playback speed for all TTS providers. The speed parameter is now consistently applied via playbackRate on the HTMLAudioElement, and removed from the server-side request for OpenAI-compatible providers.
• Robust Chunk Generation with Retries: Added a retry mechanism (up to 2 retries) for individual TTS chunk generation within the streamAndPlay method, improving robustness against transient API failures during the streaming process.

[gemini-code-assist]

Summary of Changes

Hello @Pento95, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the Text-to-Speech (TTS) service by introducing a streaming playback architecture. The primary goal is to reduce the perceived latency of TTS audio by allowing playback to start as soon as the initial audio segments are ready, rather than waiting for the entire text to be processed. The changes also centralize text chunking logic, improve audio resource management for better stability, and standardize playback speed control across different TTS providers.

Highlights

• Streaming Playback for TTS: Implemented a new streamAndPlay method in TTSProvider that allows audio playback to begin with the first generated chunk while subsequent chunks are processed in the background. This significantly reduces perceived latency for longer texts.
• Centralized Text Chunking: Refactored the TTS generation logic to centralize text splitting into chunks within the base TTSProvider class. Providers now define their maxChunkLength, and the generateSpeech method handles chunking, calling a new generateChunk method for individual chunk generation.
• Improved Audio Resource Management: Enhanced the audio element cleanup process in playSingleBlob and stopAudio to explicitly pause, remove the src attribute, and load the audio element. This ensures proper release of PulseAudio/PipeWire streams, addressing potential GStreamer crashes on Linux.
• Consistent Playback Speed Control: Standardized client-side control of playback speed for all TTS providers. The speed parameter is now consistently applied via playbackRate on the HTMLAudioElement, and removed from the server-side request for OpenAI-compatible providers.
• Robust Chunk Generation with Retries: Added a retry mechanism (up to 2 retries) for individual TTS chunk generation within the streamAndPlay method, improving robustness against transient API failures during the streaming process.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a significant improvement to the Text-to-Speech service by implementing streaming playback. This should greatly reduce the perceived latency for users, as audio can start playing as soon as the first chunk is ready. The refactoring of TTSProvider to support chunk-based generation is well-done, with common logic moved to the base class. The addition of a retry mechanism for chunk generation adds robustness.

I've identified a couple of areas for improvement in the new streamAndPlay method: one regarding a small logical flaw (dead code) and another suggesting a more efficient alternative to polling for new audio chunks. Overall, this is a great enhancement to the TTS functionality.