Skip to content

3.0 Audio Module (STT & TTS)

Jump to bottom
Lumi Bunny edited this page Aug 12, 2025 · 2 revisions

Audio Module Documentation

Overview

The audio module handles all speech processing capabilities in BnuuyBotApp, providing both Text-to-Speech (TTS) and Speech-to-Text (STT) functionality. This module enables natural voice conversations with the AI assistant through advanced audio processing and real-time transcription.

Module Structure

audio/
├── __init__.py          # Module exports (TTSEngine, SpeechToText)
├── tts_module.py        # Text-to-speech engine with queue management
└── stt_module.py        # Speech-to-text with VAD and streaming

Core Classes

TTSEngine Class

Advanced text-to-speech engine with queue management and real-time audio processing.

Initialization

from audio import TTSEngine

tts = TTSEngine(
    voice="en-US-AnaNeural",
    speed=1.15,
    api_url="http://localhost:5050/v1/audio/speech"
)

Key Features:

  • Queue-Based Processing - Handles multiple TTS requests efficiently
  • Real-Time Audio - Immediate audio playback with pygame integration
  • Voice Customization - Configurable voice selection and speech speed
  • Thread-Safe Operations - Concurrent audio processing and playback
  • Docker TTS Server - Integrates with containerized TTS service

Core Methods:

start()

  • Initializes TTS engine and starts audio processing queue
  • Begins background thread for audio playback management
  • Sets up pygame mixer for audio output

stop()

  • Gracefully shuts down TTS engine
  • Stops all background threads and clears audio queue
  • Cleans up pygame resources

speak(text)

  • Adds text to TTS queue for speech synthesis
  • Handles text preprocessing and chunking
  • Returns immediately while processing in background

speak_immediately(text)

  • Bypasses queue for urgent/priority speech
  • Immediate synthesis and playback
  • Useful for system messages and alerts

is_currently_speaking()

  • Returns current speech status
  • Thread-safe status checking
  • Used for conversation flow coordination

Audio Processing:

  • Text Chunking - Splits long text into manageable segments
  • Audio Format - MP3 format with configurable quality
  • Playback Management - Queue-based audio playback with pygame
  • Volume Control - Adjustable audio levels and normalization

Queue Management:

# Queue-based TTS processing
tts.speak("Hello there!")  # Queued
tts.speak("How are you?")  # Queued after first

# Immediate speech (bypasses queue)
tts.speak_immediately("Emergency message!")

SpeechToText Class

Advanced speech recognition with Voice Activity Detection (VAD) and streaming transcription.

Initialization

from audio import SpeechToText

stt = SpeechToText(
    model_size="small",
    device="cuda",
    compute_type="float16",
    streaming_interval=0.3,
    vad_aggressiveness=2,
    silence_threshold=1.0,
    language="en"
)

Key Features:

  • Faster-Whisper Integration - High-performance speech recognition
  • Voice Activity Detection - Intelligent speech/silence detection
  • Streaming Transcription - Real-time speech-to-text processing
  • GPU Acceleration - CUDA support for faster processing
  • Noise Filtering - Advanced audio preprocessing and cleanup

Core Methods:

start_listening()

  • Begins continuous audio capture from microphone
  • Starts VAD processing and speech detection
  • Initializes real-time transcription pipeline

stop_listening()

  • Stops audio capture and processing
  • Finalizes any pending transcriptions
  • Cleans up audio resources

get_transcription()

  • Returns latest transcription result
  • Includes confidence scores and timing
  • Thread-safe transcription retrieval

is_listening()

  • Returns current listening status
  • Used for UI state management
  • Thread-safe status checking

Voice Activity Detection:

  • WebRTC VAD - Industry-standard voice activity detection
  • Configurable Aggressiveness - Adjustable sensitivity (0-3)
  • Silence Detection - Intelligent end-of-speech detection
  • Frame-Based Processing - Efficient 30ms frame analysis

Audio Processing Pipeline:

  1. Audio Capture → Microphone input via sounddevice
  2. VAD Processing → Voice activity detection and filtering
  3. Speech Buffering → Accumulate speech frames until silence
  4. Whisper Processing → faster-whisper transcription
  5. Result Delivery → Return transcribed text with confidence

Advanced Features

Real-Time Audio Processing

TTS Audio Queue System:

# The TTS engine uses a sophisticated queue system
tts.start()

# Multiple messages are queued and processed in order
tts.speak("First message")
tts.speak("Second message") 
tts.speak("Third message")

# Check if currently speaking
if tts.is_currently_speaking():
    print("Audio is playing...")

STT Streaming Transcription:

# Real-time speech recognition with streaming
stt.start_listening()

while stt.is_listening():
    transcription = stt.get_transcription()
    if transcription:
        print(f"User said: {transcription}")
        
stt.stop_listening()

Integration with Chat Module

Conversation Flow Coordination:

  • TTS Completion Events - Notifies ConversationPrompter when speech finishes
  • STT Transcription Events - Triggers chat processing when speech detected
  • State Synchronization - Coordinates speaking/listening states

Audio-Chat Bridge:

from chat import BunnyChat, ConversationPrompter
from audio import TTSEngine, SpeechToText

# Initialize systems
bunny = BunnyChat()
tts = TTSEngine()
stt = SpeechToText()

# Set up conversation prompting with TTS coordination
prompter = ConversationPrompter(bunny, tts_engine=tts)

# Voice conversation loop
stt.start_listening()
while True:
    transcription = stt.get_transcription()
    if transcription:
        response = bunny.get_response(transcription)
        tts.speak(response)

Technical Implementation

Threading Architecture

TTS Threading:

  • Audio Queue Thread - Processes TTS requests in background
  • Playback Monitor Thread - Monitors pygame audio events
  • Thread Safety - Queue-based communication between threads

STT Threading:

  • Audio Capture Thread - Continuous microphone input processing
  • VAD Processing Thread - Real-time voice activity detection
  • Transcription Thread - Whisper model processing pipeline

Hardware Requirements

TTS Requirements:

  • Docker Support - TTS server runs in Docker container
  • Network Access - Local API communication (localhost:5050)
  • Audio Output - Sound card and speakers/headphones

STT Requirements:

  • GPU Recommended - NVIDIA GPU with CUDA for faster-whisper
  • Microphone - Quality microphone for clear audio input
  • Memory - Sufficient RAM for Whisper model loading

Performance Optimizations

TTS Optimizations:

  • Audio Caching - Reuse generated audio for repeated phrases
  • Chunk Processing - Split long text for faster synthesis
  • Queue Management - Efficient audio queue processing

STT Optimizations:

  • Model Caching - Keep Whisper model loaded in memory
  • VAD Filtering - Process only speech segments, ignore silence
  • GPU Acceleration - CUDA processing for faster transcription

Configuration Options

TTS Configuration

Voice Selection:

# Available voices (examples)
voices = [
    "en-US-AnaNeural",      # Female, clear
    "en-US-ChristopherNeural", # Male, professional
    "en-US-ElizabethNeural",   # Female, warm
    "en-US-EricNeural"         # Male, casual
]

tts = TTSEngine(voice="en-US-AnaNeural")

Speed and Quality:

tts = TTSEngine(
    voice="en-US-AnaNeural",
    speed=1.15,  # 15% faster than normal
    api_url="http://localhost:5050/v1/audio/speech"
)

STT Configuration

Model Selection:

# Model sizes (accuracy vs speed trade-off)
models = ["tiny", "base", "small", "medium", "large"]

stt = SpeechToText(
    model_size="small",      # Good balance of speed/accuracy
    device="cuda",           # GPU acceleration
    compute_type="float16"   # Memory optimization
)

VAD Sensitivity:

stt = SpeechToText(
    vad_aggressiveness=2,    # 0=least aggressive, 3=most aggressive
    silence_threshold=1.0,   # Seconds of silence before stopping
    streaming_interval=0.3   # Update frequency for real-time
)

Error Handling and Troubleshooting

Common TTS Issues

Docker TTS Server Not Running:

# Start TTS Docker container
docker run -p 5050:5050 tts-server:latest

Audio Playback Issues:

  • Check pygame initialization - Ensure audio drivers are available
  • Verify audio output - Test system audio settings
  • Monitor queue status - Check if audio queue is processing

Common STT Issues

CUDA/GPU Issues:

# Fallback to CPU if GPU unavailable
stt = SpeechToText(device="cpu", compute_type="int8")

Microphone Access:

  • Check permissions - Ensure microphone access granted
  • Test audio input - Verify microphone is working
  • Audio device selection - Configure correct input device

Model Loading Issues:

  • Internet connection - Required for initial model download
  • Storage space - Whisper models require several GB
  • Memory availability - Ensure sufficient RAM for model loading

Integration Examples

Flask Web App Integration

from flask import Flask, request, jsonify
from audio import TTSEngine, SpeechToText
from chat import BunnyChat

app = Flask(__name__)
bunny = BunnyChat()
tts = TTSEngine()
stt = SpeechToText()

@app.route('/chat', methods=['POST'])
def chat_endpoint():
    user_input = request.json.get('message')
    response = bunny.get_response(user_input)
    
    # Optional: Generate audio response
    tts.speak(response)
    
    return jsonify({'response': response})

@app.route('/voice_chat', methods=['POST'])
def voice_chat():
    # Handle voice input
    stt.start_listening()
    transcription = stt.get_transcription()
    
    if transcription:
        response = bunny.get_response(transcription)
        tts.speak(response)
        return jsonify({
            'transcription': transcription,
            'response': response
        })

Complete Voice Conversation Loop

from audio import TTSEngine, SpeechToText
from chat import BunnyChat, ConversationPrompter

# Initialize all systems
bunny = BunnyChat()
tts = TTSEngine()
stt = SpeechToText()
prompter = ConversationPrompter(bunny, tts_engine=tts)

# Start voice conversation
tts.start()
stt.start_listening()
prompter.start_timer()

print("Voice conversation started! Say something...")

try:
    while True:
        # Get user speech
        transcription = stt.get_transcription()
        
        if transcription:
            print(f"You: {transcription}")
            
            # Get AI response
            response = bunny.get_response(transcription)
            print(f"Bunny: {response}")
            
            # Speak response
            tts.speak(response)
            
        time.sleep(0.1)  # Small delay to prevent CPU overload
        
except KeyboardInterrupt:
    print("Ending conversation...")
    stt.stop_listening()
    tts.stop()
    prompter.stop_timer()

Dependencies and Requirements

TTS Dependencies

# Required packages
import requests      # API communication
import pygame        # Audio playback
import numpy as np   # Audio processing
import librosa       # Audio analysis
import sounddevice   # Audio device management
import mutagen       # Audio metadata

STT Dependencies

# Required packages
import faster_whisper  # Speech recognition
import webrtcvad      # Voice activity detection
import sounddevice    # Audio capture
import numpy as np    # Audio processing

Hardware Requirements

  • GPU: NVIDIA RTX 4060ti or better (for STT acceleration)
  • RAM: 8GB+ (for Whisper model loading)
  • Storage: 2GB+ for Whisper models
  • Audio: Quality microphone and audio output device

Performance Considerations

TTS Performance

  • Queue Processing - Background audio generation prevents blocking
  • Audio Caching - Potential for caching frequently used phrases
  • Network Latency - Local Docker server minimizes API delays

STT Performance

  • Model Size Trade-offs:
    • tiny - Fastest, lowest accuracy
    • small - Good balance (recommended)
    • medium/large - Highest accuracy, slower processing
  • GPU Acceleration - 5-10x faster processing with CUDA
  • VAD Efficiency - Processes only speech segments, ignores silence

Memory Usage

  • TTS: Minimal memory usage, mainly audio buffers
  • STT: Model-dependent (500MB-3GB depending on Whisper model size)
  • Audio Buffers: Small overhead for real-time processing

Troubleshooting Guide

TTS Troubleshooting

No Audio Output:

  1. Check Docker TTS server is running
  2. Verify pygame audio initialization
  3. Test system audio settings
  4. Check API URL and connectivity

Slow TTS Response:

  1. Monitor Docker container performance
  2. Check network latency to localhost
  3. Verify adequate system resources
  4. Consider audio queue optimization

STT Troubleshooting

Poor Recognition Accuracy:

  1. Upgrade to larger Whisper model
  2. Improve microphone quality/positioning
  3. Reduce background noise
  4. Adjust VAD aggressiveness settings

High CPU/GPU Usage:

  1. Use smaller Whisper model
  2. Reduce streaming interval
  3. Optimize VAD frame processing
  4. Monitor system resource usage

Microphone Issues:

  1. Check microphone permissions
  2. Verify audio device selection
  3. Test microphone with other applications
  4. Adjust audio input levels

Advanced Configuration

Custom TTS Voices

# Configure custom voice settings
tts = TTSEngine(
    voice="en-US-JennyNeural",  # Different voice
    speed=1.3,                  # Faster speech
    api_url="http://custom-tts-server:5050/v1/audio/speech"
)

STT Model Optimization

# Performance-optimized configuration
stt = SpeechToText(
    model_size="base",           # Faster than small
    device="cuda",               # GPU acceleration
    compute_type="float16",      # Memory optimization
    vad_aggressiveness=3,        # Aggressive noise filtering
    silence_threshold=0.8,       # Shorter silence detection
    streaming_interval=0.2       # More responsive updates
)

Audio Quality Settings

# High-quality audio processing
stt = SpeechToText(
    model_size="medium",         # Better accuracy
    vad_aggressiveness=1,        # Less aggressive filtering
    silence_threshold=1.5,       # Longer silence tolerance
    frame_duration_ms=30,        # Standard frame size
    padding_duration_ms=300      # Context padding
)

Integration Patterns

Chat Module Integration

The audio module works seamlessly with the chat module for voice conversations:

from chat import BunnyChat, ConversationPrompter
from audio import TTSEngine, SpeechToText

class VoiceChat:
    def __init__(self):
        self.bunny = BunnyChat()
        self.tts = TTSEngine()
        self.stt = SpeechToText()
        self.prompter = ConversationPrompter(
            self.bunny, 
            tts_engine=self.tts
        )
    
    def start_voice_conversation(self):
        self.tts.start()
        self.stt.start_listening()
        self.prompter.start_timer()
        
    def process_voice_input(self):
        transcription = self.stt.get_transcription()
        if transcription:
            response = self.bunny.get_response(transcription)
            self.tts.speak(response)
            return transcription, response
        return None, None

Flask Web Interface

@app.route('/toggle_listening', methods=['POST'])
def toggle_listening():
    if stt.is_listening():
        stt.stop_listening()
        status = "stopped"
    else:
        stt.start_listening()
        status = "listening"
    
    return jsonify({'status': status})

@app.route('/speak', methods=['POST'])
def speak_text():
    text = request.json.get('text')
    tts.speak(text)
    return jsonify({'status': 'speaking'})

Future Enhancements

Planned TTS Features

  • Voice Cloning - Custom voice training for personalized speech
  • Emotion Synthesis - Mood-aware speech generation
  • SSML Support - Advanced speech markup for better control
  • Audio Effects - Real-time audio processing and effects

Planned STT Features

  • Speaker Diarization - Multi-speaker conversation support
  • Language Detection - Automatic language identification
  • Custom Wake Words - Voice activation commands
  • Noise Cancellation - Advanced audio preprocessing

Performance Improvements

  • Model Quantization - Smaller, faster Whisper models
  • Streaming Optimization - Lower latency real-time processing
  • Audio Compression - Efficient audio data handling
  • Batch Processing - Optimized multi-request handling

Best Practices

TTS Best Practices

  1. Queue Management - Use speak() for normal text, speak_immediately() for urgent messages
  2. Text Preprocessing - Clean text before synthesis (remove special characters, normalize)
  3. Resource Management - Always call stop() when done to clean up resources
  4. Error Handling - Monitor TTS server status and handle connection issues

STT Best Practices

  1. Model Selection - Choose appropriate model size for your hardware
  2. VAD Tuning - Adjust aggressiveness based on environment noise
  3. Silence Threshold - Configure based on speaking patterns and use case
  4. Resource Monitoring - Watch GPU/CPU usage and optimize accordingly

Integration Best Practices

  1. State Coordination - Ensure TTS and STT don't conflict during conversations
  2. Error Recovery - Implement fallbacks for audio processing failures
  3. User Feedback - Provide clear indicators of listening/speaking status
  4. Performance Monitoring - Track audio processing latency and quality

The audio module provides the foundation for natural voice interactions in BnuuyBotApp, enabling seamless speech-based conversations with the AI assistant through advanced audio processing and intelligent coordination with the chat system.

Clone this wiki locally