3.0 Audio Module (STT & TTS)

Overview

The audio module handles all speech processing capabilities in BnuuyBotApp, providing both Text-to-Speech (TTS) and Speech-to-Text (STT) functionality. This module enables natural voice conversations with the AI assistant through advanced audio processing and real-time transcription.

Module Structure

audio/
├── __init__.py          # Module exports (TTSEngine, SpeechToText)
├── tts_module.py        # Text-to-speech engine with queue management
└── stt_module.py        # Speech-to-text with VAD and streaming

Core Classes

TTSEngine Class

Advanced text-to-speech engine with queue management and real-time audio processing.

Initialization

from audio import TTSEngine

tts = TTSEngine(
    voice="en-US-AnaNeural",
    speed=1.15,
    api_url="http://localhost:5050/v1/audio/speech"
)

Key Features:

Queue-Based Processing - Handles multiple TTS requests efficiently
Real-Time Audio - Immediate audio playback with pygame integration
Voice Customization - Configurable voice selection and speech speed
Thread-Safe Operations - Concurrent audio processing and playback
Docker TTS Server - Integrates with containerized TTS service

Core Methods:

start()

Initializes TTS engine and starts audio processing queue
Begins background thread for audio playback management
Sets up pygame mixer for audio output

stop()

Gracefully shuts down TTS engine
Stops all background threads and clears audio queue
Cleans up pygame resources

speak(text)

Adds text to TTS queue for speech synthesis
Handles text preprocessing and chunking
Returns immediately while processing in background

speak_immediately(text)

Bypasses queue for urgent/priority speech
Immediate synthesis and playback
Useful for system messages and alerts

is_currently_speaking()

Returns current speech status
Thread-safe status checking
Used for conversation flow coordination

Audio Processing:

Text Chunking - Splits long text into manageable segments
Audio Format - MP3 format with configurable quality
Playback Management - Queue-based audio playback with pygame
Volume Control - Adjustable audio levels and normalization

Queue Management:

# Queue-based TTS processing
tts.speak("Hello there!")  # Queued
tts.speak("How are you?")  # Queued after first

# Immediate speech (bypasses queue)
tts.speak_immediately("Emergency message!")

SpeechToText Class

Advanced speech recognition with Voice Activity Detection (VAD) and streaming transcription.

Initialization

from audio import SpeechToText

stt = SpeechToText(
    model_size="small",
    device="cuda",
    compute_type="float16",
    streaming_interval=0.3,
    vad_aggressiveness=2,
    silence_threshold=1.0,
    language="en"
)

Key Features:

Faster-Whisper Integration - High-performance speech recognition
Voice Activity Detection - Intelligent speech/silence detection
Streaming Transcription - Real-time speech-to-text processing
GPU Acceleration - CUDA support for faster processing
Noise Filtering - Advanced audio preprocessing and cleanup

Core Methods:

start_listening()

Begins continuous audio capture from microphone
Starts VAD processing and speech detection
Initializes real-time transcription pipeline

stop_listening()

Stops audio capture and processing
Finalizes any pending transcriptions
Cleans up audio resources

get_transcription()

Returns latest transcription result
Includes confidence scores and timing
Thread-safe transcription retrieval

is_listening()

Returns current listening status
Used for UI state management
Thread-safe status checking

Voice Activity Detection:

WebRTC VAD - Industry-standard voice activity detection
Configurable Aggressiveness - Adjustable sensitivity (0-3)
Silence Detection - Intelligent end-of-speech detection
Frame-Based Processing - Efficient 30ms frame analysis

Audio Processing Pipeline:

Audio Capture → Microphone input via sounddevice
VAD Processing → Voice activity detection and filtering
Speech Buffering → Accumulate speech frames until silence
Whisper Processing → faster-whisper transcription
Result Delivery → Return transcribed text with confidence

Advanced Features

Real-Time Audio Processing

TTS Audio Queue System:

# The TTS engine uses a sophisticated queue system
tts.start()

# Multiple messages are queued and processed in order
tts.speak("First message")
tts.speak("Second message") 
tts.speak("Third message")

# Check if currently speaking
if tts.is_currently_speaking():
    print("Audio is playing...")

STT Streaming Transcription:

# Real-time speech recognition with streaming
stt.start_listening()

while stt.is_listening():
    transcription = stt.get_transcription()
    if transcription:
        print(f"User said: {transcription}")
        
stt.stop_listening()

Integration with Chat Module

Conversation Flow Coordination:

TTS Completion Events - Notifies ConversationPrompter when speech finishes
STT Transcription Events - Triggers chat processing when speech detected
State Synchronization - Coordinates speaking/listening states

Audio-Chat Bridge:

from chat import BunnyChat, ConversationPrompter
from audio import TTSEngine, SpeechToText

# Initialize systems
bunny = BunnyChat()
tts = TTSEngine()
stt = SpeechToText()

# Set up conversation prompting with TTS coordination
prompter = ConversationPrompter(bunny, tts_engine=tts)

# Voice conversation loop
stt.start_listening()
while True:
    transcription = stt.get_transcription()
    if transcription:
        response = bunny.get_response(transcription)
        tts.speak(response)

Technical Implementation

Threading Architecture

TTS Threading:

Audio Queue Thread - Processes TTS requests in background
Playback Monitor Thread - Monitors pygame audio events
Thread Safety - Queue-based communication between threads

STT Threading:

Audio Capture Thread - Continuous microphone input processing
VAD Processing Thread - Real-time voice activity detection
Transcription Thread - Whisper model processing pipeline

Hardware Requirements

TTS Requirements:

Docker Support - TTS server runs in Docker container
Network Access - Local API communication (localhost:5050)
Audio Output - Sound card and speakers/headphones

STT Requirements:

GPU Recommended - NVIDIA GPU with CUDA for faster-whisper
Microphone - Quality microphone for clear audio input
Memory - Sufficient RAM for Whisper model loading

Performance Optimizations

TTS Optimizations:

Audio Caching - Reuse generated audio for repeated phrases
Chunk Processing - Split long text for faster synthesis
Queue Management - Efficient audio queue processing

STT Optimizations:

Model Caching - Keep Whisper model loaded in memory
VAD Filtering - Process only speech segments, ignore silence
GPU Acceleration - CUDA processing for faster transcription

Configuration Options

TTS Configuration

Voice Selection:

# Available voices (examples)
voices = [
    "en-US-AnaNeural",      # Female, clear
    "en-US-ChristopherNeural", # Male, professional
    "en-US-ElizabethNeural",   # Female, warm
    "en-US-EricNeural"         # Male, casual
]

tts = TTSEngine(voice="en-US-AnaNeural")

Speed and Quality:

tts = TTSEngine(
    voice="en-US-AnaNeural",
    speed=1.15,  # 15% faster than normal
    api_url="http://localhost:5050/v1/audio/speech"
)

STT Configuration

Model Selection:

# Model sizes (accuracy vs speed trade-off)
models = ["tiny", "base", "small", "medium", "large"]

stt = SpeechToText(
    model_size="small",      # Good balance of speed/accuracy
    device="cuda",           # GPU acceleration
    compute_type="float16"   # Memory optimization
)

VAD Sensitivity:

stt = SpeechToText(
    vad_aggressiveness=2,    # 0=least aggressive, 3=most aggressive
    silence_threshold=1.0,   # Seconds of silence before stopping
    streaming_interval=0.3   # Update frequency for real-time
)

Error Handling and Troubleshooting

Common TTS Issues

Docker TTS Server Not Running:

# Start TTS Docker container
docker run -p 5050:5050 tts-server:latest

Audio Playback Issues:

Check pygame initialization - Ensure audio drivers are available
Verify audio output - Test system audio settings
Monitor queue status - Check if audio queue is processing

Common STT Issues

CUDA/GPU Issues:

# Fallback to CPU if GPU unavailable
stt = SpeechToText(device="cpu", compute_type="int8")

Microphone Access:

Check permissions - Ensure microphone access granted
Test audio input - Verify microphone is working
Audio device selection - Configure correct input device

Model Loading Issues:

Internet connection - Required for initial model download
Storage space - Whisper models require several GB
Memory availability - Ensure sufficient RAM for model loading

Integration Examples

Flask Web App Integration

from flask import Flask, request, jsonify
from audio import TTSEngine, SpeechToText
from chat import BunnyChat

app = Flask(__name__)
bunny = BunnyChat()
tts = TTSEngine()
stt = SpeechToText()

@app.route('/chat', methods=['POST'])
def chat_endpoint():
    user_input = request.json.get('message')
    response = bunny.get_response(user_input)
    
    # Optional: Generate audio response
    tts.speak(response)
    
    return jsonify({'response': response})

@app.route('/voice_chat', methods=['POST'])
def voice_chat():
    # Handle voice input
    stt.start_listening()
    transcription = stt.get_transcription()
    
    if transcription:
        response = bunny.get_response(transcription)
        tts.speak(response)
        return jsonify({
            'transcription': transcription,
            'response': response
        })

Complete Voice Conversation Loop

from audio import TTSEngine, SpeechToText
from chat import BunnyChat, ConversationPrompter

# Initialize all systems
bunny = BunnyChat()
tts = TTSEngine()
stt = SpeechToText()
prompter = ConversationPrompter(bunny, tts_engine=tts)

# Start voice conversation
tts.start()
stt.start_listening()
prompter.start_timer()

print("Voice conversation started! Say something...")

try:
    while True:
        # Get user speech
        transcription = stt.get_transcription()
        
        if transcription:
            print(f"You: {transcription}")
            
            # Get AI response
            response = bunny.get_response(transcription)
            print(f"Bunny: {response}")
            
            # Speak response
            tts.speak(response)
            
        time.sleep(0.1)  # Small delay to prevent CPU overload
        
except KeyboardInterrupt:
    print("Ending conversation...")
    stt.stop_listening()
    tts.stop()
    prompter.stop_timer()

Dependencies and Requirements

TTS Dependencies

# Required packages
import requests      # API communication
import pygame        # Audio playback
import numpy as np   # Audio processing
import librosa       # Audio analysis
import sounddevice   # Audio device management
import mutagen       # Audio metadata

STT Dependencies

# Required packages
import faster_whisper  # Speech recognition
import webrtcvad      # Voice activity detection
import sounddevice    # Audio capture
import numpy as np    # Audio processing

Hardware Requirements

GPU: NVIDIA RTX 4060ti or better (for STT acceleration)
RAM: 8GB+ (for Whisper model loading)
Storage: 2GB+ for Whisper models
Audio: Quality microphone and audio output device

Performance Considerations

TTS Performance

Queue Processing - Background audio generation prevents blocking
Audio Caching - Potential for caching frequently used phrases
Network Latency - Local Docker server minimizes API delays

STT Performance

Model Size Trade-offs:
- tiny - Fastest, lowest accuracy
- small - Good balance (recommended)
- medium/large - Highest accuracy, slower processing
GPU Acceleration - 5-10x faster processing with CUDA
VAD Efficiency - Processes only speech segments, ignores silence

Memory Usage

TTS: Minimal memory usage, mainly audio buffers
STT: Model-dependent (500MB-3GB depending on Whisper model size)
Audio Buffers: Small overhead for real-time processing

Troubleshooting Guide

TTS Troubleshooting

No Audio Output:

Check Docker TTS server is running
Verify pygame audio initialization
Test system audio settings
Check API URL and connectivity

Slow TTS Response:

Monitor Docker container performance
Check network latency to localhost
Verify adequate system resources
Consider audio queue optimization

STT Troubleshooting

Poor Recognition Accuracy:

Upgrade to larger Whisper model
Improve microphone quality/positioning
Reduce background noise
Adjust VAD aggressiveness settings

High CPU/GPU Usage:

Use smaller Whisper model
Reduce streaming interval
Optimize VAD frame processing
Monitor system resource usage

Microphone Issues:

Check microphone permissions
Verify audio device selection
Test microphone with other applications
Adjust audio input levels

Advanced Configuration

Custom TTS Voices

# Configure custom voice settings
tts = TTSEngine(
    voice="en-US-JennyNeural",  # Different voice
    speed=1.3,                  # Faster speech
    api_url="http://custom-tts-server:5050/v1/audio/speech"
)

STT Model Optimization

# Performance-optimized configuration
stt = SpeechToText(
    model_size="base",           # Faster than small
    device="cuda",               # GPU acceleration
    compute_type="float16",      # Memory optimization
    vad_aggressiveness=3,        # Aggressive noise filtering
    silence_threshold=0.8,       # Shorter silence detection
    streaming_interval=0.2       # More responsive updates
)

Audio Quality Settings

# High-quality audio processing
stt = SpeechToText(
    model_size="medium",         # Better accuracy
    vad_aggressiveness=1,        # Less aggressive filtering
    silence_threshold=1.5,       # Longer silence tolerance
    frame_duration_ms=30,        # Standard frame size
    padding_duration_ms=300      # Context padding
)

Integration Patterns

Chat Module Integration

The audio module works seamlessly with the chat module for voice conversations:

from chat import BunnyChat, ConversationPrompter
from audio import TTSEngine, SpeechToText

class VoiceChat:
    def __init__(self):
        self.bunny = BunnyChat()
        self.tts = TTSEngine()
        self.stt = SpeechToText()
        self.prompter = ConversationPrompter(
            self.bunny, 
            tts_engine=self.tts
        )
    
    def start_voice_conversation(self):
        self.tts.start()
        self.stt.start_listening()
        self.prompter.start_timer()
        
    def process_voice_input(self):
        transcription = self.stt.get_transcription()
        if transcription:
            response = self.bunny.get_response(transcription)
            self.tts.speak(response)
            return transcription, response
        return None, None

Flask Web Interface

@app.route('/toggle_listening', methods=['POST'])
def toggle_listening():
    if stt.is_listening():
        stt.stop_listening()
        status = "stopped"
    else:
        stt.start_listening()
        status = "listening"
    
    return jsonify({'status': status})

@app.route('/speak', methods=['POST'])
def speak_text():
    text = request.json.get('text')
    tts.speak(text)
    return jsonify({'status': 'speaking'})

Future Enhancements

Planned TTS Features

Voice Cloning - Custom voice training for personalized speech
Emotion Synthesis - Mood-aware speech generation
SSML Support - Advanced speech markup for better control
Audio Effects - Real-time audio processing and effects

Planned STT Features

Speaker Diarization - Multi-speaker conversation support
Language Detection - Automatic language identification
Custom Wake Words - Voice activation commands
Noise Cancellation - Advanced audio preprocessing

Performance Improvements

Model Quantization - Smaller, faster Whisper models
Streaming Optimization - Lower latency real-time processing
Audio Compression - Efficient audio data handling
Batch Processing - Optimized multi-request handling

Best Practices

TTS Best Practices

Queue Management - Use speak() for normal text, speak_immediately() for urgent messages
Text Preprocessing - Clean text before synthesis (remove special characters, normalize)
Resource Management - Always call stop() when done to clean up resources
Error Handling - Monitor TTS server status and handle connection issues

STT Best Practices

Model Selection - Choose appropriate model size for your hardware
VAD Tuning - Adjust aggressiveness based on environment noise
Silence Threshold - Configure based on speaking patterns and use case
Resource Monitoring - Watch GPU/CPU usage and optimize accordingly

Integration Best Practices

State Coordination - Ensure TTS and STT don't conflict during conversations
Error Recovery - Implement fallbacks for audio processing failures
User Feedback - Provide clear indicators of listening/speaking status
Performance Monitoring - Track audio processing latency and quality

The audio module provides the foundation for natural voice interactions in BnuuyBotApp, enabling seamless speech-based conversations with the AI assistant through advanced audio processing and intelligent coordination with the chat system.