BookTunes

An AI-powered system that generates emotion-aligned soundtracks for books by analyzing text on a page-by-page basis and intelligently blending music from a curated library.

Team

Mia George, Aryaan Peshoton, Srija Kethireddy

Overview

BookTunes creates an immersive reading experience by matching the emotional content of literary text with dynamically generated background music. The system uses modern language models and deep learning techniques to extract emotional representations from text and creates seamless audio blends through sophisticated music retrieval and mixing.

The system includes an interactive UI for easy experimentation and real-time soundtrack generation.

Emotion Categories

The system works with nine emotion categories mapped to both text and music:

• Amazement: Wonder, awe, astonishment, breathtaking discovery, miraculous revelation, epic grandeur
• Solemnity: Reverence, dignity, sacred ceremony, spiritual depth, formal gravity, serious contemplation
• Tenderness: Warmth, gentleness, intimate affection, caring love, emotional closeness, soft vulnerability
• Nostalgia: Longing for the past, bittersweet memories, wistful reminiscence, yearning for old times
• Calmness: Peace, tranquility, serenity, stillness, relaxation, quietness, meditation
• Power: Strength, energy, boldness, heroism, triumph, victory, confidence, determination
• Joyful Activation: Joy, happiness, excitement, playfulness, celebration, cheerfulness, enthusiasm
• Tension: Stress, anxiety, nervousness, unease, suspense, fear, restlessness, apprehension
• Sadness: Sorrow, grief, melancholy, heartbreak, loss, despair, loneliness, emotional pain

Technical Architecture

1. Deep CNN Music Encoder

• Multi-modal feature extraction:
  • Mel spectrogram processing (128 mel bands, 20s segments)
  • Audio features: 20 MFCCs, 12 chroma features, 7 spectral contrast bands, 3 spectral statistics
• Architecture:
  • 4-layer CNN with batch normalization and progressive pooling (64→128→256→512 channels)
  • Parallel audio feature encoder (42D → 128D)
  • Fusion network producing 256D normalized embeddings
• Training strategy:
  • Smart contrastive loss focusing on top-2 emotions
  • Opposite emotion penalty (e.g., joy vs. sadness)
  • Data augmentation: time/frequency masking, random gain, noise injection

2. Transformer-Based Text-to-Emotion Mapping

• Model: Sentence-BERT (DeBERTa-v3-small with MiniLM-L6-v2 fallback)
• Method:
  • Encodes text and emotion descriptions into shared embedding space
  • Cosine similarity matching with configurable strategies:
    • l2_top2: Keeps top-2 emotions, L2 normalized (default)
    • softmax: Full probability distribution over all emotions
• Emotion descriptions: Rich multi-synonym representations for robust matching

3. FAISS-Powered Retrieval

• Index: Normalized inner product search on 256D embeddings
• Smart filtering:
  • Language-based data filtering (uses most common language in ratings)
  • Emotion threshold (40%) to identify strong emotional signals
  • Tracks analyzed by top-2 dominant emotions
• Efficiency: Sub-millisecond retrieval over 400 tracks

4. Intelligent Audio Blending Engine

• Harmonic-Percussive Source Separation (HPSS):
  • Separates melody from rhythm using librosa
  • Independent mixing control (85% harmonic, 15% percussive)
• Loudness normalization:
  • LUFS-based loudness metering (target: -16 LUFS)
  • Consistent perceived volume across all tracks
• Weighted mixing:
  • Combines multiple tracks based on emotion similarity scores
  • Applies dynamic envelope for natural fade-in/fade-out

Datasets

Music Dataset

Emotify Dataset from Utrecht University

• 400 emotion-annotated music tracks
• Genres: pop, classical, rock, electronic, ambient
• Multi-rater human annotations across 9 emotion dimensions
• Pre-processed features cached for efficient training

Text Dataset

Project Gutenberg

• Public domain literary works for testing
• Diverse genres and emotional content
• Page-level text segmentation for dynamic soundtrack generation

Key Features

Smart Data Preprocessing

• Automatic language filtering for consistent emotion ratings
• Emotion thresholding to identify dominant feelings
• Genre-aware track organization

Advanced Training Pipeline

• 85/15 train-validation split
• AdamW optimizer with learning rate scheduling
• Gradient clipping for training stability
• Best model checkpointing based on validation loss

Real-time Emotion Analysis

• Processes text on-demand using transformer models
• Configurable emotion distribution strategies
• Fast top-K emotion extraction for UI display

Professional Audio Quality

• HPSS-based intelligent mixing
• Psychoacoustic loudness normalization
• Smooth crossfades and transitions

Interactive UI

• Real-time text input and emotion visualization
• Live audio playback of generated soundtracks
• Emotion distribution charts and track information
• Easy export of generated audio files

Requirements

• Python 3.8+
• CUDA-enabled GPU (recommended for training)
• PyTorch 2.0+
• FAISS (CPU or GPU version)
• Sentence-Transformers
• librosa 0.10+
• scikit-learn
• pyloudnorm
• soundfile
• ipywidgets (for interactive UI)

Installation

git clone https://github.com/mia-george/BookTunes.git
cd BookTunes
pip install -q torch librosa scikit-learn faiss-cpu soundfile pyloudnorm ipywidgets tqdm sentence-transformers

Usage

Training the Model

# Load and preprocess data
db = load_data_smart()

# Cache audio features
cache_features(db)

# Train model
model, history = train()

Extracting Music Embeddings

# Extract embeddings and build search index
embeddings, track_ids, index = extract_embeddings(model)

# Visualize embedding space
visualize(embeddings, track_ids, history)

Generating Soundtracks from Text (Programmatic)

# Convert text to emotion vector
text = "The hero stood atop the mountain, filled with awe at the vast landscape below."
emotion_vec = text_to_emotion(text, method='l2_top2')

# Get top emotions
top_emotions = get_top_emotions_from_text(text, topk=3)
print(f"Detected emotions: {top_emotions}")

# Retrieve similar music tracks (use FAISS index with emotion_vec)
# Blend and generate soundtrack

Using the Interactive UI

The notebook includes an interactive widget-based interface that allows you to:

• Enter text passages directly
• View detected emotions in real-time
• Play generated soundtracks instantly
• Adjust blending parameters
• Export audio files

Simply run all cells in the notebook and use the UI widgets to experiment with different text inputs and soundtrack generations.

Model Architecture Details

CNN Encoder

Input: (batch, 1, 128, time_steps)
Conv1: 64 filters, 7×7 → BatchNorm → ReLU → MaxPool
Conv2: 128 filters, 5×5 → BatchNorm → ReLU → MaxPool
Conv3: 256 filters, 3×3 → BatchNorm → ReLU → MaxPool
Conv4: 512 filters, 3×3 → BatchNorm → ReLU → AdaptiveAvgPool
Output: (batch, 512, 1, 1) → Flatten to 512D

Audio Feature Encoder

Input: (batch, 42)  # 20 MFCCs + 12 chroma + 7 contrast + 3 spectral
FC1: 42 → 128 → ReLU → Dropout(0.2)
FC2: 128 → 128 → ReLU
Output: (batch, 128)

Fusion Network

Concat: [512D mel features, 128D audio features] → 640D
FC1: 640 → 512 → ReLU → Dropout(0.3)
FC2: 512 → 256 → ReLU → L2 Normalize
Output: (batch, 256) normalized embeddings