🎵 PLAILIST - AI-Powered DJ Assistant

An intelligent DJ system that combines user preferences, live crowd analysis, and AI recommendations to create the perfect party playlist.

---

🌟 Overview

PLAILIST is a complete AI DJ system that:

• 📊 Analyzes group music preferences from Spotify data
• 🎤 Monitors live crowd reactions through audio analysis
• 🤖 Uses Google Gemini AI for intelligent song recommendations
• 🔄 Adapts dynamically to party energy and crowd feedback

System Architecture

┌─────────────────────────────────────────────────────────────┐
│                     INITIALIZATION PHASE                     │
│                                                              │
│  Fingerprint Server (Node.js)                               │
│  ├─ Analyzes Spotify user data                              │
│  ├─ Identifies top genres, artists, languages               │
│  └─ Generates group preference profile                      │
│                                                              │
│  Playlist Initializer (Python)                              │
│  ├─ Fetches fingerprint data                                │
│  ├─ Calculates musical diversity                            │
│  ├─ Determines DJ strategy                                  │
│  └─ Generates seed playlist (5 songs)                       │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│                    SEED PLAYLIST PHASE                       │
│                                                              │
│  Flask AI DJ Server (Python)                                │
│  ├─ Plays seed songs                                        │
│  ├─ Tracks crowd reactions                                  │
│  ├─ Builds enthusiasm score history                         │
│  └─ Monitors transition criteria                            │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│                   LIVE MONITORING PHASE                      │
│                                                              │
│  Live Audio Recorder (Python)                               │
│  ├─ Captures microphone input (5-second intervals)          │
│  ├─ Sends to audio classifier                               │
│  └─ Displays real-time results                              │
│                                                              │
│  Audio Classifier (HuggingFace + DSP)                       │
│  ├─ MIT AudioSet model (93% accuracy)                       │
│  ├─ Detects: cheering, applause, chatter, booing           │
│  ├─ Calculates enthusiasm score (-1.5 to +1.0)             │
│  └─ Tracks trends (rising/falling/stable)                   │
│                                                              │
│  Gemini AI Integration                                      │
│  ├─ Analyzes crowd state + party context                    │
│  ├─ Recommends next 3 songs with reasoning                  │
│  ├─ Predicts impact of each song                            │
│  └─ Provides warnings and tips                              │
└─────────────────────────────────────────────────────────────┘

---

🎬 2-Minute Demo

Perfect for hackathon presentations!

1. Start Flask server: python backend\app.py
2. Start workflow: python integrated_dj_workflow_v2.py
3. Open: http://127.0.0.1:5000/

What you'll see:

• 0:00-0:30 - Bar chatter (score: -0.3) → "Party starting..."
• 0:30-1:00 - Crowd cheering (score: +0.7) → Vibe meter surges!
• 1:00-1:30 - Sustained cheering → 🤖 Gemini adds 3 songs!
• 1:30-2:00 - Disappointment (score: -1.2) → Energy drops
• 2:00-2:30 - Recovery cheering → 🤖 Gemini adds 2 more songs!

Result: Party playlist grows from 5 → 8 → 10 songs as AI adapts to the crowd!

See QUICK_DEMO_GUIDE.md for detailed instructions.

---

🚀 Quick Start

Prerequisites

• Python 3.11+
• Node.js 16+
• Windows (for PyAudio) or Linux/Mac
• Google Gemini API key

Installation

1. Install Python dependencies

cd "D:\Academics\SUNY Buffalo\Semester 3\UB_Hacking"
pip install -r backend/requirements.txt

# Windows: Install PyAudio
pip install pipwin
pipwin install pyaudio

2. Install Node.js dependencies

cd PLAILIST-dj-attendee-fingerprint
npm install

3. Configure Gemini API Create .env file:

GEMINI_API_KEY=your_api_key_here
GEMINI_MODEL=gemini-2.0-flash-exp
GEMINI_TEMPERATURE=0.7

Running the System

For Demo (Simplified - 2 Terminals)

Terminal 1: Start Flask Server

python backend\app.py

✅ Backend: http://127.0.0.1:5000 ✅ Frontend: http://127.0.0.1:5000/ (opens automatically in browser)

Terminal 2: Run Demo Workflow

python integrated_dj_workflow_v2.py

This will:

• Reset party state
• Add 5 seed songs from Spotify fingerprints
• Analyze demo audio every 30 seconds
• Call Gemini AI every 90 seconds
• Display real-time vibe and playlist updates

Open in Browser:

http://127.0.0.1:5000/

Watch the vibe meter respond to audio analysis in real-time!

---

For Full System (4 Terminals)

Terminal 1: Start Fingerprint Server (Optional)

cd PLAILIST-dj-attendee-fingerprint
node fingerprint-server.js

✅ http://localhost:8000

Terminal 2: Start Flask AI DJ Server

python backend\app.py

✅ http://127.0.0.1:5000

Terminal 3: Run Integrated Workflow

python integrated_dj_workflow_v2.py

Terminal 4: Live Audio Monitoring (Optional)

python record_live_audio.py

---

📁 Project Structure

PLAILIST/
├── backend/
│   ├── app.py                          # Main Flask server (1292 lines)
│   └── requirements.txt                # Python dependencies
│
├── frontend/
│   ├── home.html                       # Main party session UI
│   └── static/
│       ├── vaibify_func.js             # Vibe meter & real-time updates
│       └── vaibify_style.css           # UI styling
│
├── PLAILIST-dj-attendee-fingerprint/
│   ├── fingerprint-server.js           # Node.js fingerprint server
│   ├── spotify_tracks_arin.csv         # User 1 data (7 tracks)
│   ├── spotify_tracks_atharva.csv      # User 2 data (315 tracks)
│   └── spotify_tracks_jayanth.csv      # User 3 data (439 tracks)
│
├── playlist_initializer.py             # Group analysis & seed generation
├── integrated_dj_workflow_v2.py        # Complete workflow orchestration
├── record_live_audio.py                # Live microphone capture
├── test_integrated_system.py           # Integration test suite
│
├── Demo Audio Files/
│   ├── people-talking-at-bar-72249.mp3            # Chatter (score: -0.3)
│   ├── crowd-cheer-and-applause-406644.mp3        # Cheering (score: +0.7)
│   └── crowd-disappointment-reaction-352718.mp3   # Disappointment (score: -1.2)
│
├── Documentation/
│   ├── ABOUT_THE_PROJECT.md            # Project story & learnings
│   ├── QUICK_DEMO_GUIDE.md             # 2-minute demo instructions
│   ├── DEMO_AUDIO_SEQUENCE.md          # Demo planning guide
│   ├── INTEGRATED_SYSTEM_GUIDE.md      # Complete system guide
│   ├── GEMINI_INTEGRATION_SPEC.md      # Technical specification
│   ├── GEMINI_API_REFERENCE.md         # API documentation
│   ├── LIVE_AUDIO_SETUP.md             # Audio recording setup
│   └── QUICKSTART.md                   # Basic setup guide
│
└── .env                                # Environment variables (not in git)

---

🎯 Key Features

1. User Fingerprinting

• Analyzes Spotify listening history for 3+ users
• Identifies top 10 genres with percentages
• Tracks favorite artists and play counts
• Detects languages (10+ languages supported)
• Calculates musical diversity score

2. Live Audio Analysis

• Primary: HuggingFace AudioSet model (MIT/ast-finetuned)
  • 93% accuracy on applause detection
  • 86% accuracy on booing detection
  • Real-time classification in <2 seconds
• Fallback: DSP analysis with Silero VAD
  • RMS energy + zero-crossing rate
  • Speech detection for chatter vs crowd noise
  • Works offline without model

3. Enthusiasm Scoring

Formula: score = cheering*1.0 + applause*0.7 - chatter*0.6 - booing*1.5

Score	Interpretation	Action
> 0.7	Highly engaged	Maintain energy
0.4 - 0.7	Good energy	Continue current vibe
0.1 - 0.4	Moderate	Consider boost
-0.2 - 0.1	Neutral	Change needed
-0.7 - -0.2	Losing interest	Genre shift
< -0.7	Negative reaction	Emergency change

4. Gemini AI Recommendations

Provides:

• Analysis: Crowd mood, energy trajectory, reasoning
• Action: Recommendation type (maintain/increase/wind_down/change), urgency, confidence
• Next Songs: 3 recommendations with:
  • Title, artist, genre
  • Reasoning for selection
  • Predicted impact on enthusiasm
  • Priority ranking
• Warnings: Potential issues to watch for
• Tips: DJ advice for better results

5. Party Stage Tracking

• Early: < 30 min or low energy (< 0.3)
• Mid: 30-120 min, moderate energy
• Peak: 30-120 min, high energy (> 0.6)
• Late: > 120 min or falling energy

---

🔧 API Endpoints

Flask AI DJ Server (http://127.0.0.1:5000)

Health Check

GET /health

Classify Audio

POST /classify-audio
Content-Type: multipart/form-data
Body: audio file (WAV/MP3)

Response:
{
  "probs": {
    "cheering": 0.037,
    "applause": 0.933,
    "chatter": 0.017,
    "booing": 0.013
  },
  "enthusiasm_score": 0.68,
  "trend": "rising",
  "method": "HuggingFace"
}

Get Gemini Recommendations

POST /gemini-recommend

Response:
{
  "context": { ... },
  "recommendation": {
    "analysis": { ... },
    "action": { ... },
    "next_songs": [ ... ],
    "warnings": [ ... ],
    "tips": [ ... ]
  }
}

Update Song

# Start tracking song
POST /update-song
{
  "song": {
    "title": "Song Name",
    "artist": "Artist Name",
    "genre": "Genre"
  }
}

# End current song
POST /update-song
{
  "action": "end_song"
}

Get Party State

GET /party-state

Response:
{
  "party_active": true,
  "elapsed_minutes": 45,
  "current_score": 0.68,
  "current_trend": "rising",
  "party_stage": "peak",
  "total_songs_played": 12,
  ...
}

Reset Party

POST /reset-party

Fingerprint Server (http://localhost:8000)

Get Fingerprints

GET /api/fingerprints

Response:
{
  "total_users": 3,
  "fingerprints": [
    {
      "user_id": "arin_asm",
      "user_name": "ASM",
      "total_tracks": 7,
      "top_genres": [ ... ],
      "top_artists": [ ... ],
      "top_languages": [ ... ],
      "average_popularity": 67.8
    }
  ]
}

---

🧪 Testing

Run Integration Tests

python test_integration.py

Tests:

• ✅ Fingerprint server connectivity
• ✅ Group preference analysis
• ✅ Seed playlist generation
• ✅ Gemini context building
• ✅ Transition logic
• ✅ Flask server integration

Manual Testing

Test Audio Classification:

curl -X POST http://127.0.0.1:5000/classify-audio \
  -F "audio=@test_audio/applause.wav"

Test Gemini Recommendations:

# First, classify some audio to build history
# Then:
curl -X POST http://127.0.0.1:5000/gemini-recommend

---

📊 Example Workflow

1. Initialization

from playlist_initializer import PlaylistInitializer

initializer = PlaylistInitializer()
initializer.fetch_fingerprints()
initializer.analyze_group_preferences()
initializer.print_summary()

# Output:
# 👥 Group Profile:
#    • Total attendees: 3
#    • Total tracks: 761
#    • Musical diversity: 0.164
#    • Top genre: pop (19.82%)

2. Seed Playlist

seeds = initializer.generate_seed_playlist(5)

# Output:
# 1. Tyler, The Creator - pop (19.82% appeal)
# 2. Arctic Monkeys - rock (15.23% appeal)
# 3. Kendrick Lamar - hip-hop (12.45% appeal)
# ...

3. Live Monitoring

python record_live_audio.py

# Output:
# 📊 ANALYSIS #1 - 18:45:23
# 🔥 ENTHUSIASM SCORE: 0.68 (GOOD)
#    Trend: rising
#    
# Crowd Reaction:
#    applause     ████████████████████  93.3%
#    cheering     ██  3.7%

4. AI Recommendations

recommendation = system.get_gemini_recommendations()

# Output:
# 🎯 GEMINI AI RECOMMENDATIONS
# 
# Analysis: High energy, engaged crowd
# Action: MAINTAIN_ENERGY (Urgency: NORMAL, 85% confidence)
# 
# Next Songs:
#   1. "Song X" by Artist Y (Electronic)
#      → Matches current energy, predicted +0.15 score boost

---

📈 Performance Metrics

Audio Classification

• Accuracy: 93% on applause, 86% on booing
• Speed: <2 seconds per 5-second audio clip
• Model Size: 346MB (cached locally)
• Memory: ~500MB RAM during inference

Gemini AI

• Response Time: 1-3 seconds
• Model: gemini-2.0-flash-exp (optimized for real-time)
• Token Limit: 2000 output tokens
• Temperature: 0.7 (balanced creativity)

System Requirements

• RAM: 2GB minimum, 4GB recommended
• Storage: 1GB for models + cache
• Network: Required for Gemini API (offline model available)

---

🔒 Security & Privacy

• ✅ API keys stored in .env (not committed to git)
• ✅ Audio files processed locally (not uploaded)
• ✅ User data stays on fingerprint server
• ✅ No personal information sent to Gemini
• ⚠️ Microphone access required for live monitoring

---

🐛 Troubleshooting

Common Issues

1. "Cannot connect to fingerprint server"

# Make sure it's running
cd PLAILIST-dj-attendee-fingerprint
node fingerprint-server.js

2. "PyAudio installation failed"

# Windows
pip install pipwin
pipwin install pyaudio

# Linux
sudo apt-get install portaudio19-dev
pip install pyaudio

# Mac
brew install portaudio
pip install pyaudio

3. "Gemini API not configured"

# Check .env file exists and has key
cat .env
# Should show: GEMINI_API_KEY=...

4. "Audio classification failed"

• Check audio file is WAV or MP3
• Verify file is not corrupted
• Try DSP fallback (automatic)

5. "Microphone not working"

# Test microphone
python record_live_audio.py
# Choose option 3 (Test Microphone)

---

🚧 Future Enhancements

Phase 1 (Current)

• ✅ User fingerprinting
• ✅ Audio classification
• ✅ Gemini integration
• ✅ Live monitoring

Phase 2 (In Progress)

• 🔄 React frontend for visualization
• 🔄 Spotify Web API integration
• 🔄 Real-time dashboard
• 🔄 Manual DJ overrides

Phase 3 (Planned)

• 📋 Multi-room support
• 📋 Playlist export/import
• 📋 Historical analytics
• 📋 Mobile app
• 📋 Cloud deployment

---

👥 Team

UB Hacking Project

• Arin (ASM) - Spotify Integration
• Atharva Prabhu - Audio Analysis
• Jayanth Shanmugam - AI Integration
• [Your Name] - System Architecture

---

📝 License

MIT License - See LICENSE file for details

---

🙏 Acknowledgments

• HuggingFace - MIT AudioSet model
• Google - Gemini AI API
• Silero - Voice Activity Detection
• Spotify - User data inspiration

---

📞 Support

• Documentation: See /Documentation folder
• Issues: Create GitHub issue
• Discord: [Your Discord Server]
• Email: [Your Email]

---

🎉 Ready to revolutionize DJing with AI!

For detailed guides, see:

• INTEGRATED_SYSTEM_GUIDE.md - Complete workflow
• GEMINI_INTEGRATION_SPEC.md - Technical details
• LIVE_AUDIO_SETUP.md - Audio recording setup