🎓 Educational Purpose: This project is designed for learning and understanding Multimodal RAG (Retrieval-Augmented Generation) pipelines. It demonstrates how to combine computer vision, natural language processing, and vector databases to create intelligent search systems that work across different data modalities (text, images, video). Perfect for students, researchers, and developers wanting to understand modern AI architectures.
Have you ever wanted to find that specific moment in a video where someone explained a concept? Or search through hours of video content as easily as searching text? This educational project shows you how to build a production-ready video search engine that lets you query video content using natural language or even images.
Beautiful web interface with multi-video search, YouTube downloading, and real-time statistics
We're creating a Video RAG (Retrieval-Augmented Generation) system that:
- 📥 Downloads YouTube videos automatically
- 📸 Extracts frames from videos at regular intervals
- 🧠 Generates semantic embeddings using OpenAI's CLIP model
- 💾 Stores embeddings in Qdrant vector database
- 🔍 Enables text-to-video search ("find frames with whiteboards")
- 🖼️ Supports image similarity search
- ⏱️ Allows time-based queries
- 🎬 Supports multiple videos in one searchable collection
- 🔗 Generates direct YouTube timestamps for results
- 🌐 Includes a beautiful web interface for easy searching
- 🛡️ Robust Error Handling: Advanced 403/bot detection prevention
- 🗑️ Smart Storage Management: Auto-deletes videos after processing (95% space savings)
- 🔄 Auto-Reload Development: Server automatically restarts on code changes
- 🧪 Connection Testing: Built-in YouTube connectivity diagnostics
- 📱 Responsive UI: Works seamlessly on desktop and mobile devices
- ⚡ Performance Optimized: Multi-strategy downloads with automatic fallbacks
Before we start, make sure you have:
- Python 3.12+
- Docker and Docker Compose
- FFmpeg (for video processing)
- 4GB+ free disk space
git clone https://github.com/di37/video-rag-bot.git
cd video-rag
pip install -r requirements.txtdocker-compose up -dThis launches Qdrant on http://localhost:6333. You can view the dashboard at http://localhost:6333/dashboard.
# Download YouTube video and auto-index
python main.py download "https://youtu.be/VIDEO_ID" --auto-index
# Or download multiple videos
python main.py download "https://youtu.be/VIDEO_ID_1" --auto-index
python main.py download "https://youtu.be/VIDEO_ID_2" --auto-index
Easy-to-use web interface for downloading YouTube videos with configurable frame extraction
Comprehensive video library management with metadata, search shortcuts, and deletion options
Alternative: Manual video preparation (if needed):
# Extract frames (1 frame every 5 seconds)
mkdir screenshots
ffmpeg -i video.mp4 -vf "fps=0.2" screenshots/frame_%04d.jpg
# Generate metadata
python create_metadata.pypython main.py indexThis will:
- Load video frame metadata
- Generate CLIP embeddings for each frame
- Store them in Qdrant vector database
# Start the web server
python app.pyThen open http://localhost:7777 in your browser for a beautiful search interface!
Intelligent semantic search finds relevant frames across multiple videos with timestamp links
# Download videos
python main.py download "https://youtu.be/VIDEO_ID" --auto-index
# Text search (all videos)
python main.py query --text "person explaining neural networks"
# Search specific video
python main.py query --text "neural networks" --video-id VIDEO_ID
# Image similarity
python main.py query --image screenshots/frame_0100.jpg
# Time range search
python main.py query --time-range 5:00 10:00
# List all videos
python main.py list videosOur system includes enterprise-grade protection against YouTube's anti-bot measures:
# Test YouTube connectivity before downloading
python main.py test
# Output:
🧪 Testing YouTube connectivity...
✅ YouTube connection is working properly
🎬 Test video: Rick Astley - Never Gonna Give You Up (Official Video)
🎉 You can now download YouTube videos!Multi-Strategy Download System:
- Strategy 1: Standard download with browser-like headers
- Strategy 2: Automatic fallback with conservative settings
- Built-in Retries: Handles 403 Forbidden errors intelligently
- Smart Error Messages: Guides users on how to resolve issues
Automatic Video Cleanup (saves ~95% disk space):
# After frame extraction:
🗑️ Deleted video file: video.webm (187.3 MB saved)
✅ Kept: 416 frame screenshots + metadata (~5 MB)User Control Options:
- ✅ Default: Auto-delete videos after processing (recommended)
- 🔧 Optional: Keep video files (uncheck "Keep video file" in UI)
- 📊 Space Monitoring: Real-time storage usage reporting
Auto-Reload Server:
# Server automatically restarts when you modify code
INFO: WatchFiles detected changes in 'app.py'. Reloading...Dynamic UI Updates:
- ✅ Real-time video deletion from interface
- ✅ Instant statistics updates
- ✅ No manual refresh needed
- ✅ Complete file cleanup (database + physical files)
Built-in Connectivity Testing:
python main.py test # Test YouTube access
python main.py list stats # View collection statistics
python main.py list videos # See all processed videosComprehensive Error Handling:
- 🔍 403 Forbidden: "YouTube blocked access - try again in a few minutes"
- 🔍 404 Not Found: "Video not found - check the URL"
- 🔍 Private Videos: "Video is private and cannot be downloaded"
- 🔍 Live Streams: "Cannot download live streams"
YouTube 403 Forbidden Errors:
# 1. Update yt-dlp to latest version
pip install --upgrade yt-dlp
# 2. Test connectivity first
python main.py test
# 3. Wait a few minutes and try again (rate limiting)
# 4. Try a different video URLVideo Download Fails:
# Check video availability
- Is the video public?
- Does it require YouTube Premium?
- Is it geo-restricted in your region?
- Is it a live stream? (not supported)Out of Disk Space:
# Monitor space usage
du -sh video-downloads/ # Check download directory size
python main.py list stats # View collection statistics
# Enable auto-cleanup (default)
✅ Keep "Delete video files" enabled in UISlow Processing:
# Optimize for your hardware
- GPU: Install torch with CUDA support
- CPU: Reduce batch size in config.py
- RAM: Close other applications during indexingWeb Interface Issues:
# Common fixes
- Hard refresh: Ctrl+F5 (Windows) or Cmd+Shift+R (Mac)
- Check server logs for errors
- Restart server: python app.py
- Clear browser cachevideo-rag/
├── main.py # CLI entry point
├── app.py # Web interface
├── src/
│ ├── core/ # Base classes and config
│ ├── indexing/ # Frame indexing logic
│ ├── querying/ # Search functionality
│ └── utils/ # Helper functions
├── static/ # Web UI assets
│ ├── index.html # Main page
│ ├── style.css # Styling
│ └── script.js # Interactivity
├── screenshots/ # Extracted video frames
├── video_metadata.json # Frame timestamps
└── docker-compose.yml # Qdrant setup
- Frame Extraction: FFmpeg samples video at 0.2 fps (1 frame/5 seconds)
- Embedding Generation: CLIP model converts frames to 512D vectors
- Vector Storage: Qdrant stores embeddings with metadata
- Semantic Search: CLIP encodes queries to match against stored vectors
The Video RAG system is powered by OpenAI's CLIP model, specifically the clip-ViT-B-32 variant. CLIP is revolutionary because it understands both text and images in the same 512-dimensional vector space, making it perfect for multi-modal video search.
Traditional Approach:
- Text search → keyword matching → limited results
- Image search → pixel comparison → no semantic understanding
CLIP Approach:
- Text search → semantic understanding → finds concepts, not just words
- Image search → visual concept matching → understands what's happening in frames
- Cross-modal search: Describe an image with text, find similar visual content
- Model:
clip-ViT-B-32(Vision Transformer with 32x32 patch size) - Vector Dimensions: 512D embeddings
- Training: 400M image-text pairs from the internet
- Languages: Primarily English, with some multilingual capability
- Performance: Balances speed and accuracy for real-time search
# These all work because CLIP understands concepts:
"person drawing on whiteboard" → finds teaching moments
"neural network diagram" → locates architecture explanations
"confused facial expression" → identifies difficult concepts
"hands gesturing" → discovers animated explanationsThe ViT-B-32 architecture:
- Patches: Breaks images into 32×32 pixel patches
- Attention: Uses transformer attention across visual patches
- Efficiency: Faster than convolutional approaches
- Scalability: Handles various image sizes and resolutions
This makes it ideal for processing video frames at scale while maintaining semantic understanding of visual content.
CLIP's magic is that it understands both text and images in the same vector space:
# Text → Vector → Similar Frames
results = engine.search_by_text("whiteboard with equations")
# Image → Vector → Similar Frames
results = engine.search_by_image("reference_image.jpg")# Get all frames between 10:00 and 15:00
results = engine.search_by_time_range(600, 900)Each result includes a clickable timestamp:
https://youtu.be/VIDEO_ID?t=325 # Jumps to 5:25
- Educational Content: Find specific explanations in lecture videos
- Meeting Archives: Search through recorded meetings
- Tutorial Navigation: Jump to relevant sections in how-to videos
- Content Moderation: Detect specific visual elements
- Video Summarization: Extract key frames for highlights
Edit src/core/config.py to customize:
# Change embedding model
MODEL_NAME = "clip-ViT-L-14" # Larger, more accurate
# Adjust processing
BATCH_SIZE = 64 # For GPUs
DEFAULT_SEARCH_LIMIT = 10
# Different frame extraction rate
# In FFmpeg: -vf "fps=0.5" # 1 frame every 2 seconds- GPU Acceleration: Install
torchwith CUDA for 10x faster indexing - Batch Processing: Increase
BATCH_SIZEbased on your RAM - Optimize Storage: Use Qdrant's quantization for large datasets
- Cache Embeddings: Save computed embeddings for reuse
For production deployments:
- Use Qdrant Cloud for managed hosting
- Implement API with FastAPI for web access
- Add Authentication for secure access
- Enable Monitoring with Prometheus/Grafana
- Use CDN for frame serving
This educational project can be extended in many exciting directions:
# Add speech-to-text transcription
- Whisper integration for automatic subtitles
- Audio-based search: "find where they mention neural networks"
- Multi-language support with translation
- Speaker identification and diarization# Advanced AI capabilities
- Video summarization with GPT-4
- Automatic chapter generation
- Key moment detection
- Sentiment analysis of video content
- Object detection and tracking# Advanced analytics dashboard
- Search patterns analysis
- Popular content identification
- Usage statistics and heatmaps
- Performance metrics tracking
- A/B testing for search algorithms# Scale to production
- Kubernetes deployment with Helm charts
- AWS/GCP cloud integration
- CDN for global frame serving
- Redis caching layer
- Horizontal scaling with load balancers# External integrations
- REST API with OpenAPI documentation
- Webhook support for real-time updates
- Slack/Discord bot integration
- Browser extension for YouTube
- Mobile app with React Native# Power user features
- Custom embedding models (OpenCLIP, DINOv2)
- Facial recognition and person tracking
- Scene detection and transitions
- Playlist and collection management
- Collaborative annotations and bookmarks# Academic research directions
- Multimodal fusion techniques
- Zero-shot video understanding
- Cross-lingual video search
- Temporal consistency in embeddings
- Federated learning across video collections# Enhanced UX/UI
- Real-time search suggestions
- Keyboard shortcuts and hotkeys
- Dark mode and accessibility features
- Video preview on hover
- Batch operations and bulk actionsBy exploring this project, you'll understand:
- Multimodal AI: How to combine vision and language models
- Vector Databases: Efficient similarity search at scale
- Production MLOps: Deployment, monitoring, and maintenance
- RAG Architectures: Retrieval-augmented generation patterns
- Modern Web Development: FastAPI, async programming, real-time UIs
- Video Processing: FFmpeg, frame extraction, and media handling
Feel free to open issues or submit PRs! Some beginner-friendly ideas:
- Add new video format support (.mov, .avi)
- Improve error messages and user feedback
- Add keyboard shortcuts to web interface
- Create video thumbnails for better previews
- Implement audio transcription with Whisper
- Add batch video processing
- Create export/import functionality for collections
- Build a browser extension
- Integrate with cloud storage (S3, GCS)
- Add real-time video streaming support
- Implement federated search across instances
- Create custom embedding fine-tuning pipeline
This educational project stands on the shoulders of giants. Built with amazing open-source projects:
- OpenAI CLIP: Revolutionary multimodal understanding
- Sentence Transformers: High-quality embeddings made easy
- Hugging Face: Transformers ecosystem and model hub
- Qdrant: High-performance vector database
- Docker: Containerization and deployment
- FFmpeg: The Swiss Army knife of video processing
- yt-dlp: Robust YouTube downloading with anti-bot protection
- FastAPI: Modern, fast web framework with auto-documentation
- Uvicorn: Lightning-fast ASGI server with auto-reload
- Playwright: Reliable browser automation for testing
- Modern CSS: Responsive design without frameworks
- Vanilla JavaScript: Clean, dependency-free interactivity
This project demonstrates real-world applications of:
- Multimodal RAG pipelines in production
- Vector search at scale with semantic understanding
- Modern MLOps practices and deployment strategies
- Full-stack AI application development
Perfect for:
- 🎓 Computer Science students learning about AI systems
- 🔬 Researchers exploring multimodal applications
- 👩💻 Developers building production AI systems
- 🏫 Educators teaching modern AI architectures
Happy Learning and Video Searching! 🎥🔍🎓
If this helped you understand Multimodal RAG, please ⭐ star the repo and share your learning journey!