Video RAG (Retrieval Augmented Generation) Application

A modular application for semantic video analysis using CLIP embeddings and GPT-4o.

Figure: Video RAG pipeline diagram

Overview

This project implements a Video Retrieval Augmented Generation (RAG) system that combines computer vision and large language models to analyze and understand video content.

How It Works

1. Video Processing: Extracts key frames from videos at regular intervals
2. Semantic Embeddings: Uses CLIP (Contrastive Language-Image Pre-training) to generate embeddings for both video frames and text queries
3. Semantic Search: Finds the best matching frame(s) using cosine similarity between text and image embeddings
4. LLM Analysis: Sends the best matching frame to GPT-4o for detailed structured analysis
5. Structured Output: Returns JSON with frame metadata, object detection, and action summaries

Key Features

• Semantic Search: Find relevant video moments by describing what you're looking for in natural language
• Multi-match Support: Get single best match or top K matching frames
• Structured Analysis: Automatic JSON generation with timestamps, object detection, and action descriptions
• Modular Design: Clean separation of concerns with independent, reusable components
• Easy Configuration: Environment-based configuration for API keys and settings

Project Structure

Video_rag/
├── app.py              # Main entry point
├── agent.py            # VideoAI Agent orchestrator
├── models.py           # CLIP model management
├── video_utils.py      # Video processing utilities
├── llm_analyzer.py     # LLM integration for analysis
├── search.py           # Semantic search functionality
├── config.py           # Configuration management
├── requirements.txt    # Python dependencies
└── video/              # Video files directory
    └── test_1.mp4

Modules

app.py

Main entry point of the application. Initializes the VideoAI_Agent and runs the analysis pipeline with default or custom settings.

Key Function:

• main(): Loads configuration, initializes the agent, and executes video analysis

agent.py

VideoAI_Agent: Main orchestrator that coordinates all components in the analysis pipeline.

Key Methods:

• analyze_video(video_path, query_text, frame_interval=30): Find the single best matching frame for a given query
• analyze_video_top_k(video_path, query_text, k=5, frame_interval=30): Find top K matching frames and analyze each

How it works:

1. Extracts frames from video
2. Generates CLIP embeddings for frames and query
3. Performs semantic search using cosine similarity
4. Sends top match(es) to GPT-4o for detailed analysis
5. Returns structured JSON output

models.py

CLIPModelManager: Manages CLIP model loading and inference for embedding generation.

Key Methods:

• get_image_features(frames): Convert PIL images to normalized embeddings
• get_text_features(text): Convert text to normalized embeddings

Technical Details:

• Uses openai/clip-vit-base-patch32 model by default
• Loads models with safetensors format (secure alternative to PyTorch pickle)
• Supports GPU acceleration via CUDA when available

video_utils.py

VideoProcessor: Handles all video-related operations.

Key Methods:

• extract_frames(video_path, frame_interval=30): Extract frames at regular intervals, returns frames list and timestamps
• frame_to_base64(pil_image): Convert PIL Image to base64 JPEG string for LLM transmission

Technical Details:

• Uses OpenCV for efficient video reading
• Preserves frame timing information for timestamp accuracy
• Converts frames to RGB for CLIP compatibility

llm_analyzer.py

LLMAnalyzer: Handles communication with OpenAI's GPT-4o model for structured frame analysis.

Key Methods:

• analyze_frame(base64_image, query_text, timestamp): Send image to GPT-4o with context and receive structured JSON

Output Format:

{
  "timestamp": 15.015,
  "is_match": true,
  "description": "detailed description of what is happening",
  "detected_objects": ["object1", "object2"],
  "action_summary": "what action is taking place"
}

search.py

SemanticSearcher: Performs semantic search using CLIP embeddings.

Key Methods:

• find_best_match(query_embedding, frame_embeddings, frames, timestamps): Returns best matching frame using cosine similarity
• find_top_k_matches(query_embedding, frame_embeddings, frames, timestamps, k=5): Returns top K matches sorted by relevance

Algorithm: Uses dot product between normalized embeddings to compute cosine similarity scores (values between -1 and 1, higher is better).

config.py

Centralized configuration management for the application.

Configuration Options:

• OPENAI_API_KEY: Your OpenAI API key (from environment or defaults)
• CLIP_MODEL_ID: HuggingFace model identifier for CLIP
• FRAME_INTERVAL: Extract 1 frame every N frames
• DEFAULT_VIDEO_PATH: Path to video file for testing
• DEFAULT_QUERY: Default search query for testing

Configuration

Environment Setup

Set up your OpenAI API key via environment variable:

export OPENAI_API_KEY="sk-..."

Or create a .env file in the project root:

OPENAI_API_KEY=your-api-key-here

The config.py module will automatically load from .env if available, otherwise use the environment variable or default placeholder.

Customizing Settings

Edit config.py to modify default behavior:

# Adjust frame sampling rate (extract 1 frame every 60 frames instead of 30)
FRAME_INTERVAL = 60

# Use a different CLIP model
CLIP_MODEL_ID = "openai/clip-vit-large-patch14"

# Change default video path
DEFAULT_VIDEO_PATH = "video/your_video.mp4"

Usage

Basic Usage

Run the application with default settings:

python app.py

This will:

1. Load the video from video/test_1.mp4
2. Search for: "A person typing on a laptop"
3. Return the best matching frame with structured analysis

Example: Custom Analysis

from agent import VideoAI_Agent
from config import Config

# Initialize agent
agent = VideoAI_Agent(api_key=Config.OPENAI_API_KEY)

# Find single best match
result = agent.analyze_video(
    "video/test_1.mp4",
    "A person typing on a laptop"
)
print(result)

Example: Top K Matches

Find and analyze multiple matching frames:

from agent import VideoAI_Agent
from config import Config

agent = VideoAI_Agent(api_key=Config.OPENAI_API_KEY)

# Find top 5 matches
results = agent.analyze_video_top_k(
    "video/test_1.mp4",
    "A person typing on a laptop",
    k=5,
    frame_interval=30  # Extract 1 frame every 30 frames
)

# Process results
for i, result in enumerate(results, 1):
    print(f"\n--- Match {i} ---")
    print(f"Timestamp: {result['timestamp']}s")
    print(f"Description: {result['description']}")
    print(f"Objects: {', '.join(result['detected_objects'])}")

Example: Batch Video Analysis

Process multiple videos:

from agent import VideoAI_Agent
from config import Config

agent = VideoAI_Agent(api_key=Config.OPENAI_API_KEY)

videos = [
    "video/clip1.mp4",
    "video/clip2.mp4",
    "video/clip3.mp4"
]

queries = [
    "person working at desk",
    "coffee cup on table",
    "computer screen"
]

for video, query in zip(videos, queries):
    print(f"\nAnalyzing: {video}")
    result = agent.analyze_video(video, query)
    print(result)

Understanding the Pipeline

Video Input
    ↓
[VideoProcessor] Extract Frames → [timestamps, PIL images]
    ↓
[CLIPModelManager] Generate Image Embeddings → [frame embeddings]
    ↓
[CLIPModelManager] Generate Query Embedding → [text embedding]
    ↓
[SemanticSearcher] Compute Similarity & Find Best Match → [best_frame, timestamp, score]
    ↓
[VideoUtils] Convert Frame to Base64 → [base64 JPEG]
    ↓
[LLMAnalyzer] Send to GPT-4o → [JSON response]
    ↓
Structured Output (JSON)

Dependencies

See requirements.txt for all dependencies. Key packages:

• torch: Deep learning framework for CLIP model
• transformers: HuggingFace library for CLIP models
• opencv-python (cv2): Video processing and frame extraction
• Pillow (PIL): Image handling and conversion
• openai: GPT-4o API client
• python-dotenv: Environment variable loading (optional)

Installation

1. Clone or download the project
2. Create a Python virtual environment:

   python3 -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Set your OpenAI API key:

   export OPENAI_API_KEY="your-key-here"

5. Run the application:

   python app.py

How Semantic Search Works

This project uses CLIP embeddings for semantic understanding:

1. CLIP Model: Trained on 400M image-text pairs to understand visual and textual semantics
2. Embeddings: Both images and text are converted to 512-dimensional vectors
3. Similarity: Cosine similarity between embeddings measures how well an image matches a text query
4. Search: The frame with highest similarity to the query is selected as the best match

Example Queries

• "A person typing on a laptop"
• "Close-up of coffee cup"
• "Two people talking"
• "Computer screen showing code"
• "Office workspace"

The system handles semantic variations, so "someone working" might match "person at desk" even though the exact words differ.

Troubleshooting

Invalid API Key Error

• Ensure OPENAI_API_KEY environment variable is set correctly
• Check your key at https://platform.openai.com/account/api-keys

CUDA Out of Memory

• Set device to CPU in config.py or modify CLIPModelManager initialization
• Or reduce FRAME_INTERVAL to extract fewer frames

Video File Not Found

• Ensure video path is correct relative to the project directory
• Check DEFAULT_VIDEO_PATH in config.py

Slow Performance

• Increase FRAME_INTERVAL to extract fewer frames (faster but less precise)
• Use GPU acceleration if available (CUDA)