Vidapter

A powerful Python library for converting RTSP streams and video files into RAG-ready embeddings using vision and language models. Built on top of LlamaIndex with GPT-4 Vision support.

✨ Features

• 🎬 YouTube Video Processing - Download and process YouTube videos automatically
• 🖼️ Frame Extraction - Extract frames at configurable rates with multiple format support
• 🎤 Speech Recognition - Transcribe audio using Whisper for accurate text extraction
• 🔍 Multimodal RAG - Build indexes combining text and image embeddings
• 🤖 GPT-4 Vision - Query your videos using state-of-the-art vision-language models
• 📹 RTSP Streaming - Real-time processing of RTSP camera streams
• 💾 Vector Storage - Efficient storage using LanceDB for fast retrieval
• 📊 Progress Tracking - Detailed logging and progress bars for all operations

🚀 Installation

From PyPI (Recommended)

pip install vidapter

From Source

git clone https://github.com/aeyei/vidapter.git
cd vidapter
pip install -e .

With Development Dependencies

pip install -e ".[dev]"

⚙️ Configuration

Environment Variables

Variable	Description	Required
OPENAI_API_KEY	Your OpenAI API key for GPT-4 Vision	Yes (for querying)
VIDAPTER_OUTPUT_DIR	Default output directory	No (defaults to ./output)

Set your API key:

export OPENAI_API_KEY="your-api-key-here"

📖 Usage

Command Line Interface

Process a Video

# Basic usage - process a YouTube video
vidapter process "https://youtube.com/watch?v=VIDEO_ID"

# With custom output directory
vidapter process "https://youtube.com/watch?v=VIDEO_ID" --output-dir ./my_video

# With custom frame rate (frames per second)
vidapter process "https://youtube.com/watch?v=VIDEO_ID" --frame-rate 1.0

# With different image format
vidapter process "https://youtube.com/watch?v=VIDEO_ID" --image-format jpg

Query a Processed Video

# Basic query
vidapter query "What is happening in the video?"

# With custom output directory (must match processing directory)
vidapter query "What is happening in the video?" --output-dir ./my_video

# With custom retrieval parameters
vidapter query "Describe the main events" --similarity-top-k 5 --image-similarity-top-k 3

# Using a specific model
vidapter query "What objects are visible?" --model gpt-4-turbo

Python API

Video Processing

from vidapter import VideoProcessor, VideoConfig

# Create configuration
config = VideoConfig(
    output_video_path="./video_data/",
    output_folder="./mixed_data/",
    frame_rate=0.5,  # Extract 1 frame every 2 seconds
    image_format="png"
)

# Initialize processor
processor = VideoProcessor(config=config)

# Process a YouTube video
metadata, transcription = processor.process_video("https://youtube.com/watch?v=VIDEO_ID")

print(f"Video Title: {metadata['Title']}")
print(f"Transcription: {transcription[:500]}...")

RAG Engine

from vidapter import RAGEngine, RAGConfig

# Create RAG configuration
config = RAGConfig(
    output_folder="./mixed_data/",
    similarity_top_k=3,
    image_similarity_top_k=3,
    model_name="gpt-4-turbo"
)

# Initialize RAG engine
rag_engine = RAGEngine(config=config, api_key="your-openai-api-key")

# Build the index from processed video data
rag_engine.setup_index()

# Retrieve relevant content
retrieved_images, retrieved_text = rag_engine.retrieve("What is the main topic?")

# Generate a response
response = rag_engine.generate_response(
    query_str="What is the main topic discussed?",
    context_str="\n".join(retrieved_text),
    metadata={"source": "video"},
    image_paths=retrieved_images
)

print(response)

RTSP Streaming

from vidapter import RTSPStreamer, RAGConfig

# Create RAG configuration for RTSP
config = RAGConfig(
    output_folder="./rtsp_frames/",
    vector_store_uri="lancedb",
    text_collection="rtsp_text",
    image_collection="rtsp_images"
)

# Initialize RTSP streamer
streamer = RTSPStreamer(
    rtsp_url="rtsp://your-camera-ip:554/stream",
    frame_interval=1.0,  # Capture one frame per second
    output_dir="./rtsp_frames/"
)

# Set up RAG engine for real-time embedding
streamer.setup_rag_engine(config=config, api_key="your-openai-api-key")

# Start streaming (runs in background)
streamer.start()

# ... do other work ...

# Query the captured frames
images, text = streamer.rag_engine.retrieve("What is visible in the stream?")

# Stop streaming when done
streamer.stop()

Using Context Manager

from vidapter import RTSPStreamer, RAGConfig

config = RAGConfig(output_folder="./rtsp_frames/")

with RTSPStreamer("rtsp://camera-url", frame_interval=2.0) as streamer:
    streamer.setup_rag_engine(config=config, api_key="your-key")
    
    # Stream is automatically started and will be stopped on exit
    import time
    time.sleep(60)  # Capture for 60 seconds

🔧 CLI Options

Global Options

Option	Description	Default
--verbose, -v	Enable verbose logging	False
--api-key	OpenAI API key	$OPENAI_API_KEY
--output-dir	Base output directory	./output

Video Processing Options

Option	Description	Default
--frame-rate	Frames per second to extract	0.5
--image-format	Image format (png, jpg, jpeg)	png

RAG Options

Option	Description	Default
--similarity-top-k	Number of text chunks to retrieve	3
--image-similarity-top-k	Number of images to retrieve	3
--model	OpenAI model for generation	gpt-4-turbo

📁 Output Structure

output/
├── video_data/           # Downloaded video files
│   └── input_vid.mp4
└── mixed_data/           # Processed data
    ├── frame0001.png     # Extracted frames
    ├── frame0002.png
    ├── ...
    └── output_text.txt   # Transcribed audio

rtsp_frames/              # RTSP stream output (if used)
├── frame_2024-01-01T12-00-00.jpg
├── frame_2024-01-01T12-00-01.jpg
└── ...

lancedb/                  # Vector store database
├── text_collection/
└── image_collection/

🧪 Development

Setup

# Clone the repository
git clone https://github.com/aeyei/vidapter.git
cd vidapter

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install development dependencies
pip install -e ".[dev]"

# Install pre-commit hooks (optional)
pre-commit install

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=vidapter --cov-report=html

# Run specific test file
pytest tests/test_video_processor.py -v

Code Quality

# Format code
black vidapter tests
isort vidapter tests

# Lint
flake8 vidapter tests

# Type check
mypy vidapter

Building

# Build distribution packages
python -m build

# Check package
twine check dist/*

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• LlamaIndex for the RAG framework
• OpenAI for GPT-4 Vision
• LanceDB for vector storage
• MoviePy for video processing
• pytubefix for YouTube downloads

📬 Contact

• Author: Arnav Gupta
• Email: ar9avg@gmail.com
• GitHub: @aeyei

---

Made with ❤️ for the AI community