A system that transforms YouTube videos into queryable multimodal semantic graphs. Built for research in video understanding, featuring transcript-first graph construction, visual enrichment, and cross-video semantic search.
When you provide a YouTube URL, the system:
- Downloads the video using
yt-dlp - Extracts audio as 16kHz mono WAV (optimal for speech recognition)
- Detects scenes using ffmpeg's built-in scene detection filter
- Extracts frames at configurable FPS (default: 1 fps)
- Segments clips based on scene boundaries
All heavy processing uses ffmpeg directly—no OpenCV or PySceneDetect dependencies.
Audio is sent to OpenAI Whisper API for transcription:
- Segment-level timestamps for precise alignment
- Automatic audio compression for files exceeding 24MB
- Outputs both raw segments and sentence-level groupings
Each video clip is analyzed using OpenAI GPT-4o Vision:
- Visual Captioning: Describes what's happening in the scene
- Entity Detection: Identifies objects, people, UI elements
- Scene Classification: talking_head, presentation_slide, screencast, etc.
- OCR Extraction: Reads on-screen text via Vision API
Processing runs in parallel (5 concurrent calls by default) for speed.
The system builds a semantic graph with this structure:
Nodes:
| Type | Description |
|---|---|
TranscriptNode |
Sentence with text, timestamps, topic assignment |
VisualNode |
Clip with description, entities, OCR, scene type |
TopicNode |
Topic segment with title, keywords, time range |
EntityNode |
Canonical entity (person, product, concept) with aliases |
Edges:
| Type | Description |
|---|---|
TEMPORAL_NEXT |
Sequential ordering between segments |
ALIGNED_TO |
Transcript ↔ Visual temporal overlap |
DISCOURSE_RELATION |
EXPLAINS, SUPPORT, COUNTER, EXAMPLE_OF, etc. |
BELONGS_TO_TOPIC |
Segment → Topic membership |
CONTAINS_ENTITY |
Node mentions entity |
The construction is transcript-first: the semantic structure comes from analyzing the transcript, then visual nodes are attached via temporal alignment.
📖 For detailed explanation of the graph building methodology, see GRAPH_BUILDING.md
After graph construction, embeddings are computed for all nodes using text-embedding-3-small:
- Runs in parallel (10 concurrent calls by default)
- Saved to
embeddings.jsonfor instant retrieval - Enables semantic search without query-time embedding computation
When you ask a question:
- Hybrid Search: Combines embedding similarity + keyword matching
- k-hop Expansion: Retrieves related nodes through graph edges
- Context Assembly: Collects relevant transcript and visual evidence
- Answer Generation: LLM synthesizes answer with timestamp citations
Responses include:
- Direct answer to the question
- Evidence items with node IDs and timestamps
- Relevant subgraph for visualization
The system supports searching across all processed videos:
- Semantic search using pre-computed embeddings
- Keyword fallback for videos without embeddings
- LLM synthesizes answers citing multiple videos
- Reports which videos contain relevant information
The system identifies these semantic relationships between transcript segments:
| Relation | Meaning |
|---|---|
EXPLAINS |
A clarifies or elaborates on B |
EXAMPLE_OF |
A is a concrete instance of B |
SUPPORT |
A provides evidence for B |
COUNTER |
A contradicts or opposes B |
CAUSES |
A leads to or results in B |
STEP_BEFORE |
A precedes B in a sequence |
ELABORATES |
A adds detail to B |
DEFINES |
A provides definition of term in B |
Each topic segment is classified by its discourse type:
- Narrative: Story-telling, chronological events
- Descriptive: Explaining what something is
- Informative: Presenting facts or information
- Instructional: Teaching how to do something
- Argumentative: Making a case or persuading
- Visual captioning: 5 concurrent API calls
- OCR extraction: 5 concurrent API calls
- Embedding computation: 10 concurrent API calls
- Reduces processing time by 60-70% for LLM-heavy stages
- Graph Viewer: Limits to 200 nodes with Topics/Sampled view toggle
- Transcript Panel: Virtual scrolling, renders only visible segments
- Time Updates: Throttled to 2/second to prevent re-render storms
- All OpenAI API calls are cached
- Embeddings pre-computed and stored
- Graph exported to JSON and GraphML formats
For a 10-minute video:
| Stage | Output |
|---|---|
| Download | video.mp4 (50-200 MB) |
| Audio | audio.wav (16kHz mono) |
| Frames | ~600 JPGs at 1 fps |
| Clips | ~15-20 scene-based clips |
| Transcript | ~80-120 segments |
| Visual | 15-20 clip analyses |
| Graph | 100-200 nodes, 200-400 edges |
| Embeddings | 100-200 vectors (1536-dim each) |
Videos are primarily experienced through spoken content. Building the semantic structure from transcript ensures the graph reflects the narrative flow, with visual content enriching rather than driving the structure.
Using a single provider (OpenAI) for transcription, vision, OCR, embeddings, and generation:
- Minimizes dependencies (no Tesseract, OpenCV, local models)
- Smaller Docker images
- Consistent quality across modalities
- Simpler deployment
Computing embeddings at query time for 100+ nodes adds 10-30 seconds of latency. Pre-computing during ingestion makes queries instant.
Pure semantic search can miss exact keyword matches (e.g., "Cisco" might not be semantically similar to query). Combining embedding similarity + keyword matching ensures both conceptual and lexical matches are found.
Backend:
- Python 3.12
- FastAPI + Uvicorn
- OpenAI API (Whisper, GPT-4o, embeddings)
- ffmpeg for video processing
- yt-dlp for YouTube downloads
Frontend:
- React 18
- Cytoscape.js for graph visualization
- TanStack Query for data fetching
- Framer Motion for animations
Infrastructure:
- Docker + Docker Compose
- Alpine Linux base (minimal image size)
After processing a video, the system creates:
data/videos/{video_id}/
├── video.mp4 # Downloaded video
├── audio.wav # Extracted audio
├── metadata.json # Video metadata
├── transcript.json # Timestamped transcript
├── visual.json # Visual analysis results
├── graph.json # Full graph structure
├── graph.graphml # GraphML export
├── embeddings.json # Pre-computed embeddings
├── frames/ # Extracted frames
└── clips/ # Scene-based clips
This system is designed for research in:
- Video Understanding: Semantic analysis of video content
- Multimodal Learning: Integrating text, speech, and vision
- Knowledge Graphs: Temporal, discourse-aware graph construction
- Explainable AI: Evidence-based answers with citations
- Video Search: Semantic retrieval across video corpora