SemanticFrameScoring
title: Semantic Frame Scoring type: technique created: 2026-06-22 last_updated: 2026-06-22 related: ["radar/languages/Ultralytics", "radar/tools/FireRedOpenStoryline"] sources: ["https://nvarma.com/blog/2026-04-05-vibecoding-a-video-editing-pipeline"] radar_quadrant: Techniques radar_ring: Assess radar_position: inner
Semantic Frame Scoring is a video discovery technique in which keyframes are extracted from video shots and scored against natural language text prompts using a text-image similarity model (typically CLIP). Frames are ranked by similarity score and filtered by classical quality signals (blur, camera shake, colorfulness) to produce a ranked candidate list for downstream clip selection or editing. The approach replaces manual review with a query-driven selection pass: the operator describes the desired content in natural language, and the pipeline surfaces matching frames automatically.
| Stage | Responsibility | Typical Tool |
|---|---|---|
| Shot segmentation | Detect cut boundaries; group frames into shots | PySceneDetect |
| Frame extraction | Sample N representative frames per shot | OpenCV / ffmpeg |
| Semantic scoring | Score each frame against text prompt(s) | CLIP |
| Quality filtering | Remove blurry, shaky, or low-color frames | OpenCV (Laplacian, optical flow) |
| Selection | Pick highest-scoring frames while enforcing diversity | Custom ranker |
| Output | Timestamped clip list for downstream editing | JSON / CSV |
CLIP (Contrastive Language-Image Pretraining) encodes both the text prompt and the frame into a shared embedding space. Cosine similarity between the two embeddings produces a score in [-1, 1]. A prompt such as "dramatic Pacific Ocean cliffs" returns high scores for matching frames and low scores for "blurry photo" or "empty parking lot". Multiple prompts can be combined: a positive prompt raises the score, a negative prompt penalises it.
Zero-shot location tagging is a secondary application: GPS coordinates (when available) are mapped to a place label, and CLIP confirms the visual match. When GPS is absent, CLIP zero-shot classification across candidate place names substitutes.
CLIP scoring and YOLO-based detection address different selection criteria. YOLO answers "is there a person/car/ball in this frame?" CLIP answers "does this frame look like X?" The two are composable: run YOLO first to filter frames containing the required subject class, then apply CLIP to rank by aesthetic or contextual quality. radar/techniques/DetectionThenIdentityVideoPipeline is the natural predecessor stage.
Navin Varma applied this pattern to a personal holiday highlight reel (April 2026) on a Mac Mini M4 Pro (24GB RAM). The 10-stage pipeline processed an unspecified number of video files; CLIP ran via Apple Silicon MPS. Limitations documented: CLIP confused two similar-looking piers (zero-shot location tagging failure); a 35-image batch limit before memory exhaustion; technically well-scored frames that "felt out of place in the flow" -- requiring a manual iMovie pass for narrative reordering. No public code. Output published as a YouTube highlight reel and two Shorts.
A separate attempt used Gemma 4 (gemma-4-E4B-it-4bit) for frame captioning; it hallucinated filenames and failed at 112px resolution. CLIP was retained as the scoring backbone.
The technique selects visually representative or aesthetically matching frames but has no concept of narrative order or pacing. Frames selected by score alone may be individually good but collectively incoherent as a sequence. Human judgment or a separate ordering pass remains necessary for story-driven edits. CLIP zero-shot location tagging degrades when visual environments are similar across candidate labels.
Placed in Techniques / Assess / inner.
Semantic Frame Scoring is a practical alternative to manual clip review for large video libraries. CLIP is a mature model (OpenAI, widely available via pip install clip or transformers), and the scoring loop requires no training data or domain-specific fine-tuning. The inner Assess position reflects that the pattern is immediately evaluable using off-the-shelf libraries and that the use case -- surfacing the best frames from hours of footage -- is directly relevant to any personal or professional video workflow.
The ring stays at Assess rather than Trial because the sole documented production application is a single personal project with no public code, and the author explicitly notes residual manual editing was required. Confirmed use of CLIP scoring in a pipeline that produces a segment list consumed programmatically by a downstream editing tool, with no required manual reordering, would satisfy the Trial gate.
