PersonalVideoProcessingPipeline
title: Personal Video Processing Pipeline type: technique created: 2026-07-04 last_updated: 2026-07-04 related: ["radar/languages/Ultralytics", "radar/languages/DeepFace", "radar/tools/FireRedOpenStoryline", "radar/techniques/DetectionThenIdentityVideoPipeline", "radar/techniques/SemanticFrameScoring", "radar/tools/EditMind", "radar/tools/Sentrysearch", "radar/languages/PySlowFast", "radar/tools/VideoAgent"] radar_quadrant: Techniques radar_ring: Assess radar_position: inner
Turning hours of raw personal video footage into a finished edit is a manual bottleneck: scrubbing timelines to find the moment a specific person appears, and then cutting that moment into a coherent sequence, does not scale past a handful of clips. This technique composes three independently-maintained open-source stages into a single footage-to-edit pipeline, so that discovery and editing happen through natural language and structured data rather than manual timeline scrubbing.
The pipeline treats video processing as three composable stages, each backed by an existing open-source library or agent rather than custom code: a discovery stage that locates and identifies subjects per frame, an editing stage that turns natural language instructions into cuts, and an optional scoring stage that ranks frames by semantic and aesthetic quality before they reach the editor. Each stage produces structured output (timestamps, identities, scores) consumed programmatically by the next.
Stage 1 -- Discovery. The radar/techniques/DetectionThenIdentityVideoPipeline pattern: radar/languages/Ultralytics (YOLO) locates subjects per frame and produces bounding boxes and track IDs; radar/languages/DeepFace matches bounding-box crops to a reference identity database, with results cached per track ID to avoid a per-frame identity call. Output is a timestamped segment list -- who appears, and when.
Stage 2 -- Editing. radar/tools/FireRedOpenStoryline consumes natural language instructions and, optionally, the Stage 1 segment list, to perform agentic editing via an MCP server and reusable Claude Code Skills. Output is a finished edit.
Optional scoring layer. The radar/techniques/SemanticFrameScoring pattern: PySceneDetect segments shots, CLIP scores keyframes against natural language prompts, and OpenCV quality filters (blur, shake, colorfulness) remove poor candidates. This layer sits between discovery and editing when the goal is aesthetic or contextual selection rather than subject identity -- the two are composable but address different selection criteria.
Stage 1 -- library composition vs. bespoke application. Composing radar/languages/Ultralytics and radar/languages/DeepFace directly as libraries was chosen over building a bespoke wrapper application. Both libraries already expose a video-in, structured-output contract (JSON/CSV for ultralytics, DataFrames via find() for deepface); a bespoke application would add a service layer around a problem two already-installed libraries solve directly.
Stage 1 -- alternative discovery tools considered. radar/tools/EditMind packages the same primitives (YOLO, DeepFace, Whisper) but was rejected: its output is locked in PostgreSQL and ChromaDB with no documented export API, making it non-composable with a downstream editing stage regardless of detection quality. radar/tools/Sentrysearch was rejected as a proof-of-concept only -- a personal demo built on Gemini multimodal embeddings, with no production-grade foundation yet. radar/languages/PySlowFast was rejected on direct evidence: a third-party production pipeline (Ippon Technologies, rugby analytics) attempted it for action classification and abandoned it due to annotation cost.
Stage 2 -- alternative editing agent considered. radar/tools/VideoAgent was rejected in favor of radar/tools/FireRedOpenStoryline: it requires four separate LLM providers (Claude for routing, GPT-4o for editing, Gemini for captioning, plus a fourth for voice synthesis) alongside an 8GB GPU and multiple HuggingFace checkpoints, and all documented use is marked research-only with no confirmed production deployments.
Ranked by severity, one per considered path:
Library composition (Stage 1) -- DeepFace stream() gap (high). find() returns DataFrames usable in a batch pipeline; stream() returns None, so live-stream composition is not possible today. The workaround is batch identity matching once per new track ID rather than per frame.
edit-mind (rejected) -- no export path (high). Output is locked in PostgreSQL and ChromaDB behind an undocumented internal API; there is no way to get a segment list out to feed Stage 2.
sentrysearch (rejected) -- proof-of-concept maturity (high). A single personal demo with no production-grade foundation; the embedding-based discovery technique it demonstrates is promising but not yet a candidate for adoption.
PySlowFast (rejected) -- confirmed abandonment (high). The only production data point available shows it dropped from a real pipeline over annotation cost; adopting it for action classification would repeat a documented failure.
VideoAgent (rejected) -- operational cost (medium). A four-LLM-provider hard requirement plus GPU and checkpoint installation is a heavier operational footprint than FireRed-OpenStoryline's single-provider MCP architecture, for overlapping capability.
Whole pipeline -- Trial gate unmet (medium). Stage 1's pattern has third-party production evidence (Ippon Technologies), but no stage of this pipeline -- Stage 1, Stage 2, or the optional scoring layer -- has confirmed first-person production use yet.
Placed in Techniques / Assess / inner.
The pipeline is fully composable from existing open-source libraries: every stage has a structured input/output contract and no stage requires custom infrastructure beyond the reference identity database in Stage 1. The inner position reflects direct relevance to personal video workflows and that all required components are evaluable today without waiting on any single tool's maturity.
The ring stays at Assess rather than Trial because no first-person production run has been confirmed across the full composition. Stage 1 discovery (YOLO + DeepFace) was implemented and verified first-person in the pynanoclip project. Stage 2 editing changed shape: a first-person trial disqualified the agentic-editing tool radar/tools/FireRedOpenStoryline (its agent has no working hand-off for structured discovery data, routing prompts into speech_rough_cut and producing an empty timeline). Editing pivoted to a deterministic recipe-to-CapCut approach (radar/tools/CapcutCli, sibling playcapcut project): highlight windows and timelapse bridges compile from a versioned JSON recipe into a CapCut draft, with no agent in the loop. The pipeline's discovery-then-edit shape holds, but its editing stage is no longer agentic.
Trial gate: confirmed first-person use of discovery feeding a deterministic recipe-driven edit against real personal footage, producing a finished reel without manual timeline scrubbing at the discovery boundary.
