VideoAgent
title: VideoAgent type: tool created: 2026-06-22 last_updated: 2026-06-22 related: ["radar/tools/FireRedOpenStoryline", "radar/tools/EditMind", "radar/techniques/DetectionThenIdentityVideoPipeline"] sources: ["https://github.com/HKUDS/VideoAgent"] radar_quadrant: Tools radar_ring: Assess radar_position: inner
VideoAgent is an open-source agentic framework from Hong Kong University Data Science (HKUDS) for video understanding, editing, and remaking via natural language. It decomposes user instructions into sub-intents, maps them to a graph of tool-call nodes, and executes them via a self-evaluating agent loop. MIT license, 764 stars, 112 forks, 259 commits, 9 open issues. Requires Python 3.10, Conda, 8GB GPU, ffmpeg, and multiple HuggingFace model checkpoints.
Three layers operate in sequence:
- Intent Analysis -- the user's natural language command is decomposed into explicit and implicit sub-intents via Claude (required; no substitute).
- Graph-Powered Planning -- sub-intents map to an executable workflow graph. Nodes are tool capabilities; edges are execution dependencies. A two-step self-evaluation loop allows the agent to detect and correct errors mid-execution.
- Multi-Modal Understanding -- a Storyboard Agent converts the intent into visual sub-queries for retrieval from the video.
Model assignments are fixed per function:
| Function | Model |
|---|---|
| Agentic graph routing | Claude (required) |
| Editing and overview | GPT-4o |
| Fine-grained captioning | Gemini |
| TTS / voice cloning | CosyVoice, Fish-Speech, seed-vc |
| Music synthesis | DiffSinger |
| Audio/retrieval embeddings | ImageBind |
| Transcription | Whisper |
| Comedy / remixing | DeepSeek |
Movie edits, meme videos, music videos (beat-synced), rhythm edits, cross-cultural comedy adaptations, talk show commentary, news overview videos. Demos published on Bilibili and YouTube.
Both tools apply an LLM agent loop to natural language video editing. VideoAgent's strength is understanding and retrieval: it can answer questions about video content, summarise footage, and retrieve segments by semantic query. radar/tools/FireRedOpenStoryline is stronger on editing operations: ASR rough cut, AI-generated transitions, beat-synced music, and reusable Skills compatible with Claude Code. VideoAgent requires four separate LLM providers; FireRed-OpenStoryline requires one. VideoAgent has stronger voice synthesis options (CosyVoice, DiffSinger); FireRed-OpenStoryline has none. The two tools address overlapping but not identical use cases.
Full setup requires cloning the repo, creating a Conda environment, installing system dependencies (pynini 2.1.5 via conda-forge, ffmpeg, git-lfs), running pip install -r requirements.txt, downloading model checkpoints from HuggingFace for each desired output type, and configuring four API keys in config.yml. Users are advised to download only models relevant to their use case. GPU with 8GB VRAM is required; no CPU fallback is documented.
Placed in Tools / Assess / inner.
VideoAgent is the most architecturally complete open-source video agent on the radar: intent decomposition, graph-based planning, self-evaluation, multi-modal retrieval, and a broad set of output styles are all present. The inner Assess position reflects that the use case is immediately relevant and the demos confirm working functionality. The ring stays at Assess because: all content is marked "research and demonstration purposes only," no external production deployments are documented, the four-LLM-provider hard requirement adds cost and operational complexity, and the GPU + multi-checkpoint installation is substantially heavier than any other tool in this cluster.
Distinct from radar/tools/FireRedOpenStoryline, which has lower installation overhead and better composability with existing Claude Code workflows. Distinct from radar/tools/EditMind, which indexes and searches video libraries but does not perform editing.
Trial gate: a confirmed external (non-academic) production deployment using VideoAgent's graph router and at least one editing or remaking output type, with documented evidence that natural language directives produced the final output without manual intervention.
