ai-vedit

A CLI tool that turns a narrated audio script into an edited video by automatically matching your narration to categorized visual assets (images and video clips) and assembling the result with ffmpeg.

Pipeline

script.mp3 --> transcribe --> agent plans shot list --> user fills asset categories --> render (ffmpeg)

1. Input: an MP3 recording of you reading a script (audio only, no video).
2. Transcribe: the audio is sent to the OpenAI Whisper API, producing a transcript with segment/word-level timestamps.
3. Plan: an AI agent (OpenAI API) reads the timestamped transcript and breaks it into narrative "beats" (start time, end time, description). For each beat it picks a matching asset category from your existing asset library, or proposes a new category name if nothing fits.
4. Fill assets: the plan reports how much total time each category needs (e.g. "city-broll: ~42s across 6 beats") so you know roughly how many assets to gather. You add image/video files into the corresponding category folders.
5. Render: the tool picks assets from each beat's category, fits them to the beat duration, and stitches everything together with ffmpeg over the original narration audio.

Install

ffmpeg (required at runtime)

ai-vedit shells out to ffmpeg and ffprobe, so both must be on your PATH no matter how you install ai-vedit itself.

# Debian / Ubuntu
sudo apt install ffmpeg

# Fedora
sudo dnf install ffmpeg          # or ffmpeg-free from the default repos

# Arch
sudo pacman -S ffmpeg

If your distro ships an old ffmpeg (or none), grab a static build from johnvansickle.com/ffmpeg or ffmpeg.org/download.html and put ffmpeg and ffprobe somewhere on your PATH. Verify with:

ffmpeg -version && ffprobe -version

Prebuilt binary (x86_64 Linux)

Each tagged release publishes a statically linked x86_64 Linux binary that runs on any modern distro. Grab the latest from the Releases page:

ver=0.1.0   # set to the release you want
base="ai-vedit-${ver}-x86_64-unknown-linux-musl"

curl -LO "https://github.com/Jancera/ai-vedit/releases/download/v${ver}/${base}.tar.gz"
curl -LO "https://github.com/Jancera/ai-vedit/releases/download/v${ver}/${base}.tar.gz.sha256"
sha256sum -c "${base}.tar.gz.sha256"

tar -xzf "${base}.tar.gz"
install -Dm755 "${base}/ai-vedit" ~/.local/bin/ai-vedit
# make sure ~/.local/bin is on your PATH
ai-vedit --version

From source

Requires a stable Rust toolchain (rustup):

cargo install --path .          # installs `ai-vedit` onto your PATH
# or: cargo build --release     # produces target/release/ai-vedit

Quickstart

Prerequisites:

• ai-vedit installed (see Install above) and ffmpeg on your PATH.
• An OPENAI_API_KEY (used for transcription and planning, set below).

export OPENAI_API_KEY=sk-...

mkdir -p assets/general
# `ai-vedit plan` creates assets/ and a folder per planned category
# automatically if they don't exist yet -- add images/videos to
# assets/general/, or drop them into the category folders it created
# for you, before running `ai-vedit render`

ai-vedit plan --audio script.mp3
# writes plan.json, prints a time-budget report, and lists any new
# categories the plan proposes that you still need to create

# fill in any newly-proposed category folders under assets/, then:
ai-vedit render --plan plan.json --out output.mp4

See CLI usage below for the full flag reference.

Tech stack

• Language: Rust
• Transcription: OpenAI Whisper API (whisper-1, verbose_json for timestamps)
• Planning agent: OpenAI API (chat completions with structured output)
• Rendering: shells out to the ffmpeg binary

Asset library

Assets live in a plain folder structure — no manifest file required:

assets/
  city-broll/
    clip1.mp4
    photo1.jpg
  product-shots/
    shot1.png
    shot2.webp
  general/        # fallback category, used when nothing else fits
    filler1.mp4

• The folder name is the category name, discovered by scanning the directory.
• Supported asset types: images (.jpg, .jpeg, .png, .webp) and video (.mp4).
• A reserved general/ category acts as a fallback when no specific category fits a beat, or when the chosen category has no assets at all (empty or missing folder).
• Within a category, assets are drawn from a shuffled bag: the file list is shuffled into a random order, then walked in order, reshuffling once every file has been used. Every file is still used once before any file repeats, but the order (and which asset lands on which beat) differs on each render. No content-matching in the MVP.

CLI usage

plan

ai-vedit plan --audio script.mp3 [--assets ./assets] [--min-beat-duration 5]

• Transcribes the audio (transcript is cached to disk at <audio_dir>/.cache/<sha256-hash>.json to avoid re-billing).
• Produces a shot list: beats with timestamps, assigned category, and a per-category time-budget report.
• Writes a plan file (JSON) and prints the time-budget report to the terminal.
• If new categories were proposed, lists which folders need to be created before rendering.
• plan.json is written to the current working directory and is overwritten on each run (there is no --out flag for plan yet).

render

ai-vedit render --plan plan.json [--assets ./assets] [--out output.mp4] [--aspect 16:9|9:16]

• For each beat: picks an asset from its category, falling back to general/ if the category has no assets, and erroring out at that beat if general/ is also empty.
  • Images: held for the beat's duration with a Ken Burns (slow zoom/pan) effect.
  • Videos: trimmed to fit if longer than the beat, looped if shorter.
• Fitting an asset to the frame:
  • An asset larger than the output in both dimensions whose aspect ratio is within 1% of the output's is scaled down to the output resolution (no cropping).
  • Otherwise the asset is placed at its native size — excess cropped from the center, any shortfall padded with black.
• Concatenates all beat clips, overlays the original narration audio, and encodes to the target resolution.

Configuration

• OPENAI_API_KEY — required, read from the environment.
• Default output aspect ratio: 16:9 (1920x1080), overridable via --aspect.

Error handling

• A category with no assets at render time fails with a message naming the category that needs assets (unless the general/ fallback covers it).
• ffmpeg failures are surfaced with the failing command for debuggability.

Status

M0 (CLI skeleton), M1 (transcription), M2 (planning agent), M3 (asset library), M4 (render pipeline), and M5 (polish) are implemented: the plan and render subcommands parse arguments and validate config (OPENAI_API_KEY), and plan transcribes audio via the OpenAI Whisper API (caching the result locally), then segments the transcript into beats matched to asset categories via the OpenAI chat completions API, writes plan.json, and prints a per-category time-budget report. render wires up asset selection (file discovery + shuffled-bag selection with general/ fallback) with an ffmpeg rendering pipeline: it generates a full video with a Ken Burns effect for images, loop-and-trim for video clips, concatenates all beat clips, and overlays the narration audio. M5 added case/whitespace- tolerant category matching, symlink-following asset/category discovery, clearer error messages, a real end-to-end integration test, and this Quickstart. The full plan → render pipeline is functionally complete end to end, completing the MVP (M0-M5). Anything further is tracked under "Ideas beyond the MVP" in ROADMAP.md. See CONTRIBUTING.md if you'd like to help.

License

MIT