Framewise

Analyze video with multimodal AI, one frame at a time.

Framewise is a Node.js library that extracts chronologically ordered frames from a local video, groups them into batches, and sends them to OpenAI, Anthropic, or Google multimodal models.

Features

• Extract JPEG frames from video with FFmpeg
• Preserve natural frame order across model requests
• Limit frames and tune extraction rate and batch size
• Select OpenAI, Anthropic, or Google models through one API
• Receive typed lifecycle and progress events
• Remove generated frames automatically or keep them for inspection
• Import as a native ESM package
• Use bundled TypeScript declarations

Requirements

• Node.js 20 or newer
• FFmpeg available on your system PATH
• An API key for the selected model provider

Install FFmpeg on macOS:

brew install ffmpeg

Install FFmpeg on Ubuntu or Debian:

sudo apt update
sudo apt install ffmpeg

Verify the installation:

ffmpeg -version

Installation

npm install framewise

Framewise is an ESM package:

import { Framewise } from "framewise";

TypeScript resolves the bundled declarations automatically through the package export map.

Quick start

import { Framewise } from "framewise";

const framewise = new Framewise({
  videoPath: "./videos/demo.mp4",
  framesDirectory: "./tmp/frames",
  provider: "openai",
  apiKey: process.env.OPENAI_API_KEY,
  model: "gpt-4.1-mini",
  prompt: [
    "Analyze these frames as one continuous video.",
    "Summarize the important actions and changes in chronological order.",
  ].join(" "),
  frameRate: 1,
  batchSize: 5,
  maxFrames: 20,
});

framewise.on("progress", (progress) => {
  console.log(
    `Processed ${progress.processedFrames} of ${progress.totalFrames} frames`,
  );
});

const analysis = await framewise.analyze();

for (const batch of analysis.results) {
  console.log(batch.outputText);
}

Framewise creates framesDirectory when necessary. It clears and recreates the directory before extraction, then removes it after analysis unless keepFrames is enabled.

Configuration

Pass these options to new Framewise(options):

Option	Type	Required	Default	Description
videoPath	string	Yes	—	Path to the local video file
framesDirectory	string	Yes	—	Directory used for extracted frames
apiKey	string	Yes	—	API key for the selected provider
provider	string	Yes	"chatgpt" at runtime	Provider name or alias
prompt	string	Yes	Provider prompt when empty	Instructions used for every batch
model	string	No	"gpt-5.6"	Model name used to resolve an adapter
frameRate	number	No	1	FFmpeg frame extraction rate
batchSize	number	No	5	Frames sent in each sequential request
maxFrames	number | "all"	No	10	Maximum number of naturally sorted frames to process
keepFrames	boolean	No	false	Keep the extracted-frame directory after analysis

Set maxFrames to "all", 0, -1, or Infinity to process every extracted frame.

The extraction rate and frame limit are separate controls. A higher frameRate extracts more frames per second; maxFrames caps how many of those frames are sent to the provider.

Providers and models

Framewise resolves adapters from the model name and provider alias.

Provider	Accepted aliases	Recognized model families
OpenAI	openai, oai, chatgpt	gpt-*, o*
Anthropic	anthropic, claude	claude-*
Google	google, gemini	gemini-*

Examples:

const openaiframewise = new Framewise({
  videoPath: "./video.mp4",
  framesDirectory: "./tmp/openai-frames",
  provider: "openai",
  apiKey: process.env.OPENAI_API_KEY,
  model: "gpt-4.1-mini",
  prompt: "Summarize the video in chronological order.",
});

const claudeframewise = new Framewise({
  videoPath: "./video.mp4",
  framesDirectory: "./tmp/claude-frames",
  provider: "anthropic",
  apiKey: process.env.ANTHROPIC_API_KEY,
  model: "claude-sonnet-4",
  prompt: "Describe the important visual changes.",
});

const geminiframewise = new Framewise({
  videoPath: "./video.mp4",
  framesDirectory: "./tmp/gemini-frames",
  provider: "google",
  apiKey: process.env.GEMINI_API_KEY,
  model: "gemini-2.5-flash",
  prompt: "Identify the main actions in order.",
});

Do not commit provider API keys to source control.

Analysis result

analyze() resolves to:

interface AnalysisResult {
  startTime: number;
  completedAt: number;
  durationMs: number;
  totalFrames: number;
  totalBatches: number;
  results: Array<{
    batchIndex: number;
    frameCount: number;
    processedFrames: number;
    outputText: string;
  }>;
}

Each batch produces its own model response. Frame labels continue across batches, and requests run sequentially.

const result = await framewise.analyze();
const combinedOutput = result.results
  .map((batch) => batch.outputText)
  .join("\n\n");

Events

Attach listeners before calling analyze():

framewise.on("started", ({ startTime }) => {
  console.log("Started:", startTime);
});

framewise.on("progress", (event) => {
  console.log({
    totalFrames: event.totalFrames,
    totalBatches: event.totalBatches,
    processedFrames: event.processedFrames,
    processedBatches: event.processedBatches,
  });
});

framewise.on("completed", (result) => {
  console.log(`Completed in ${result.durationMs}ms`);
});

framewise.on("error", (error) => {
  console.error("Analysis failed:", error);
});

await framewise.analyze();

Event	Emitted when
started	Input paths have been validated and processing begins
progress	Batches are initialized and after each batch completes
completed	Every batch has been processed
error	Frame extraction or another pipeline operation fails

An error is emitted and then rethrown, so callers can use both an event listener and try/catch:

try {
  await framewise.analyze();
} catch (error) {
  console.error(error);
}

How it works

Local video
  -> extract JPEG frames with FFmpeg
  -> naturally sort and limit frames
  -> split frames into ordered batches
  -> upload or encode each batch
  -> send each batch to the selected model
  -> return per-batch output and timing data
  -> remove extracted frames unless keepFrames is true

Each image is preceded by a text label such as Frame 4 (4_frame.jpg) so frame numbers remain continuous across batches.

Development

Install dependencies:

npm install

Run the type checker:

npm run typecheck

Run the build and tests:

npm test

Build the publishable dist package:

npm run build

Preview the files included in the npm package:

npm pack --dry-run

License

ISC