Note
Latest Release: SAM3 object segmentation is now completely optional, allowing you to run the entire pipeline directly with just your Gemini API key (running whiteboard animations in single-pass mode).
An intelligent agentic pipeline that automates the creation of high-quality, fully narrated whiteboard animation videos from a simple text prompt.
Storyboard AI is a complete end-to-end framework. It takes in a high-level topic or context and handles everything: researching the topic, writing a compelling narrative script, planning the visual storyboard, generating custom whiteboard-style artwork, animating the drawing process, synthesizing voiceover narration, and burning perfectly timed subtitles.
It operates autonomously using an agentic approach, meaning the Director Agent breaks down the user request into manageable scenes, delegates tasks to specialized sub-agents/tools, and finally stitches everything back together.
Important
The entire demo video below was generated automatically from a single input prompt/instruction: the title itself.
storyboard_final_video.mp4
Major Pipeline Steps Executed:
- Web-search (not deepresearch): Performed web-grounded research to gather facts about Adhik Maas and Shalivahan Shaka.
- Grounded Image Generation: Generated custom whiteboard illustrations utilizing internet-grounded reference images for scene visual accuracy.
- Whiteboard Animation + Veo: Segmented objects and calculated vector sketch contours using SAM 3 for custom drawing animation, alongside Veo video generation to stitch dynamic segments.
- Gemini Native Audio TTS: Synthesized high-quality narration audio in Hindi natively.
- Full Gemini Stack: Uses the latest Gemini models for planning (LLM), custom whiteboard image prompt expansion, and narrating (TTS).
- Web Search & Deep Research Options: Supports both quick Google Search web grounding and comprehensive Deep Research agents to write highly detailed and factual scripts.
- Reference Image Grounding: Automatically searches the web for reference images of real-world entities (e.g., historical figures or landmarks) for each scene and feeds them to the image generator to maintain accurate structural accuracy.
- Multi-lingual Support: Prompts the Director, generates script narration, and creates subtitles dynamically across multiple languages (e.g., Hindi, English, Spanish).
- SAM 3 Integration: Integrates the state-of-the-art Segment Anything Model 3 to isolate object boundaries.
- Custom Animation Engine: Translates segmented object contours into fluid, custom stroke-by-stroke hand-drawn whiteboard animations.
- Veo Video Generation Integration: Optionally integrates with Veo video generation models to insert rich AI-generated video segments dynamically into planned scenes.
- Grounded & Dynamic Videos: Requires only a single text instruction or prompt to start. The Director Agent autonomously handles research, scriptwriting, scene composition, image/video generation, audio pacing, and compilation.
- Huge Cost Savings: Stretches and paces static line-art animations dynamically to match the audio narration length. For example, a 4-scene project requires only 32 seconds of total raw visual sequences (8 seconds * 4 scenes), but the animation engine stretches and times the sketch paths to create a complete, high-quality 2 min 30 sec video without expensive video-generation API calls.
Create a .env file in the genai-pipeline folder (or copy genai-pipeline/.env.example) and configure the following variables:
# Google API Key for LLM, TTS, Image Gen, and Veo Video Gen
GOOGLE_API_KEY="your-google-api-key-here"
# Hugging Face Access Token (Optional: Only required to download SAM 3 model weights if hosting SAM3 yourself)
# HF_API_KEY="your-huggingface-token-here"
# Set to TRUE if using Vertex AI, or FALSE to use Google Developer API (default)
GOOGLE_GENAI_USE_VERTEXAI=FALSEThe whiteboard drawing sequence generator can utilize instance segmentation for advanced multi-pass drawing. We host a self-contained FastAPI server that wraps the Segment Anything Model 3 (SAM 3).
- Optional Setup: If no
SAM_API_URLis provided, the pipeline will skip the segmentation phase and run the whiteboard animation in single-pass mode. This allows new users to start running the pipeline directly using just their Gemini API key. - Hosting Instructions: For complete setup instructions on obtaining weights, configuring the Docker container, and deploying to Google Cloud Run with GPU accelerators (NVIDIA L4), please refer to the detailed SAM 3 Hosting Guide in the
sam3-hosting/folder.
Core configuration parameters are set in genai-pipeline/config.py:
SAM_API_URL: Set this to your deployed SAM 3 Cloud Run endpoint (e.g.,https://sam3-service-xxxx-xx.a.run.app/predict). If left as an empty string"", SAM3 segmentation is skipped, and whiteboard animations are drawn in single-pass mode.MODEL_NAME: The model used for the Director Agent (default:gemini-2.5-pro).IMAGE_GEN_MODEL: The image generation model used for drawing line art (default:gemini-3-pro-image).VEO_MODEL: The video generation model (default:veo-3.1-generate-preview).
Set up your Python environment (Conda environment recommended) and install the verified dependencies:
- Core GenAI Pipeline: Install dependencies via the root requirements.txt.
- SAM 3 Model Server: Install dependencies via sam3-hosting/requirements.txt if not using the Docker image.
- Refer to the How to Run & View Outputs section below for installation and CLI execution steps.
Depending on the component you are running, install the appropriate requirements:
-
For the Core GenAI Pipeline: Install the main dependencies in your Python/Conda environment using the root requirements.txt:
pip install -r requirements.txt
-
For the SAM 3 Self-Hosting Server: If you are running or building the SAM 3 endpoint locally (instead of using the pre-configured Dockerfile), install the dependencies listed in sam3-hosting/requirements.txt:
pip install -r sam3-hosting/requirements.txt
# Navigate to the core agent directory
cd genai-pipeline
# Run the interactive pipeline script
python pipeline.pyThe CLI will guide you through:
- Context/Prompt: Enter the main topic for your video (e.g., "The History of Space Travel").
- Research Mode: Choose between
[1]Deep Research,[2]Web Search (Fast), or[3]None. - Reference Images: Enable or disable internet search for visual references (
Y/n). - Fast Mode: Enable parallel image/audio generation for all scenes to save time (
Y/n). - Narration Language: Enter the target language for the script (e.g.,
hindiorenglish). - Veo Video: Enable or disable Veo AI video generation (
y/N).
All intermediate assets and final outputs are saved under the genai-pipeline/output/run_<timestamp>/ folder:
storyboard_final_video.mp4: The completed, stitched whiteboard animation video with narration, background drawing paths, and burned subtitles.scene_<N>/: Individual folders for each scene containing the raw generated images, voiceover audio (.mp3), SAM 3 segmentation masks, subtitles, and scene-level sketch videos.
This release marks the official launch of Storyboard AI v1.0.0! With this launch, the agentic whiteboard generation pipeline is fully operational with complete end-to-end automation, multimodal asset generation (images, speech pacing, video stitching), and optional GPU-accelerated SAM 3 segments hosting on GCP Cloud Run.
We are actively developing new features to expand compatibility and ease of deployment:
- Broad Model Support (Beyond Gemini): Expanding language model coverage starting with Sarvam AI support.
- Standalone Mode (No SAM 3 Server Needed): Added support for running the pipeline out-of-the-box using only a Gemini API key. If no
SAM_API_URLis provided, it skips the SAM3 model server requirement and automatically runs whiteboard drawing in single-pass mode.