FrameForge — AI Video Generation Studio

A locally-deployable, full-stack image-to-video generation platform. Upload an image, write a prompt, pick a model, get a video — entirely on your own GPU, with no API keys or cloud dependencies.

Models

Model	Variant	VRAM	FPS	Resolution
CogVideoX	5B I2V	~18 GB (single GPU)	8 fps	480p / 49 frames
CogVideoX	1.5-5B I2V	~22 GB (single GPU)	16 fps	768p / 81 frames
LTX-Video	13B Distilled	~20 GB (single GPU)	24 fps	480p / 121 frames
Wan2.1	14B I2V 480P	~40 GB (4 GPUs)	16 fps	480p / 81 frames
Wan2.1	14B I2V 720P	~60 GB (4 GPUs)	16 fps	720p / 81 frames

Project Structure

video-gen/
├── backend/
│   ├── main.py                        # FastAPI app + all inference functions
│   ├── requirements.txt
│   ├── CogVideoX/                     # git clone of zai-org/CogVideo (code only)
│   ├── LTX-Video/                     # git clone of Lightricks/LTX-Video (code only)
│   │   └── wan/distributed/
│   │       └── xdit_context_parallel.py   # patched version (see note below)
│   └── Wan2.1/                        # git clone of Wan-Video/Wan2.1 (code only)
├── frontend/
│   ├── src/
│   │   ├── pages/
│   │   │   ├── GeneratePage.jsx
│   │   │   └── HistoryPage.jsx
│   │   └── components/
│   │       ├── Layout.jsx
│   │       ├── ModelSelector.jsx
│   │       ├── ImageUploader.jsx
│   │       ├── JobStatus.jsx
│   │       └── AdvancedSettings.jsx
│   ├── package.json
│   └── vite.config.js
└── ckpt/                              # model weights (download separately)
    ├── CogVideoX/
    │   ├── CogVideoX-5b-I2V/
    │   └── CogVideoX1.5-5b-I2V/
    ├── LTX-Video/
    │   └── LTX-Video-0.9.8-13B-distilled/
    └── Wan2.1/
        ├── Wan2.1-I2V-14B-480P/
        └── Wan2.1-I2V-14B-720P/

Requirements

Hardware

Component	Minimum
GPU	1× GPU with ≥ 20 GB VRAM for CogVideoX / LTX-Video
GPU (Wan2.1)	4× GPU with ≥ 40 GB VRAM each (e.g. 4× L40S)
CUDA	12.1+
RAM	32 GB system RAM

Software

Python 3.10
Node.js 18+
CUDA 12.1

Installation

1 — Clone this repo

git clone https://github.com/your-username/video-gen.git
cd video-gen

2 — Clone model repos (code only, no weights)

cd backend

git clone https://github.com/zai-org/CogVideo.git CogVideoX
git clone https://github.com/Lightricks/LTX-Video.git
git clone https://github.com/Wan-Video/Wan2.1.git

3 — Set up Python environment

conda create -n videogen python=3.10 -y
conda activate videogen

# PyTorch with CUDA 12.1
pip install torch==2.4.1 torchvision==0.19.1 torchaudio==2.4.1 \
    --index-url https://download.pytorch.org/whl/cu121

# Install backend dependencies
pip install -r requirements.txt

# Install model-specific dependencies
cd LTX-Video && pip install -e ".[inference]" && cd ..
cd Wan2.1 && pip install -r requirements.txt && cd ..
pip install "xfuser>=0.4.1"

# flash_attn (required by Wan2.1's CLIP module)
pip install flash-attn==2.4.2 --no-build-isolation

Note on flash_attn: Wan2.1 requires flash_attn for its CLIP encoder. Version 2.4.2 is the tested compatible version with the patched xdit_context_parallel.py included in this repo.

4 — Apply the Wan2.1 patch

The included backend/Wan2.1/wan/distributed/xdit_context_parallel.py replaces xFuserLongContextAttention with a direct call to Wan2.1's own flash_attention. This resolves version incompatibilities between xfuser/yunchang and flash_attn while preserving FSDP model sharding across GPUs.

If you cloned Wan2.1 fresh, copy the patched file over:

cp backend/wan_patch/xdit_context_parallel.py \
   backend/Wan2.1/wan/distributed/xdit_context_parallel.py

5 — Download model weights

pip install "huggingface_hub[cli]"

# CogVideoX 5B
huggingface-cli download THUDM/CogVideoX-5b-I2V \
    --local-dir ckpt/CogVideoX/CogVideoX-5b-I2V

# CogVideoX 1.5-5B
huggingface-cli download THUDM/CogVideoX1.5-5b-I2V \
    --local-dir ckpt/CogVideoX/CogVideoX1.5-5b-I2V

# LTX-Video 13B Distilled (Diffusers format)
huggingface-cli download Lightricks/LTX-Video-0.9.8-13B-distilled \
    --local-dir ckpt/LTX-Video/LTX-Video-0.9.8-13B-distilled

# Wan2.1 480P
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-480P \
    --local-dir ckpt/Wan2.1/Wan2.1-I2V-14B-480P

# Wan2.1 720P (optional)
huggingface-cli download Wan-AI/Wan2.1-I2V-14B-720P \
    --local-dir ckpt/Wan2.1/Wan2.1-I2V-14B-720P

Disk space: CogVideoX ~18 GB · LTX-Video ~26 GB · Wan2.1 480P ~28 GB · Wan2.1 720P ~28 GB

6 — Set up frontend

cd frontend
npm install

Configuration

Open backend/main.py and update the paths and GPU config at the top of the file:

# ── Checkpoint base path ──────────────────────────────────────────────────
CKPT_BASE = "/home/zyh/workspace/video-gen/ckpt"   # ← change to your path

# ── Wan2.1 multi-GPU config ───────────────────────────────────────────────
WAN_NUM_GPUS     = 4           # number of GPUs to use for Wan2.1
WAN_ULYSSES      = 4           # must equal WAN_NUM_GPUS and divide num_heads (40)
WAN_RING         = 1           # keep 1 when using Ulysses strategy
WAN_CUDA_DEVICES = "0,1,2,3"  # GPU indices to use (skip occupied GPUs here)

Choosing WAN_NUM_GPUS and WAN_ULYSSES:

The 14B model has 40 attention heads. Valid values where 40 % N == 0:

N (GPUs)	Works?
1	✓ (single GPU, needs offload)
2	✓
4	✓ (recommended)
5	✓
8	✓
3, 6, 7	✗ (40 not divisible)

Running

Terminal 1 — Backend

cd backend
conda activate videogen
python main.py

Expected output:

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Verify it's healthy:

curl http://localhost:8000/api/health

Terminal 2 — Frontend

cd frontend
npm run dev

Expected output:

  ➜  Local:   http://localhost:3000/

Access the UI

If running on a remote server, set up an SSH tunnel from your local machine:

ssh -L 3000:localhost:3000 -L 8000:localhost:8000 username@server-ip

Then open http://localhost:3000 in your local browser.

If running locally, open http://localhost:3000 directly.

Usage

Select a model from the left panel — each card shows FPS, frame count, and resolution
Upload an image — drag and drop or click to browse (PNG/JPG/WEBP)
Write a prompt describing the desired motion
Adjust settings (optional) — expand Advanced Settings to change inference steps, guidance scale, and seed
Click Generate Video — the job is submitted and the right panel shows live progress
Watch and download — when complete, the video plays automatically; click Download to save

To cancel a running generation, click Cancel Generation and submit a new job immediately.

Demo Walkthrough

The following is a representative end-to-end example using CogVideoX 5B.

Step 1 — Select a model and upload an image

The left panel lists all available models grouped by family. Each card shows the model's key specs (FPS, frame count, resolution) so you can choose the right tradeoff between quality and speed. In this example, CogVideoX 5B is selected. The input is a personal ID-style photo.

Step 2 — Enter a prompt and adjust generation parameters

After uploading the image, enter a text prompt describing the desired motion. Here the prompt is: "Generate a video of this person playing the guitar." Expanding the Advanced Settings panel reveals three controls:

Inference Steps — more steps give higher quality at the cost of generation time (default: 50)
Guidance Scale — higher values make the output follow the prompt more strictly (default: 6)
Seed — fix a value to reproduce the same result across runs (default: 42)

Step 3 — Monitor generation progress

Clicking Generate Video submits the job immediately and the right panel switches to the generation view. A progress bar and spinner provide real-time feedback while the model synthesizes frames on the GPU. Generation typically takes several minutes depending on the model and step count.

![Step 3 — Generation in progress]

Step 4 — View and download the result

When generation completes, the video plays automatically in the browser. A Download button in the bottom-right corner of the player saves the .mp4 file locally. The prompt and generation time are shown below the player for reference.

In this example, the model successfully animates the static portrait into a video of the person playing guitar — demonstrating the model's ability to synthesize plausible motion from a single image and a short text description.

API Reference

Method	Endpoint	Description
`GET`	`/api/health`	GPU status and available models
`POST`	`/api/generate`	Submit a generation job
`GET`	`/api/jobs/{id}`	Poll job status and result URL
`POST`	`/api/jobs/{id}/cancel`	Cancel a running job
`GET`	`/api/jobs`	List all jobs (newest first)
`DELETE`	`/api/jobs/{id}`	Delete a job and its output files

POST /api/generate (multipart/form-data)

Field	Type	Required	Description
`image`	file	✓	Input image (PNG/JPG/WEBP)
`prompt`	string	✓	Motion description
`model`	string	✓	One of: `cogvideox-5b`, `cogvideox-1.5-5b`, `ltx-13b-distilled`, `wan-480p`, `wan-720p`
`num_inference_steps`	int	—	Diffusion steps (default: 50)
`seed`	int	—	Random seed (default: 42)

Troubleshooting

CUDA out of memory on Wan2.1

Check if other processes are using GPU memory:

nvidia-smi

Kill any blocking processes and retry. If you have fewer GPUs available, adjust WAN_NUM_GPUS, WAN_ULYSSES, and WAN_CUDA_DEVICES accordingly.

assert FLASH_ATTN_2_AVAILABLE on Wan2.1

Wan2.1's CLIP encoder requires flash_attn. Install it:

pip install flash-attn==2.4.2 --no-build-isolation

_flash_attn_forward() got an unexpected keyword argument 'softcap'

This means the patched xdit_context_parallel.py was not applied. Copy it over:

cp backend/wan_patch/xdit_context_parallel.py \
   backend/Wan2.1/wan/distributed/xdit_context_parallel.py

cannot import name 'AutoencoderKLWan' from 'diffusers'

Upgrade diffusers and peft:

pip install -U diffusers peft

Frontend shows connecting… in the GPU status card

The backend is not running or not reachable on port 8000. Start the backend first, then refresh the frontend.

SSH tunnel drops and the UI becomes unresponsive

Reconnect the tunnel. The backend continues running on the server — any jobs that were in progress will complete and results will be available when you reconnect.

Technical Stack

Layer	Technology
Frontend	React 18, Vite 5, React Router 6
Backend	FastAPI 0.115, Uvicorn, Python 3.10
ML Framework	PyTorch 2.4.1 + CUDA 12.1
Diffusers	Hugging Face diffusers 0.33.0
CogVideoX	`CogVideoXImageToVideoPipeline` (diffusers)
LTX-Video	`LTXImageToVideoPipeline` (diffusers, native format)
Wan2.1	Official `generate.py` + `torchrun` + FSDP
Multi-GPU	PyTorch FSDP + xDiT Ulysses sequence parallelism
Concurrency	Python `threading.Thread`, in-memory job store

License

Apache 2.0 — see LICENSE for details.

Model weights are subject to their respective licenses:

CogVideoX: CogVideoX Model License
LTX-Video: LTX-Video Open Weights License
Wan2.1: Apache 2.0

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
backend		backend
docs		docs
frontend		frontend
README.md		README.md

Navigation Menu

Repository files navigation

FrameForge — AI Video Generation Studio

Models

Project Structure

Requirements

Hardware

Software

Installation

1 — Clone this repo

2 — Clone model repos (code only, no weights)

3 — Set up Python environment

4 — Apply the Wan2.1 patch

5 — Download model weights

6 — Set up frontend

Configuration

Running

Terminal 1 — Backend

Terminal 2 — Frontend

Access the UI

Usage

Demo Walkthrough

API Reference

POST /api/generate (multipart/form-data)

Troubleshooting

Technical Stack

License

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

FrameForge — AI Video Generation Studio

Models

Project Structure

Requirements

Hardware

Software

Installation

1 — Clone this repo

2 — Clone model repos (code only, no weights)

3 — Set up Python environment

4 — Apply the Wan2.1 patch

5 — Download model weights

6 — Set up frontend

Configuration

Running

Terminal 1 — Backend

Terminal 2 — Frontend

Access the UI

Usage

Demo Walkthrough

API Reference

POST /api/generate (multipart/form-data)

Troubleshooting

Technical Stack

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages