CycleFlow-Inpaint is a research prototype for flow-guided video inpainting with cycle-consistent temporal optimization. It allows users to upload a video, draw a mask over an object, optionally enter a prompt, and remove or replace the object.
Removing or replacing objects in videos often leads to temporal flickering and inconsistencies. This project implements a cycle-consistent self-supervised optimization approach. By enforcing that propagating an inpainted frame forward and backward should match the original inpainted frame, we drastically reduce temporal flickering without requiring large 3D models.
graph TD
A[Video] --> B[Frame Extraction]
B --> C[Optical Flow RAFT]
C --> D[Flow-guided Propagation]
D --> E[Diffusion Inpainting]
E --> F[Cycle-consistency Optimization]
F --> G[Temporal Smoothing]
G --> H[Final Video]
- Frame Extraction: Extract frames from the input video.
- Optical Flow: Use RAFT to compute forward and backward optical fields between consecutive frames.
- Flow-guided Propagation: Propagate masks and context to align sequences.
- Diffusion Inpainting: Apply Stable Diffusion 1.5 tailored for preserving original content and seamlessly painting masked regions.
- Cycle Optimization: A localized optimization loop modifies only masked pixels to ensure temporal consistency (backward(forward(frame)) ≈ frame).
- Smoothing: A temporal median filter reduces remaining noise.
The backend automatically detects the best hardware and adapts its configuration:
- CUDA: 512px resolution, 15 optimization steps
- MPS (Apple Silicon): 512px resolution, 10 optimization steps
- CPU: 384px resolution, 5 optimization steps
# Clone the repository
git clone https://github.com/yourusername/cycleflow-inpaint.git
cd cycleflow-inpaint
# Run the setup script to install dependencies
bash scripts/setup.sh- Start the backend:
python scripts/start_backend.py- Start the frontend:
cd frontend
npm run dev- Open
http://localhost:3000to interact with the web interface.