ReCo

🖥️ GitHub    ｜    🌐 Project Page    |   🤗 ReCo-Data   |    📈 ReCo-Bench   |    🤗 ReCo-Models    |    📖 Paper   

ReCo: Region-Constraint In-Context Generation for Instructional Video Editing

🔆 If you find ReCo useful, please give a ⭐ for this repo, which is important to Open-Source projects. Thanks!

Here, we will gradually release the following resources, including:

• ReCo training dataset: ReCo-Data
• Evaluation code: ReCo-Bench
• Model weights, inference code, and training code

Video Demos

video_edit_demo.mp4

Examples of different video editing tasks by our ReCo.

📢 News!!!

• 2026.04.25: Updates released today:

  • ReCo_ref: Our ReCo architecture naturally supports IP-reference-conditioned video editing. We further trained a multi-task IP-conditioned editing model with additional Kiwi-Edit data.

    • Released training code for ReCo_ref, including a mixed dataloader with Kiwi-Edit data.
    • Released inference code for ReCo_ref.
    • Released ReCo_ref inference videos, intermediate outputs, and final results on ReCo-Bench, RefViE-Bench, and OpenVE-Bench. See ReCo-Bench/ReCo_Ref_results_md/

  • ReCo_ori: We released two variants, 2025_m12 with stronger overall performance and 2026_01_16_v1 with improved removal performance.

    • Released ReCo_ori inference videos, intermediate outputs, and final results on ReCo-Bench.
    • Released a Diffusers-based implementation script for ReCo_ori supporting 30-step sampling; note that it uses direct parameter conversion and may incur some quality loss. See tools/run_reco_diffusers.sh.
    • Added region-loss training code. See tools\train_reco_add_region_loss_raw.py.

• 2026.03.05: We are excited to see that Kiwi-Edit (NUS) has further refined our HQ-ReCo dataset and added reference image pairs. Check out their DATASET.md for further instructions.

🔥 Updates

• [2026.02.26] Release training code.
• [2026.01.16] Release ReCo Model weights and inference code.
• [2026.01.16] Uploaded raw video object masks to ReCo-Data.
• [2025.12.23] Release ReCo-Data and Usage code.
• [2025.12.23] Release ReCo-Bench and evaluation code.
• [2025.12.22] Upload Our arXiv Paper.

📊 ReCo-Data Preparation

ReCo-Data is a large-scale, high-quality video editing dataset consisting of 500K+ instruction–video pairs, covering four video editing tasks: object addition (add), object removal (remove), object replacement (replace), and video stylization (style).

Downloading ReCo-Data

Please download each task of ReCo-Data into the ./ReCo-Data directory by running:

bash ./tools/download_dataset.sh

Before downloading the full dataset, you may first browse the visualization examples.

These examples are collected by randomly sampling 50 instances from each task (add, remove, replace, and style), without any manual curation or cherry-picking, and are intended to help users quickly inspect and assess the overall data quality.

Note: The examples are formatted for visualization convenience and do not strictly follow the dataset format.

Directory Structure

After downloading, please ensure that the dataset follows the directory structure below:

ReCo-Data directory structure

ReCo-Data/
├── add/
│   ├── add_data_configs.json
│   ├── src_videos/
│   │   ├── video1.mp4
│   │   ├── video2.mp4
│   │   └── ...
│   └── tar_videos/
│       ├── video1.mp4
│       ├── video2.mp4
│       └── ...
├── remove/
│   ├── remove_data_configs.json
│   ├── src_videos/
│   └── tar_videos/
├── replace/
│   ├── replace_data_configs.json
│   ├── src_videos/
│   └── tar_videos/
└── style/
    ├── style_data_configs.json
    ├── src_videos/
    │   ├── video1.mp4
    │   └── ...
    └── tar_videos/
        ├── video1-a_Van_Gogh_style.mp4
        └── ...

Testing and Visualization

After downloading the dataset, you can directly test and visualize samples from any single task using the following script (taking the replace task as an example):

python reco_data_test_single.py \
  --json_path ./ReCo-Data/replace/replace_data_configs.json \
  --video_folder ./ReCo-Data \
  --debug

Mixed Task Loading

You can also load a mixed dataset composed of the four tasks (add, remove, replace, and style) with arbitrary ratios by running:

python reco_data_test_mix_data.py \
  --json_folder ./ReCo-Data \
  --video_folder ./ReCo-Data \
  --debug

Notes

• src_videos/ contains the original source videos.
• tar_videos/ contains the edited target videos corresponding to each instruction.
• *_data_configs.json stores the instruction–video mappings and metadata for each task.

📈 Evaluation

VLLM-based Evaluation Benchmark

ReCo-Bench details

Traditional video generation metrics often struggle to accurately assess the fidelity and quality of video editing results. Inspired by recent image editing evaluation protocols, we propose a VLLM-based evaluation benchmark to comprehensively and effectively evaluate video editing quality.

We collect 480 video–instruction pairs as the evaluation set, evenly distributed across four tasks: object addition, object removal, object replacement, and video stylization (120 pairs per task). All source videos are collected from the Pexels video platform.

For local editing tasks (add, remove, and replace), we utilize Gemini-2.5-Flash-Thinking to automatically generate diverse editing instructions conditioned on video content. For video stylization, we randomly select 10 source videos and apply 12 distinct styles to each, resulting in 120 stylization evaluation pairs.

---

1. Downloading ReCo-Bench

Please download ReCo-Bench into the ./ReCo-Bench directory by running:

bash ./tools/download_ReCo-Bench.sh

---

2. Usage

After downloading the benchmark, you can directly start the evaluation using:

cd tools
bash run_eval_via_gemini.sh

This script performs the evaluation in two stages:

Step 1: Per-dimension Evaluation with Gemini

In the first stage, Gemini-2.5-Flash-Thinking is used as a VLLM evaluator to score each edited video across multiple evaluation dimensions.

Key arguments used in this step include:

• --edited_video_folder: Path to the folder containing the edited (target) videos generated by the model.

• --src_video_folder: Path to the folder containing the original source videos.

• --base_txt_folder: Path to the folder containing task-specific instruction configuration files.

• --task_name: Name of the evaluation task, one of {add, remove, replace, style}.

This step outputs per-video, per-dimension evaluation results in JSON format.

Step 2: Final Score Aggregation

After all four tasks have been fully evaluated, the second stage aggregates the evaluation results and computes the final scores.

• --json_folder: Path to the JSON output folder generated in Step 1

  (default: all_results/gemini_results)

• --base_txt_folder: Path to the instruction configuration folder

This step produces the final benchmark scores for each task as well as the overall performance.

3. Benchmark Results (Downloads and Summaries)

We provide downloadable evaluation outputs for different model releases. Summary tables/markdown files are stored in this repository, while full result packages are hosted on Hugging Face.

Model	Release	Benchmarks	All results (download)	Summary tables
ReCo_ori	2025-12	ReCo-Bench	Hugging Face	The ReCo paper
ReCo_ref	2026-04	RefViE-bench, OpenVE-bench, ReCo-Bench	Hugging Face	ReCo-Bench/ReCo_Ref_results_md/

🏃 Inference

1. Environment Preparation

Create and activate the specialized Conda environment:

conda create -n reco python=3.11 -y
conda activate reco
pip install -r requirements.txt

2. Model Weights Setup

You need to prepare both the base model and our specific checkpoints.

Model	Source	Description
Wan-2.1-VACE-1.3B	🤗 Hugging Face	Base VACE weights. Place in ./Wan-AI
ReCo_ori	🤗 Hugging Face	Our original ReCo checkpoint trained on the four editing tasks. Place in all_ckpts/.
ReCo_ref	🤗 Hugging Face	Our multi-task editing checkpoint with IP-reference conditioning, additionally fine-tuned on Kiwi-Edit data. Supports IP-guided background replacement, object replacement, and object addition. Place in all_ckpts/.

Organize the files as follows:

.
├── Wan-AI/                      
├── all_ckpts/                   
│   └── ReCo_ori_rank128-2025_m12_version.ckpt
|   |__ ReCo_ref_rank256-2026_m4_version.ckpt
├── assets/                      
└── inference_reco_single.py

3. Running Inference

We provide a bash script to automate the execution of different tasks (Replace, Remove, Style, Add and Propagation). Run the following command:

bash infer_server_single.sh

To run a specific task manually or customize the execution, use the python command directly:

python inference_reco_single.py \
    --task_name replace \
    --test_txt_file_name assets/replace_test.txt \
    --lora_ckpt ReCo_ori_rank128-2025_m12_version.ckpt

Argument	Type	Default	Description
test_txt_file_name	str	assets/...	Path to the .txt file containing test prompts/configs.
task_name	str	replace	Task type: remove, replace, add, style. Use the _wf suffix (e.g., remove_wf) for Propagation tasks given the first frame.
base_video_folder	str	assets/test_videos	Directory containing the source videos.
base_wan_folder	str	./Wan-AI	Path to the pre-trained Wan-AI model weights.
lora_ckpt	str	all_ckpts/...	Path to the specific LoRA checkpoint file.

4. Running Inference with IP condition

Run the IP-conditioned inference script:

bash infer_server_single_ref_rank256.sh

This script calls inference_reco_single_ref.py and demonstrates different modes: prompt-only, IP-image-only, first-frame-only, or using both IP image and first-frame conditioning together.

🚀 Training

1) Basic Training

Run:

bash train.sh

Before launching training:

• Update the pretrained model weight paths in your script to match local paths.
• In train.py, update dataset paths in LightningModelForTrain.train_dataloader:
  • JSON annotation directory
  • Video data directory

2) Multi-task Training with IP Reference Data

We additionally provide multi-task training code with IP-image references, which additionally supports:

• background replacement with a given reference image
• object replacement with a given reference image
• object addition with a given reference image

To start this training pipeline, follow two steps:

Step 1. Prepare data configs and local paths

1. Download task config/data package from
   kiwidata.zip,
   then place/extract it under the current project.
2. Update related configs in:
   • train_multitask_add_kiwi_ref_data.py (around 196-210)
   • kiwidata/test_dataset_mixdata.py (around 32-54)

Step 2. Launch training

bash train_multitask_add_kiwi_ref_data_2node.sh

This pipeline includes mixed data loading from ReCo-Data, DiTTO, and OpenVE-3M, and also integrates kiwi-edit paired IP-reference data. The kiwidata folder provides our filtered and organized config files that better match the original dataset formats. Feel free to use and adapt them.

🌟 Star and Citation

If you find our work helpful for your research, please consider giving a star⭐ on this repository and citing our work.

@article{zhang2025region,
  title={Region-Constraint In-Context Generation for Instructional Video Editing},
  author={Zhang, Zhongwei and Long, Fuchen and Li, Wei and Qiu, Zhaofan and Liu, Wu and Yao, Ting and Mei, Tao},
  journal={arXiv preprint arXiv:2512.17650},
  year={2025}
}

💖 Acknowledgement

Our code is inspired by several works, including WAN, ObjectClear--a strong object remover, VACE, Flux-Kontext-dev. Thanks to all the contributors!