Install

Use the following commands to build a specific environment of Edit4D:

conda create -n Edit4D python=3.10
conda activate Edit4D

git clone https://github.com/DINGYANB/Edit4D.git

cd Edit4D
pip install -r requirement.txt

Usage

Download models for image/video editing:

• Image editing by StyleBooth: the ckpts will be automated download via ModelScope after rununing the main script.

• Video editing by FRESCO: use python FRESCO_src/install.py to download the corresponding models weights.

To run Edit4D on a single input video of 21 frames:

• Download SV3D models (sv3d_u.safetensors or sv3d_p.safetensors) from here and SV4D model (sv4d.safetensors) from here.

• Run the following command, where input_path is the input video path, and edit_prompt is the prompt used for editing.

python scripts/sampling/simple_video_sample_4d.py --input_path assets/sv4d_videos/test_video1.mp4 --edit_prompt "Apply the style of digital pixel art to this image"

Other parameters includes:

• num_steps : default is 20, can increase to 50 for better quality but longer sampling time.

• sv3d_version : To specify the SV3D model to generate reference multi-views, set --sv3d_version=sv3d_u for SV3D_u or --sv3d_version=sv3d_p for SV3D_p.

• elevations_deg : To generate novel-view videos at a specified elevation (default elevation is 10) using SV3D_p (default is SV3D_u), e.g., set --elevations_deg 30.0.

• Background removal : For input videos with plain background, (optionally) use rembg to remove background and crop video frames by setting --remove_bg=True. To obtain higher quality outputs on real-world input videos with noisy background, try segmenting the foreground object using Clipdrop or SAM2 before running Edit4D.

• Low VRAM environment : To run on GPUs with low VRAM, try setting --encoding_t=1 (of frames encoded at a time) and --decoding_t=1 (of frames decoded at a time) or lower video resolution like --img_size=512.