TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing (CVPR 2024)

[Arxiv] [Poster] [Youtube]

TiNO-Edit is an image editing algorithm built on top of Stable Diffusion (SD) by optimizing noise and timesteps in the SD latent space.

Capabilities

LatentCLIPvis & LatentVGG

LatentCLIPvis and LatentVGG are the SD latent space equivalents of the CLIP vision model and VGG. To train these models, see latentclip and latentvgg respectively. We will provide an explanation of our code soon. We also provide pretrained checkpoints here.

Method

For pure text-guided image editing, run

python app_text_guided_image_editing.py \
  --original_image_path <original_image_path> \
  --original_prompt <original_prompt> \
  --target_prompt <target_prompt>

See example in app_text_guided_image_editing.py, which enables face swap from the left to the right image below:

For stroke-guided image editing or image composition, run

python app_stroke_guided_image_editing.py \
  --original_image_path <original_image_path> \
  --user_input_image_path <user_input_image_path> \
  --original_prompt <original_prompt> \
  --target_prompt <target_prompt>

See example in app_stroke_guided_image_editing.py, which convert user input stroke as follows:

BibTeX

@inproceedings{chen2024tino,
  title={TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing},
  author={Chen, Sherry X and Vaxman, Yaron and Ben Baruch, Elad and Asulin, David and Moreshet, Aviad and Lien, Kuo-Chin and Sra, Misha and Sen, Pradeep},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={6337--6346},
  year={2024}
}