PyTorch implementation of SDEdit: Image Synthesis and Editing with Stochastic Differential Equations.
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon
Stanford and CMU
The key intuition of SDEdit is to "hijack" the reverse stochastic process of SDE-based generative models, as illustrated in the figure below. Given an input image for editing, such as a stroke painting or an image with color strokes, we can add a suitable amount of noise to make its artifacts undetectable, while still preserving the overall structure of the image. We then initialize the reverse SDE with this noisy input, and simulate the reverse process to obtain a denoised image of high quality. The final output is realistic while resembling the overall image structure of the input.
The code will automatically download pretrained SDE (VP) PyTorch models on CelebA-HQ, LSUN bedroom, and LSUN church outdoor.
We save the image and the corresponding mask in an array format [image, mask], where
"image" is the image with range [0,1] in the PyTorch tensor format, "mask" is the corresponding binary mask (also the PyTorch tensor format) specifying the editing region.
We provide a few examples, and functions/process_data.py will automatically download the examples to the colab_demo folder.
Given an input stroke painting, our goal is to generate a realistic image that shares the same structure as the input painting. SDEdit can synthesize multiple diverse outputs for each input on LSUN bedroom, LSUN church and CelebA-HQ datasets.
To generate results on LSUN datasets, please run
python main.py --exp ./runs/ --config bedroom.yml --sample -i images --npy_name lsun_bedroom1 --sample_step 3 --t 500 --ni
python main.py --exp ./runs/ --config church.yml --sample -i images --npy_name lsun_church --sample_step 3 --t 500 --ni
Given an input image with user strokes, we want to manipulate a natural input image based on the user's edit. SDEdit can generate image edits that are both realistic and faithful (to the user edit), while avoid introducing undesired changes.
python main.py --exp ./runs/ --config church.yml --sample -i images --npy_name lsun_edit --sample_step 3 --t 500 --ni
If you find this repository useful for your research, please cite the following work.
@article{meng2021sdedit,
title={SDEdit: Image Synthesis and Editing with Stochastic Differential Equations},
author={Chenlin Meng and Yang Song and Jiaming Song and Jiajun Wu and Jun-Yan Zhu and Stefano Ermon},
year={2021},
journal={arXiv preprint arXiv:2108.01073},
}
This implementation is based on / inspired by: