Object-aware Inversion and Reassembly for Image Editing

Zhen Yang* · Ganggui Ding* · Wen Wang* · Hao Chen* · Bohan Zhuang† · Chunhua Shen*
*Zhejiang University      †Monash University

Setup

This code was tested with Python 3.9, Pytorch 2.0.1 using pre-trained models through huggingface / diffusers. Specifically, we implemented our method over Stable Diffusion 1.4. Additional required packages are listed in the requirements file. The code was tested on a NVIDIA GeForce RTX 3090 but should work on other cards.

Getting Started

1. Download OIR-Bench.
2. Create the environment and install the dependencies by running:

conda create -n oir python=3.9
conda activate oir
pip install -r requirements.txt

3. Change the basic_config.py in configs/, change the model path and hyperparameters.
4. Modify multi_object_edit.yaml or single_object_edit.yaml in configs/ according to multi_object.yaml and single_object.yaml in OIR-Bench/.
5. Run single_object_edit.py (Search Metric in paper) or multi_object_edit.py (OIR in paper) to implement image editing.
6. Option: Adjust reassembly_step and repeat the above process to get better results.

TODO

1. Use prompt_change as dict's key may lead to error.
2. Different editing pairs' masks mustn't have overlap.
3. Search metric can be an ensemble learning tool. For example, we can use pnp, p2p, OIR ... method to edit an image and we can use search metric to select the optimal editing result.
4. We can also use the method in TODO 3 to build a high quality dataset to train instruct-based image editing method.
5. Deploy our method on different foundation model (SDXL, LCM ...)

Results

OIR results

Visualization of the search metric

Acknowlegment

Many thanks for the generous help in building the project website from Minghan Li.

Citing

If you find our work useful, please consider citing:

@article{yang2023OIR,
  title={Object-aware Inversion and Reassembly for Image Editing},
  author={Yang, Zhen and Ding, Ganggui and Wang, Wen and Chen, Hao and Zhuang, Bohan and Shen, Chunhua},
  publisher={arXiv preprint arXiv:2310.12149},
  year={2023},
}