Eye-for-an-eye: Appearance Transfer with Semantic Correspondence in Diffusion Models

arXiv | Project Page

Eye-for-an-eye: Appearance Transfer with Semantic Correspondence in Diffusion Models
Sooyeon Go, Kyungmook Choi, Minjung Shin, Youngjung Uh

Abstract:
As pretrained text-to-image diffusion models have become a useful tool for image synthesis, people want to specify the results in various ways. In this paper, we introduce a method to produce results with the same structure of a target image but painted with colors from a reference image, especially following the semantic correspondence between the result and the reference. E.g., the result wing takes color from the reference wing, not the reference head. Existing methods rely on the query-key similarity within self-attention layer, usually producing defective results. To this end, we propose to find semantic correspondences and explicitly rearrange the features according to the semantic correspondences. Extensive experiments show the superiority of our method in various aspects: preserving the structure of the target and reflecting the color from the reference according to the semantic correspondences, even when the two images are not aligned.

Code coming Soon!

Citation

If you use this code for your research, please cite the following work:

@misc{go2024eyeforaneye,
      title={Eye-for-an-eye: Appearance Transfer with Semantic Correspondence in Diffusion Models}, 
      author={Sooyeon Go and Kyungmook Choi and Minjung Shin and Youngjung Uh},
      year={2024},
      eprint={2406.07008},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}