SFI-Swin: Symmetric Face Inpainting with Swin Transformer by Distinctly Learning Face Components Distributions

[Project page]

Official Pytorch implementation

by MohammadReza Naderi , MohammadHossein Givkashi , Nader Karimi , Shahram Shirani , Shadrokh Samavi

Implementations

• Coming Soon ...

Blockdiagram

What is the general explainations of the proposed method?
First, the generator takes the masked image as input and attempts to inpaint it. Then the inpainted image is fed to the patch discriminator to check the overall reality of the patches. Meanwhile, the inpainted image is also fed into a semantic segmentation network to separate the semantic parts of the face, such as eyes, and ears. (The architecture of these six semantic discriminators is the same). In the next step, six distinct discriminators calculate the total realness of each semantic part of the face. Then, the generator parameters will be updated based on these seven discriminators and the pixel-wise loss gradient signals, which are backpropagated to the generator.

FID and LPIPS comparison

SCS comparison

what is symmetry concentration score (SCS)?
Symmetry concentration Score is a newly introduce metric which explained compeletly in the paper to measure the symmerty of the repaired faces base on the concentration of the inpaintor.
How to calculate it?
First, we mask an eye and a K×K patch of the face, and reconstruct the missing eye and the K×K patch. Then the absolute difference between the image with inpainted eye with the image with a missing K×K patch and the missing eye is computed. The difference shows the effect of that K×K patch on inpainting result of the missing eye. The effect of all K×K patches is computed and shown as a heatmap. The face borders are also depicted to investigate the impact of each part of the face on inpainting the missed eye.

Acknowledgments

• lama.
• Swin-Unet.
• Segmentation code and models if form CSAILVision.
• LPIPS metric is from richzhang
• FID is from mseitzer

Citation

If you found this code helpful, please consider citing:

@article{naderi2023sfi,
  title={SFI-Swin: Symmetric Face Inpainting with Swin Transformer by Distinctly Learning Face Components Distributions},
  author={Naderi, MohammadReza and Givkashi, MohammadHossein and Karimi, Nader and Shirani, Shahram and Samavi, Shadrokh},
  journal={arXiv preprint arXiv:2301.03130},
  year={2023}
}