Skip to content
Generative adversarial networks integrating modules from FUNIT and SPADE for face-swapping.
Jupyter Notebook Python
Branch: master
Clone or download
shaoanlu Fix import error
There are some API changes in Keras 2.2.5 that causes import errors. We fix this error by installing Keras 2.2.4.
Latest commit edcb090 Sep 2, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
face_toolbox_keras @ fb472da Add face_toolbox_keras as submodule. Jun 3, 2019
images Delete images Jul 14, 2019
networks Upload networksand utils Jun 2, 2019
utils Upload networksand utils Jun 2, 2019
.gitignore Update architecture descriptions May 27, 2019
.gitmodules Add face_toolbox_keras as submodule. Jun 3, 2019
README.md Update README.md Jul 4, 2019
colab_demo.ipynb Fix import error Sep 2, 2019
models.py Upload models.py Jun 2, 2019

README.md

Few-shot face translation Source face: Mona Lisa

A GAN based approach for one model to swap them all.

The table below shows our priliminary face-swapping results requiring one source face and <=5 target face photos. Notice that almost all of the identities, except Stephen Curry, are not in our training data (which is a subset of VGGFace2). More translation results can be found here.

Also, our model is capable of producing faces that has its gaze direction, glasses, and hiar occlusions being consistent with given source face. However, our model has suboptimal performance in terms of translating to asian faces. This is possibly due to limited representability of the feature extractor.

Src.\Tar. Andrej Karpathy Andrew Y. Ng Du Fu Elon Musk Emilia Clarke Geoffrey Hinton Stephen Curry Yann Lecun Yoshua Benjio
Andrej
Karpathy
N/A
Andrew
Y. Ng
N/A
Du Fu N/A
Elon
Musk
N/A
Emilia
Clarke
N/A
Geoffrey
Hinton
N/A
Stephen
Curry
N/A
Yann
Lecun
N/A
Yoshua
Benjio
N/A

I really like the Du Fu translation: such an interesting demostration how the GAN imagine the appearance of the prominent Chinese poet from just a painting.

Try in Google Colab Open In Colab

We only provide pre-trained weights and inference script for now. Training script will be released after code cleanup.

Architecture

The above image illustrates our generator, which is a encoder-decoder based network, at test phase. Our swap-them-all approach is basically a GAN conditioned on the latent embeddings extracted from a pre-trained face recognition model. SPADE and AdaIN modules are incorporated in order to inject semantic priors to the networks.

During training phase, the input face A is heavily blurred and we train the model with resonctruction loss. Other objectives that aimed to improve translation performance while keeping semantic consistency, such as perceptual loss on rgb output and cosine similarity loss on latent embeddings, are also introduced.

Things that didn't work

  1. We tried to distort (spline warp, downsample) the input image as in faceswap-GAN instead of masking it. However, the model did not learn proper identity translation but output face that is similar to its input.

Requirements

  • Python 3.6
  • Keras 2.2.4
  • TensorFlow 1.12.0 or 1.13.1

References

  1. Semantic Image Synthesis with Spatially-Adaptive Normalization
  2. Few-Shot Unsupervised Image-to-Image Translation
  3. DEEP LEARNING FOR FASHION AND FORENSICS
You can’t perform that action at this time.