Generative Adversarial Network Colorizer

Colorization of large domain greyscaled images with GAN and transfer learning

The purpose of this experimentation is to test some basic GANs architectures applied to the problem of greyscaled images colorization and merge different approaches in order to obtain the colorization on a large domain of different images. We tested different the different implementations on the place365 dataset wich offers a good start point in terms of variety.

The tested networks are:

• Pix2pix model starting from this nice implementation. This network is a conditional GAN that uses a U-NET generator and a classic convolutional discriminator. This is our adapted implementation.

• Encoder-decoder model with a fusion layer starting from this Keras implementation. The fusion layer is built extracting high-level features from Inception-ResNet-v2 pre-trained on ImageNet dataset. Instead of using Inception-ResNET-v2 we tested the Emil Wallner implementation with MobileNet wich is lighter on Emil advice (thanks, Emil!). The strenght of this implementation is that we can do transfer learning with few samples, because the fusion layer is pre-trained. This

• Finally we implemented our own model wich combines the previous ones: we used the encoder-decoder model with the fusion layer as generator and the pix2pix discriminator. We wanted to use the MobileNet knowledge trained on ImageNet to colorize greyscaled photos and make them better throught the Pix2pix's LSGAN discriminator wich take in input the real and the generated photos, combines discriminates how much the generated image is good.

We conducted on this model different type of preliminary trainings.

We tested the trained models on 50 images taken from the place365 dataset not used during the traininig phase, evaluating the results with a direct turing test. The preliminary results shows that the models are able to recognize and to colorize skies and vegetation. With a longer training we should appreciate improvements.

Some results with different training and model conditions

In the images below, we can see the different preliminary experiments on the architectures above with different training parameters. Starting from left :

• Our model with MAE for the generator on 9000 images from place365 (100 epochs)
• Our model with the above conditions on the fullset trained (10 epocs)
• Our model with MSE for the generator and different Adam optimizer for discriminator on 9000 images from place365 (50 epochs)
• Our model with MSE for the generator on 9000 images from place365 (100 epochs)
• Our model with MSE for the generator on just 40 samples (17000 epochs)
• pix2pix on the full place365
• Ground truth

Next steps

Next, we're going to train the above models with different datasets and different hyperparameters. We are considering to starting the GAN training with a pretrained generator, like suggested in the Emil Wallner GAN implementation and in Jason Antic and his DeOldify coloring network.

Implementation details:

• We used a 11GB GPU (GTX 1080TI); the training on 10 epochs on our models required 11 hours.
• We also tried to train the models on a configuration with 4 TeslaP100 working in parallel, but this didn't improve the training time probably due to a functional bottleneck in the GAN train implementation.