Skip to content
No description, website, or topics provided.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network implemented in Keras

For more about topic check Single Image Super Resolution Using GANs — Keras

Problem Statement:

Enhancing low resolution images by applying deep network with adversarial network (Generative Adversarial Networks) 
to produce high resolutions images.


Basic Architecture

Generator and Discriminator Network:


Network Details:

* 16 Residual blocks used.
* PixelShuffler x2: This is feature map upscaling. 2 sub-pixel CNN are used in Generator.
* PRelu(Parameterized Relu): We are using PRelu in place of Relu or LeakyRelu. It introduces learn-able parameter 
  that makes it possible to adaptively learn the negative part coefficient.
* k3n64s1 this means kernel 3, channels 64 and strides 1.
* Loss Function: We are using Perceptual loss. It comprises of Content(Reconstruction) loss and Adversarial loss.

How it Works:

* We process the HR(High Resolution) images to get down-sampled LR(Low Resolution) images. Now we have both HR 
  and LR images for training data set.
* We pass LR images through Generator which up-samples and gives SR(Super Resolution) images.
* We use a discriminator to distinguish the HR images and back-propagate the GAN loss to train the discriminator
  and the generator.
* As a result of this, the generator learns to produce more and more realistic images(High Resolution images) as 
  it trains.


You can find more about this implementation in my post : Single Image Super Resolution Using GANs — Keras


You will need the following to run the above:
Python 3.5.4
tensorflow 1.11.0
keras 2.2.4
numpy 1.10.4
matplotlib, skimage, scipy
conda install tensorflow-gpu
conda install tensorflow

For training: Good GPU, I trained my model on NVIDIA Tesla P100

Data set:

* Used COCO data set 2017. It is around 18GB having images of different dimensions.
* Used 800 images for training(Very less, You can take more (approx. 350 according to original paper) thousand is you can
  collect and have very very good GPU). Preprocessing includes cropping images so that we can have same dimension images. 
  Images with same width and height are preferred. I used images of size 384 for high resolution.
* After above step you have High Resolution images. Now you have to get Low Resolution images which you can get by down 
  scaling HR images. I used down scale = 4. So Low resolution image of size 96 we will get. Sample code for this.

File Structure: : Contains Generator and Discriminator Network   : Contains utilities to process images : Contains optimizer and content loss code   : Used for training the model    : To test the model
Simplified : This folder contains code without Agrparse etc. If you hate commandline arguements just dive in here.
             There are just two files. Modify according to requirement and start training.


Note : Image shape and downscale factor you can set in file.Set according to requirement.

 * Training:
    Run below command to train model. Set parameters accordingly.
    > python --input_dir='./data/' --output_dir='./output/' --model_save_dir='./model/' --batch_size=64 --epochs=3000 --number_of_images=1000 --train_test_ratio=0.8
    > python -i='./data/' -o='./output/' -m='./model/' -b=64 -e=2 -n=100 -r=0.8
    All Parameters have default values. For mode help on parameters run:
    > python -h
 * Testing: file contains code to test. Testing can be done in two ways using option test_type:
        1. Test Model- Here you can test the model by providing HR images. It will process to get resulting LR images and then will generate SR images.
           And then will save output file comprising of all LR, SR and HR images.
           Run following command to test model:
           > python --input_high_res='./data_hr/' --output_dir='./output/' --model_dir='./model/gen_model3000.h5' --number_of_images=25 --test_type='test_model'
           For more help run:
           > python -h
           python -ihr='./data_hr/' -o='./program_output/' -m='./model/gen_model3000.h5' -n=3 -t='test_model'
           python -ihr='./data_hr/' -o='./program_output/' -m='./model/gen_model3000.h5' -n=4 -t='test_model' 
        2. Test LR images- This option directly take LR images and give resulting HR images.
           Run following command to get HR images from LR images:
           > python --input_low_res='./data_lr/' --output_dir='./output/' --model_dir='./model/gen_model3000.h5' --number_of_images=25 --test_type='test_lr_images'
           For more help run:
           > python -h
 If you hate commandline arguements please reffer Simplified folder. Modify parameters in file like image_shape, input folder
 etc. according to your need and start training.

Things's Learned:

* GAN's sometimes are hard to train. Network can be very deep sometimes, but use of residual blocks make it easier.
* Once you get to learn about Perceptual loss things get easier. Same Perceptual loss can be usefull for Image Style Transfer and Photo Realistic Style Transfer.
* This is one of the problem where i struggled to get data. You need to be carefull while choosing data and also preprossing is little bit tough.
* Better to use images with same width and height.
* Use GPU for training else it will take months to train(even you can run out of memory).


Below are few results- Output 1 Output 2 Output 2

More results are in output folder


Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network:
Perceptual Losses for Real-Time Style Transfer and Super-Resolution:

Projects doing the same thing:
Help on GANS: (Various GANS implemented in Keras)

VGG loss help:

SubpixelConv2D(Deconvolution) help:
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network:

Improved Techniques for Training GANs:
You can’t perform that action at this time.