Adaptive-threshold-based Multi-model Fusion Network for Compressed Face Image Hallucination (ATMFN)

This is an implementation of the SRGAN model proposed in the paper (Adaptive-threshold-based Multi-model Fusion Network for Compressed Face Image Hallucination) with TensorFlow.

Requirements

• Python 3
• TensorFlow 1.12.0
• OpenCV
• tqdm
• glob
• sys

Usage

I. Pretrain the VGG-19 model

Download the ImageNet dataset and preprocess them with:

$ cd vgg19/imagenet
$ python get_urls.py
$ python create_db.py
$ python download_images.py
$ python preprocess.py

Train with:

$ cd vgg19
$ python train.py

Or you can download the pretrained model file: vgg19_model.tar.gz

II. Train the ATMFN (Generator and Discriminator) model

Dataset Organization Form

If you prepare your own dataset, perform the compress.m to generate the compressed samples with special quality ratios. Then put the data into the data file with the following form:

|--train_data

|--compressed samples  
    |--file1
            ：  
    |--file2
        :
    |--filen
    
|--HR samples
    |--file1
            ：  
    |--file2
        :
    |--filen

Then you can produce the corresponding '.npy' in the '/lfw/data/npy' file.

$ python lfw.py

Training

Download training dataset ((raw images)Baidu Cloud, (Password:vd57) (.npy)Baidu Cloud, (Password:tg0z)), or prepare your own dataset like above form.

Run the following commands:

$ cd /src
$ python train_ATMFN.py

The evaluated result will be stored in "src/result".

II. Test the ATMFN model

Test the Retraining Model With Your Own Dataset (TEST_ATMFN.PY)

Download the commonly used testing datasets (lfw, CelebA, Helen) (Baidu Cloud, (Password:7ov5)). Put your dataset in './test/test_data/'.

Change the data path. Run the following commands:

cd ./src/test
python test.py

The SR results will be in './test/test_data/TEST_ATMFN'.

Loss function

Adversarial loss

This implementation adopts the least squares loss function instead of the sigmoid cross entropy loss function for the discriminator. See the details: Least Squares Generative Adversarial Networks

Content loss

The paper says VGG54 is the perceptually most convincing results. But this implemetation uses all the feature maps generated by every layer (i.e. phi12, phi22, phi34, phi44, phi54) within the VGG19 network.

Citation

@ARTICLE{8936424,
  author={K. {Jiang} and Z. {Wang} and P. {Yi} and G. {Wang} and K. {Gu} and J. {Jiang}},
  journal={IEEE Transactions on Multimedia}, 
  title={ATMFN: Adaptive-Threshold-Based Multi-Model Fusion Network for Compressed Face Hallucination}, 
  year={2020},
  volume={22},
  number={10},
  pages={2734-2747},
  doi={10.1109/TMM.2019.2960586}}