Skip to content
Super-scale your images and run experiments with Residual Dense and Adversarial Networks.
Python Shell
Branch: master
Clone or download
Pull request Compare This branch is 15 commits ahead of idealo:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ISR Fix versioning. Jul 16, 2019
data/input/sample Corrected configuration dictionary. Closes idealo#20. May 7, 2019
figures Added documentation for new model and large image inference. Version … Jul 16, 2019
mkdocs Added patch flatness scheduled check. Minor refactoring. Apr 11, 2019
notebooks Corrected configuration dictionary. Closes idealo#20. May 7, 2019
scripts
tests
weights/sample_weights Merge pull request idealo#42 from idealo/dev Jul 16, 2019
.bumpversion.cfg Add bumpversion. Jul 16, 2019
.dockerignore Remove requirements.txt and adjust Dockerfiles. Mar 25, 2019
.gitattributes Added RRDN VGG-GANS no noise-cancelling weights. Jul 16, 2019
.gitignore Corrected configuration dictionary. Closes idealo#20. May 7, 2019
.travis.yml
CHANGELOG Added documentation for new model and large image inference. Version … Jul 16, 2019
CONTRIBUTING.md
Dockerfile.cpu Remove requirements.txt and adjust Dockerfiles. Mar 25, 2019
Dockerfile.gpu idealo#9: Fix TF-cpu overriding TF-gpu in Docker. Closes idealo#9. Apr 3, 2019
LICENSE Make image super resolution project open source. Nov 26, 2018
README.md Fix README. Closes idealo#39. Jul 16, 2019
config.yml
pypi.sh Bump up version from 1.9.0 to 1.9.1. Mar 15, 2019
runway.yml install tensorflow-gpu on gpu build Jun 26, 2019
runway_model.py
setup.cfg New version, initial commit. Beta. Mar 14, 2019
setup.py Add bumpversion. Jul 16, 2019

README.md

Image Super-Resolution (ISR)

Build Status License

The goal of this project is to upscale and improve the quality of low resolution images.

This project contains Keras implementations of different Residual Dense Networks for Single Image Super-Resolution (ISR) as well as scripts to train these networks using content and adversarial loss components.

The implemented networks include:

Read the full documentation at: https://idealo.github.io/image-super-resolution/.

Docker scripts and Google Colab notebooks are available to carry training and prediction. Also, we provide scripts to facilitate training on the cloud with AWS and nvidia-docker with only a few commands.

ISR is compatible with Python 3.6 and is distributed under the Apache 2.0 license. We welcome any kind of contribution. If you wish to contribute, please see the Contribute section.

Contents

Pre-trained networks

The weights used to produced these images are available under sample_weights (see Additional Information).

IMPORTANT: the weights are stored on git lfs. To download them either:

  • download the raw file (e.g. here -> Download);
  • clone the repository and run git lfs pull.

Basic model

RRDN model, PSNR driven, weights here.

butterfly-sample
Low resolution image (left), ISR output (center), bicubic scaling (right). Click to zoom.

GANS model

RRDN model, trained with Adversarial and VGG features losses, weights here.

baboon-comparison
RRDN GANS model (left), bicubic upscaling (right).
-> more detailed comparison

Artefact Cancelling GANS model

RDN model, trained with Adversarial and VGG features losses, weights here.

temple-comparison
Standard vs GANS model. Click to zoom.
sandal-comparison
RDN GANS artefact cancelling model (left), RDN standard PSNR driven model (right).
-> more detailed comparison

Installation

There are two ways to install the Image Super-Resolution package:

  • Install ISR from PyPI (recommended):
pip install ISR
  • Install ISR from the GitHub source:
git clone https://github.com/idealo/image-super-resolution
cd image-super-resolution
git lfs pull
python setup.py install

Usage

Prediction

Load image and prepare it

import numpy as np
from PIL import Image

img = Image.open('data/input/test_images/sample_image.jpg')
lr_img = np.array(img)

Load model and run prediction

from ISR.models import RDN

rdn = RDN(arch_params={'C':6, 'D':20, 'G':64, 'G0':64, 'x':2})
rdn.model.load_weights('weights/sample_weights/rdn-C6-D20-G64-G064-x2/ArtefactCancelling/rdn-C6-D20-G64-G064-x2_ArtefactCancelling_epoch219.hdf5')

sr_img = rdn.predict(lr_img)
Image.fromarray(sr_img)

Large image inference

To predict on large images and avoid memory allocation errors, use the by_patch_of_size option for the predict method, for instance

sr_img = model.predict(image, by_patch_of_size=50)

Check the documentation of the ImageModel class for further details.

Training

Create the models

from ISR.models import RRDN
from ISR.models import Discriminator
from ISR.models import Cut_VGG19

lr_train_patch_size = 40
layers_to_extract = [5, 9]
scale = 2
hr_train_patch_size = lr_train_patch_size * scale

rrdn  = RRDN(arch_params={'C':4, 'D':3, 'G':64, 'G0':64, 'T':10, 'x':scale}, patch_size=lr_train_patch_size)
f_ext = Cut_VGG19(patch_size=hr_train_patch_size, layers_to_extract=layers_to_extract)
discr = Discriminator(patch_size=hr_train_patch_size, kernel_size=3)

Create a Trainer object using the desired settings and give it the models (f_ext and discr are optional)

from ISR.train import Trainer
loss_weights = {
  'generator': 0.0,
  'feature_extractor': 0.0833,
  'discriminator': 0.01
}
losses = {
  'generator': 'mae',
  'feature_extractor': 'mse',
  'discriminator': 'binary_crossentropy'
}

log_dirs = {'logs': './logs', 'weights': './weights'}

learning_rate = {'initial_value': 0.0004, 'decay_factor': 0.5, 'decay_frequency': 30}

flatness = {'min': 0.0, 'max': 0.15, 'increase': 0.01, 'increase_frequency': 5}

trainer = Trainer(
    generator=rrdn,
    discriminator=discr,
    feature_extractor=f_ext,
    lr_train_dir='low_res/training/images',
    hr_train_dir='high_res/training/images',
    lr_valid_dir='low_res/validation/images',
    hr_valid_dir='high_res/validation/images',
    loss_weights=loss_weights,
    learning_rate=learning_rate,
    flatness=flatness,
    dataname='image_dataset',
    log_dirs=log_dirs,
    weights_generator=None,
    weights_discriminator=None,
    n_validation=40,
)

Start training

trainer.train(
    epochs=80,
    steps_per_epoch=500,
    batch_size=16,
    monitored_metrics={'val_PSNR_Y': 'max'}
)

Additional Information

You can read about how we trained these network weights in our Medium posts:

RDN Pre-trained weights

The weights of the RDN network trained on the DIV2K dataset are available in weights/sample_weights/rdn-C6-D20-G64-G064-x2/PSNR-driven/rdn-C6-D20-G64-G064-x2_PSNR_epoch086.hdf5.
The model was trained using C=6, D=20, G=64, G0=64 as parameters (see architecture for details) for 86 epochs of 1000 batches of 8 32x32 augmented patches taken from LR images.

The artefact can cancelling weights obtained with a combination of different training sessions using different datasets and perceptual loss with VGG19 and GAN can be found at weights/sample_weights/rdn-C6-D20-G64-G064-x2/ArtefactCancelling/rdn-C6-D20-G64-G064-x2_ArtefactCancelling_epoch219.hdf5 We recommend using these weights only when cancelling compression artefacts is a desirable effect.

RDN Network architecture

The main parameters of the architecture structure are:

  • D - number of Residual Dense Blocks (RDB)
  • C - number of convolutional layers stacked inside a RDB
  • G - number of feature maps of each convolutional layers inside the RDBs
  • G0 - number of feature maps for convolutions outside of RDBs and of each RBD output


source: Residual Dense Network for Image Super-Resolution

RRDN Network architecture

The main parameters of the architecture structure are:

  • T - number of Residual in Residual Dense Blocks (RRDB)
  • D - number of Residual Dense Blocks (RDB) insider each RRDB
  • C - number of convolutional layers stacked inside a RDB
  • G - number of feature maps of each convolutional layers inside the RDBs
  • G0 - number of feature maps for convolutions outside of RDBs and of each RBD output


source: ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks

Contribute

We welcome all kinds of contributions, models trained on different datasets, new model architectures and/or hyperparameters combinations that improve the performance of the currently published model.

Will publish the performances of new models in this repository.

See the Contribution guide for more details.

Bump version

To bump up the version, use

bumpversion {part} setup.py

Citation

Please cite our work in your publications if it helps your research.

@misc{cardinale2018isr,
  title={ISR},
  author={Francesco Cardinale et al.},
  year={2018},
  howpublished={\url{https://github.com/idealo/image-super-resolution}},
}

Maintainers

Copyright

See LICENSE for details.

You can’t perform that action at this time.