ResNet + Fractional Max Pooling

Introduction

This repository implements two research papers for state-of-the-art Convolutional Neural Networks on the CIFAR - 10 dataset. Further, these two models have been combined to form a hybrid model. The performances of all these models have been documented and shown and a comparison has been drawn between them. The advantages offered by the Hybrid have been identified and listed.

ResNet

• ResNet is a fairly recent architecture (2015) which has surpassed human level performance on the ImageNet dataset.
• ResNet solves the problem of vanishing gradients that occurs in many deep CNNs, by implementing skip connections.
• Typical skip connections are built over 2 or 3 layers. That is a skip connection is present every 2 or 3 layers.

Fractional Max Pooling

• Traditionally neural nets use max pooling with 2x2 grids (2MP), which reduces the image dimensions by a factor of 2.
• An alternative would be to use pooling schemes that reduce by factors other than two, e.g. 1 < factor < 2.
• Pooling by a factor of sqrt(2) would allow twice as many pooling layers as 2MP, resulting in "softer" image size reduction throughout the network. This is called Fractional Max Pooling (FMP).
• FMP has been shown to successfully remove overfitting without even using dropout.

Models

ResNet

• Number of Layers (ResNet blocks): 20
• Number of Dense Layers: 2
• Number of Parameters: 1,612,042
• Model Summary

Fractional Max Pooling

• Number of Layers (Convolutional blocks): 18
• Number of Dense Layers: 2
• Number of Parameters: 66,865,738
• Model Summary

HYBRID: ResNet + Fractional Max Pooling

This model aims to combine both the above techniques. To accomplish this, the Max Pooling layers in ResNet have been replaced by Fractional Pooling Layers, which were created using Keras Layer Sub-classing (Custom Layer)

• Number of Layers (ResNet blocks): 12
• Number of Dense Layers: 2
• Number of Parameters: 868,362
• Model Summary

Results

These models were evaluated on the CIFAR - 10 dataset with a training set size of 50,000 and a validation set size of 10,000. Training was done over 50 epochs with a step size 100. There were 64 images per batch.

ResNet

• Train accuracy: ~78%
• Validation accuracy: ~71%
  
  
  

Fractional Max Pooling

• Train accuracy: ~70%
• Validation accuracy: ~69%
  
  
  

Hybrid Model

• Train accuracy: ~80%
• Validation accuracy: ~72%
  
  

Conclusions

• The Hybrid Model gives comparable results to the ResNet model, with almost half the number of layers and parameters.
• This leads to much lesser scope of overfitting in the Hybrid Model since the number of parameters is 50%.
• Since the size of the model is significantly smaller, the memory consumption of the model is lesser but gives similar levels of performance.
• For more complex models, the depth of the hybrid model can thus be increased without worrying about computation and overfitting as with regular ResNets

Papers

• Deep Residual Learning for Image Recognition, by Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun

• Fractional Max-Pooling by Benjamin Graham