Skip to content
Pytorch tutorial series. How to build feedforward and convolutional neural network for CIFAR-10 classification. How to build LSTM recurrent network for counting chars and sentiment analysis with torchtext
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
data
data_helpers
.gitignore
LICENSE
Pipfile
README.md
conv_net_cifar.py
feedforward_1_hid_nn.py
feedforward_3_hid_nn.py
lstm_net_counting_chars.py

README.md

Pytorch neural networks tutorial

This repository contains easy to follow Pytorch tutorial for beginners and intermediate students. The goal is to introduce you to pytorch on practical examples. I hope that it will help you to start your journey with neural networks.

There are few popular neural network architecture which I teach on workshops or bootcamps like: feedforward, convolutional and recurrent.

In this tutorial we build CIFAR-10 classifiers:

  • Single layer fully connected neural network for CIFAR-10 classification.
  • Feedforward neural network with three hidden layers for CIFAR-10 classification with ReLu activation
  • Convolutional neural network for CIFAR-10 with 3 convolution layer with and fully connected output layer, as activation we use ReLu

Recurent network

  • (todo) LSTM recurrent neural network for counting chars in long text
  • (todo) LSTM recurrent neural network for IMDB sentiment analisys with torchtext

Prerequistis

This tutorial was written and tested on Ubuntu 18.10,

  • Python - version >= 3.6
  • pipenv - package and virtual environment management
  • numpy
  • matplotlib
  • pytorch - version >= 1.0
  • torchtext - version >= 0.4
  1. Install Python.
  2. Install pipenv
  3. Git clone the repository
  4. Install all necessary python packages executing this command in terminal
git clone https://github.com/ksopyla/pytorch_neural_networks.git
cd pytorch_neural_networks
pipenv install

Repository structure

  • ./data - folder for downloaded dataset, all data we are working with are automatically downloaded at first use

Feedforward neural network for classifying CIFAR-10

Network for classifying CIFAR-10 images into one of 10 categories: ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

It is worth checking current SotA results on CIFAR-10

Feedforward neural network with one hidden layer

We build simple network with 1 hidden layer and output layer. As input we pass raw image pixels as 32x32 vector of numbers.

File: feedforward_1_hid_nn.py This model achieve ~ 48% accuracy after 5 epoch.

Model summary:

  • input layer: 3x32x32 (3 rgb channels times image resolution 32pixels)
  • hidden layer: 512 neurons
  • output layer: 10 neurons, each reflects probability of belonging to class

Sample output

Epoch [1/5]], Loss: 1.7713 Test acc: 0.4423
Epoch [2/5]], Loss: 1.6124 Test acc: 0.4582
Epoch [3/5]], Loss: 1.5280 Test acc: 0.466
Epoch [4/5]], Loss: 1.4640 Test acc: 0.4889
Epoch [5/5]], Loss: 1.4130 Test acc: 0.4799

Feedforward neural network with three hidden layers

Analogous to previous model feedforward network with 3 hidden layers and output layer.

This is upgraded version of the previous model, between input and output we added 3 fully connected hidden layers. Adding more layers makes the network more expressive but harder to train. The three new problems could emerge vanishing gradients, model overfitting, and computation time complexity. In our case where the dataset is rather small, we did not see those problems in real scale.

File: feedforward_3_hid_nn.py This model achieve ~ 51% accuracy after 5 epoch.

Model summary:

  • input layer: 3x32x32 (3 rgb channels times image resolution 32pixels)
  • 3 x hidden layers: 512, 256, 128 neurons
  • output layer: 10 neurons, each reflects probability of belonging to class

Sample output

Epoch [1/5]], Loss: 1.6971 Test acc: 0.4627
Epoch [2/5]], Loss: 1.4874 Test acc: 0.4947
Epoch [3/5]], Loss: 1.3887 Test acc: 0.493
Epoch [4/5]], Loss: 1.3108 Test acc: 0.5144
Epoch [5/5]], Loss: 1.2406 Test acc: 0.5166

Convolutional neural network for classifying CIFAR-10

This model uses convolutional neural network with 3 convolution layers and output layer. As input we pass raw image pixels as 32x32 vector of numbers.

File: conv_net_cifar.py This model achieve ~ 67% accuracy after 5 epoch.

Model summary:

  • conv 1 layer - kernel size 3x3, channels in 3, channels out 8,stride=1, padding=1
  • relu - activation
  • conv 2 layer - kernel size 3x3, channels in 8, channels out 16,stride=1, padding=1
  • relu - activation
  • max pooling - window size 2x2, it will downsize image 2x (from 32x32 to 16x16)
  • conv 3 layer - kernel size 3x3, channels in 16, channels out 24,stride=1, padding=1
  • relu - activation
  • max pooling- window size 2x2, it will downsize image 2x (from 16x16 to 8x8)
  • fully connected - 24x8x8x100
  • fully connected(output) - 100x10

Sample output

Epoch [1/5]], Loss: 1.3465 Test acc: 0.6038 time=0:00:42.301045
Epoch [2/5]], Loss: 1.0122 Test acc: 0.64 time=0:00:42.168382
Epoch [3/5]], Loss: 0.8989 Test acc: 0.6649 time=0:00:41.995531
Epoch [4/5]], Loss: 0.8214 Test acc: 0.6834 time=0:00:42.099388
Epoch [5/5]], Loss: 0.7627 Test acc: 0.6761 time=0:00:42.047874
Finished Training
Accuracy of plane : 76 %
Accuracy of   car : 73 %
Accuracy of  bird : 42 %
Accuracy of   cat : 60 %
Accuracy of  deer : 63 %
Accuracy of   dog : 53 %
Accuracy of  frog : 72 %
Accuracy of horse : 69 %
Accuracy of  ship : 79 %
Accuracy of truck : 85 %

Recurrent neural network

TODO:

References and further reading

You can’t perform that action at this time.