Repository for the course "Efficient Deep Learning" at IMT Atlantique

Course organisation / Syllabus

Here is a detailed schedule, session by session:

1. Introduction / Refresher on Deep Learning
   1. General Intro - Why do we need to optimize deep learning ? Introduction of the MicroNet Challenge.
   2. Course - Deep Learning and Transfer Learning.
   3. Practical session - introduction to PyTorch
   4. Short project - exploring hyper parameters on a fixed architecture
   5. Presentation of the Challenge Objectives and Rules (Long Project)
2. Data Augmentation, Regularization
   1. Short evaluation on Deep Learning Essentials and CNN layers
   2. Course - Data Augmentation, Regularization and Self Supervised Learning
   3. Practical Session - Data Augmentation
   4. Short project - exploring hyper parameters on a fixed architecture
3. Quantization
   1. Short evaluation on Data Augmentation
   2. Students presentations of short project - exploring hyper parameters on a fixed architecture
   3. Course - Quantifying Deep neural networks
   4. Practical session - quantification on a small convolutional network
   5. Long project 1 - Challenge
4. Pruning
   1. Short evaluation on Quantization
   2. Course - Pruning Deep neural networks
   3. Practical session - pruning on a small convolutional network.
   4. Long project 2 - Challenge
5. Factorization
   1. Short evaluation on Pruning
   2. Students presentations on current work on Challenge
   3. Course - Factorizing Deep neural networks
   4. Long Project 3 - Challenge
6. Distillation
   1. Short evaluation on Factorization
   2. Course - Distillation of knowledge and features between neural networks
   3. Long Project 4 - Challenge
7. Embedded Software and Hardware for Deep Learning
   1. Long evaluation on Distillation and previous courses
   2. Course - Embedded Software and Hardware for Deep Learning
   3. Long Project 5 - Challenge
8. Final Session
   1. Students presentations - Challenge final results
   2. Students feedback

What is expected for the Long Project

Short version : Exploration of the accuracy / complexity tradeoff

Long version : this course is mostly based on the long project, and you have a lot of freedom, which we expect you to use. The overarching goal is to explore the tradeoff between the performances of neural networks (= Accuracy on the test set) and complexity. Complexity can be either computational complexity (number of arithmetic operations), or memory complexity (memory size necessary to use the network).

We encourage students to get creative and test combinations of the various ideas that we present. Starting from the end of Session 1, you already have enough knowledge to explore the tradeoff between architecture, number of parameters, and accuracy. Then, we study new notions that open new avenues to explore this tradeoff : quantization, pruning, factorization, distillation. In session 7, you'll have a deeper insight on how to thing about specific software or hardware architecture in order to fully exploit all the optimizations that can be done.

Evaluation in this course

There are short written evaluations during the first 10 minutes of each session starting from session 2. Don't be late!

For the final session, we ask you to prepare a 15 minutes presentation, that will be followed by 5 Minutes of question.

What we expect for the presentations :

1. Explain your strategy to explore the complexity / accuracy tradeoff. We will judge whether you understood the goal, and whether the proposed strategy follows a rigourous approach.
2. The clarity of your exposition and quality of your support (slides)

General References

List of references IMT Atlantique and AI

Amazon Book - Dive into Deep learning

Tutorial presentation on Efficient Deep Learning from NeurIPS'19

Training Deep Networks

Here are some academic papers discussing learning rate strategies :

• Cyclic learning rates
• Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks
• A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation

Main strategies are readily available in pytorch.

Pytorch

Start page to access the full python API of pytorch, to check all existing functions.

A useful tutorial on Saving and Loading models.

Pytorch Cheat Sheet.

Data Augmentation

Popular methods :

Cut Out

Auto Augment

Other ressources :

A list of papers and code for data augmentation

IMGAUG and Colab Notebook showing how to use IMGAUG with pytorch

A popular python package in Kaggle competitions : Albumentations

Quantization

Binary Connect

XnorNet

BNN+

Whitepaper of quantization

Pruning

Pruning Filters for Efficient ConvNets

ThiNet

AutoML for Model Compression (AMC)

Pruning Channel with Attention Statistics (PCAS)

BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization

Factorization and operators

Deep Compression

Deep K-means

SqueezeNet

MobileNet

MobileNetV2

Shift Attention Layers

Distillation

Distilling the knowledge in a neural network

Fitnets: Hints for thin deep nets

LIT: Learned Intermediate Representation Training for Model Compression

A Comprehensive Overhaul of Feature Distillation

And the bit goes down: Revisiting the quantization of neural networks

Self-Supervised Learning

Pretext tasks used for learning Representations from data without labels

Embedded Software and Hardware

See references section of Tutorial presentation on Efficient Deep Learning from NeurIPS'19.

Companies / private sector

13 highest funded startups for hardware for DL

More complete list of companies working on hardware DL

Setting up on personal computer

Please see here for instructions on how to setup your environment on your personal computer.