Course Material: Numerical Methods for Deep Learning

This repository contains lecture slides for a one-semester graduate-level class on numerical methods for deep learning.

This course was first designed and taught by Lars Ruthotto at Emory University and Eldad Haber at University of British Columbia in the Spring of 2018. The course was last offered in the Spring of 2020 at Emory and the slides reflect this version.

Scope

This course provides students with the mathematical background needed to analyze and further develop numerical methods at the heart of deep learning. The course is simultaneously geared towards science students, who may be interested/experienced with deep learning and would like to strengthen their theoretical foundation, and mathematics students, who may have a background in numerical analysis and who are curious about the emerging technology of deep learning. The course primarily targets graduate students. However, parts of the material may be accessible to advanced undergraduate students.

The ultimate goal is to implement a simple, mathematically sound deep learning code from scratch. This look under the hood will provide valuable insight into successful and well-tested software packages for deep learning tasks and understand their success (and failure).

Course Outline

Introduction, Notation, References

1. Overview
2. Introduction
3. Selected Literature
4. Notation

Module 1: Shallow Models

1. Spectral Clustering for Unsupervised Learning
2. Linear Least-Squares
3. Linear Models for Classification
4. Single Layer Networks
5. Convolutional Neural Networks

Module 2: Deep(er) Models

1. Deep Networks (Multilayer Perceptron)
2. Residual Neural Networks
3. Optimal Control

Acknowledgements

Development of this material is in part supported by the National Science Foundation under Grant Numbers 1522599 and 1751636. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.