Skip to content
Materials for class on topics in deep learning (STAT 991, UPenn/Wharton)
HTML TeX R
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Code/First Look at NNs syllabus + code Aug 28, 2019
Lecture Notes update May 22, 2019
Stat 991 presentations Update stat991_ntk_sep12.pdf Sep 12, 2019
Syllabus syllabus + code Aug 28, 2019
.gitignore update May 22, 2019
LICENSE Initial commit May 14, 2019
README.md Update README.md Sep 12, 2019

README.md

STAT 991: Topics in deep learning (UPenn)

STAT 991: Topics in Deep Learning is a seminar class at UPenn started in 2018. It surveys advanced topics in deep learning based on student presentations.

Fall 2019

Potential topics: Uncertainty quantification, Adversarial Examples, Symmetry, Theory and Empirics, Interpretation, Fairness, ...

Potential papers:

Uncertainty quantification

Predictive inference with the jackknife+. Slides.

High-Quality Prediction Intervals for Deep Learning: A Distribution-Free, Ensembled Approach

Adversarial Examples

Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

Certified Adversarial Robustness via Randomized Smoothing

On Evaluating Adversarial Robustness

VC Classes are Adversarially Robustly Learnable, but Only Improperly

Adversarial Examples Are Not Bugs, They Are Features

See section 6.1 of my lecture notes for a collection of materials.

Symmetry

Spherical CNNs

Learning SO(3) Equivariant Representations with Spherical CNNs

Invariance reduces Variance: Understanding Data Augmentation in Deep Learning and Beyond

Theory and empirical wonders

Understanding deep learning requires rethinking generalization

Spectrally-normalized margin bounds for neural networks

Neural Tangent Kernel: Convergence and Generalization in Neural Networks. GNTK. Slides

Gradient Descent Provably Optimizes Over-parameterized Neural Networks

Mean-field theory of two-layers neural networks. Youtube talk

Interpretation

Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)

Sanity checks for saliency maps

Fairness

[TBA]

Scalability

[TBA]

Applications

Climate, energy, healthcare...

Other resources

Course on Coursera. A good way to learn the basics.

Stanford classes: CS231N (Computer vision). CS224N (NLP). Cheat sheet.

Conferences: NeurIPS, ICML, ICLR

Convenient ways to run code online: https://colab.research.google.com/notebooks/welcome.ipynb, https://www.kaggle.com/kernels

Keras is a user-friendly language for DL. Interfaces to R, see this book

Foundations of Deep Learning program at the Simons Institute for the Theory of Computing. workshops: 1, 2, 3. Reading groups and papers

IAS Special Year on Optimization, Statistics, and Theoretical Machine Learning

Materials from previous editions

Lecture notes

The materials draw inspiration from many sources, including David Donoho's course Stat 385 at Stanford, Andrew Ng's Deep Learning course on deeplearning.ai, CS231n at Stanford, David Silver's RL course, Tony Cai's reading group at Wharton. They may contain factual and typographical errors. Thanks to several people who have provided parts of the notes, including Zongyu Dai, Georgios Kissas, Jane Lee, Barry Plunkett, Matteo Sordello, Yibo Yang, Bo Zhang, Yi Zhang, Carolina Zheng. The images included are subject to copyright by their rightful owners, and are included here for educational purposes.

Compared to other sources, these lecture notes are aimed at people with a basic knowledge of probability, statistics, and machine learning. They start with basic concepts from deep learning, and aim to cover selected important topics up to the cutting edge of research.

The entire latex source is included, encouraging reuse (subject to appropriate licenses).

Spring 2019

Topics: sequential decision-making (from bandits to deep reinforcement learning), distributed learning, AutoML, Visual Question Answering.

Presentations

Fall 2018

Topics: basics (deep feedforward networks, training, CNNs, RNNs). Generative Adversarial Networks, Learning Theory, Sequence Learning, Neuroscience, etc.

Presentations

You can’t perform that action at this time.