- PDL: Probabilstic Deep Learning models
- SR: Statistical Rethinking: A Bayesian Course with Examples in R and Stan (Second Edition)
- TF Machine Learning with Tensor Flow
Concepts
- Hardware model behind Deep learning
- Tensorflow
- Pytorch
Additional materials
- ??
Concepts
- Fully connected neural networks
- RELU and sigmoid neurons
- Max pooling
- Convolutional neural networks
Additional materials
- ??
Concepts
- LSTM
Additional materials:
Concepts
- Learning rate
- Gradient decent
- Stochastic gradient decent
- Automatic differentation
- Static and dynamic backpropagation
- Hyperparameter optimisation
Additional materials
- ??
Concepts
- Maximum likelihood
- Maximum aposteriori probability
- Loss function derivation
- Crossentropy and negative log-likelihood
- Kullback-Leibler divergence
- Mean square error and negative log-likelihood
Additional materials about the interpretation of Kullback-Leibler divergence
- Sanjeev R. Kulkarni. Information, Entropy, and Coding
- Zachary Robertson. KL Divergence as Code Patching Efficiency
- Will Kurt. Kullback-Leibler Divergence Explained
Main method: Use data to fix a single model and use the model to assign probabilities to observations. Does not work well if there are many near optimal models with widely different predictions.
Concepts
- Negative log-likelihood as loss
- Prediction intervals from predicted distribution
- Models with varying error terms aka Heteroskedasticity
- Predicting mean and variance
Additional materials
- ??
Concepts
- Binomial distribution
- Poisson distribution
- Zero-Inflated Poisson distribution
- Negative binomial distribution
- Logistic regression
- Poisson regression
- Diagnostic methods
Additional materials
Concepts
- Multinomial distribution
- Discretised logistic mixture distribution
- Regression with discretised logistic mixture distribution
- WaveNet and PixelCNN
Additional materials
- DeepMind Blog. WaveNet: A generative model for raw audio
- A. Oord et al. WaveNet: A Generative Model for Raw Audio
- A. Moussa. A Guide to Wavenet
- A. Oord et al. Pixel Recurrent Neural Networks
- T. Salimans et al. PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
- Transformation functions (bijectors)
- Probability density function and Jacobian
- Maximum likelihood to estimate parameters of Transformations
- Neural networks as Transformation functions
- Glow model for faces
Additional materials
- Pyro. Normalizing Flows - Introduction
- Open AI blog. Glow
- D. P. Kingma, P. Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions
Main method Use data to fix the weight of individual models and use these to average over predictions. Allow to measure the uncertainty due to variability of training data.
Concepts
- Model averaging
- Bayes formula and corresponding inference rules
- Coin-tossing example
- Bayesian linear regression model
Additional materials
Concepts
- Kullback-Leibler difference
- Parametric posterior approximation
- Variational inference for a single neuron
- Bayes Backprop algorithm in practice
- Stochastic Variational Inference
Additional materials
- Blundell et al. Weight Uncertainty in Neural Networks
- Jospin et al. Hands-on Bayesian Neural Networks - A Tutorial for Deep Learning Users
- An Introduction to Stochastic Variational Inference in Pyro
Concepts
- Dropout layer as a regulariser
- Monte-Carlo dropout architecture
- Uncertainty measures for Bayesian classification
Additional materials
- Yarin Gal, Zoubin Ghahramani. Dropout as a Bayesian Approximation
- Yarin. Dropout as a Bayesian Approximation. Source code
Concepts
- Metropolis algorithm
- Gibbs sampling
- Hamiltonian Monte Carlo
- Adaptation, burn-in and convergence diagnostics
Additional materials