Advanced Machine Learning

This repository contains information about the course on Advanced Data Analysis and Machine Learning, spanning from weekly plans to lecture material and various reading assignments. The emphasis is on deep learning algorithms, starting with the mathematics of neural networks (NNs), moving on to convolutional NNs (CNNs) and recurrent NNs (RNNs), autoencoders and other dimensionality reduction methods to finally discuss generative methods. These will include Boltzmann machines, variational autoencoders, generalized adversarial networks, diffusion methods and other.

Time: Each Tuesday at 1015am-12pm CET (The sessions will be recorded), first time January 16, 2024

Lab session: Each Thursday at 215pm-4pm CET, room FØ397

FYS5429 zoom link https://msu.zoom.us/j/6424997467?pwd=TEhTL0lmTmpGbHlnejZQa1pCdzRKdz09

Meeting ID: 642 499 7467 Passcode: FYS4411

Furthermore, all teaching material is available from this GitHub link.

January 15-19: Presentation of couse, review of neural networks and deep Learning and discussion of possible projects

• Presentation of course and overview
• Discussion of possible projects
• Deep learning, neural networks, basic equations
• Recommended reading Goodfellow et al chapters 6 and 7
• Video of first lecture at https://youtu.be/dP8g_tjQ_9c
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week1/ipynb/week1.ipynb

January 22-26

• Mathematics of deep learning, basics of neural networks
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week2/ipynb/week2.ipynb
• Recommended reading Goodfellow et al chapters 6 and 7 and Raschka et al chapter 11. For Pytorch see Raschka et al chapter 12.
• Link to video of lecture at https://youtu.be/SEYuOoMws_k
• Link to whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesJanuary23.pdf

January 29-February 2

• Mathematics of deep learning
• Discussion of first project
• Video of lecture at https://youtu.be/OUTFo0oJadU
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week3/ipynb/week3.ipynb
• Recommended reading Goodfellow et al chapters 6 and 7 and Raschka et al chapter 11. For Pytorch see Raschka et al chapter 12.

February 5-9

• Mathematics of deep learning
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week4/ipynb/week4.ipynb
• Recommended reading Goodfellow et al chapter 9. Raschka et al chapter 13
• Video of lecture at https://youtu.be/b9ni34-sMRI
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesFebruary6.pdf

February 12-16

• Convolutional neural networks (CNNs), basic mathematics of CNNs
• Video of lecture at https://youtu.be/iNNVYdFw8CI
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesFebruary13.pdf
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week5/ipynb/week5.ipynb
• Recommended reading Goodfellow et al chapter 9. Raschka et al chapter 13

February 19-23

• Mathematics of CNNs and discussion of codes
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week6/ipynb/week6.ipynb
• Recommended reading Goodfellow et al chapters 9 and Raschka et al chapter 14
• Video of lecture at https://youtu.be/jqgSED0tF70
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesFebruary20.pdf

February 26-March 1

• Repetion of CNNs and discussion codes
• Recurrent neural networks, basic mathematics and structure
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week7/ipynb/week7.ipynb
• Video of lecture at https://youtu.be/VkQGq84ml_0
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesFebruary27.pdf
• Recommended reading Goodfellow et al chapters 10 and Raschka et al chapter 15

March 4-8

• Structure of RNNs
• Long-Short-Term memory and applications to differential equations
• Start discussing autoencoders
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week8/ipynb/week8.ipynb
• Recommended reading Goodfellow et al chapter 14 for Autoenconders
• Whiteboard notes https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesMarch5.pdf

March 11-15: Autoencoders

• Autoencoders and discussions of codes and links with PCA
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week9/ipynb/week9.ipynb
• Reading recommendation: Goodfellow et al chapter 14
• Video of Lecture at https://youtu.be/PU_8riCscQg
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesMarch12.pdf

March 18-22: Autoencoders and start discussion of generative models

• Autoencoders and links with Principal Component Analysis. Discussion of AE implementations
• Summary of deep learning methods and links with generative models and discussion of possible paths for project 2
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week10/ipynb/week10.ipynb
• Reading recommendation: Goodfellow et al chapters, 14 and 16
• Video of lecture at https://youtu.be/8s0QC1MvdYg
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesMarch19.pdf

April 1-5: Deep generative models

• Monte Carlo methods and structured probabilistic models for deep learning
• Partition function and Boltzmann machines
• Boltzmann machines
• Reading recommendation: Goodfellow et al chapters 16, 17 and 18
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week11/ipynb/week11.ipynb
• Video of lecture at https://youtu.be/zIG0iEGN05c
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesApril2.pdf

April 8-12: Deep generative models

• Restricted Boltzmann machines, reminder from last week
• Reminder on Markov Chain Monte Carlo and Gibbs sampling
• Discussions of various Boltzmann machines
• Implementation of Boltzmann machines using TensorFlow and Pytorch
• Reading recommendation: Goodfellow et al chapters 18.1-18.2, 20.1-20-7; To create Boltzmann machine using Keras, see Babcock and Bali chapter 4
• See also Foster, chapter 7 on energy-based models
• Video of lecture at https://youtu.be/hEjcK0ZkuAA
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesApril9.pdf

April 15-19: Deep generative models

• Energy-based models and Langevin sampling
• Variational autoencoders
• Reading recommendation: Goodfellow et al chapter 20.10-20.14
• See also Foster, chapter 7 on energy-based models
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week13/ipynb/week13.ipynb
• Video of lecture at https://youtu.be/rw-NBN293o4
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesApril16.pdf

April 22-26: Deep generative models

• Variational Autoencoders
• Reading recommendation: An Introduction to Variational Autoencoders, by Kingma and Welling, see https://arxiv.org/abs/1906.02691
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week14/ipynb/week14.ipynb
• Video of lecture at https://youtu.be/tkOweMYCMVg
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesApril23.pdf

April 29-May 3: Deep generative models

• Summarizing discussion of VAEs
• Generative Adversarial Networks (GANs)
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week15/ipynb/week15.ipynb
• Video of lecture at https://youtu.be/Cg8n9aWwHuU
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesApril30.pdf

May 6-10: Deep generative models

• Generative Adversarial Networks (GANs)
• Diffusion models
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week15/ipynb/week16.ipynb
• Video of lecture at https://youtu.be/lYgKGCQRUhQ
• Whiteboard notes at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/HandwrittenNotes/2024/NotesMay7.pdf

May 13-17: Deep generative models

• Summary of course and discussion of projects
• Slides at https://github.com/CompPhysics/AdvancedMachineLearning/blob/main/doc/pub/week15/ipynb/week17.ipynb
• Video of lecture at https://youtu.be/HWW3vnR4RZE

May 20-31: Lab only and work on project 2 each Thursday

• Only project work May 20 to end of May, Thursdays 215pm-4pm, room FØ397

Recommended textbooks:

o Goodfellow, Bengio and Courville, Deep Learning at https://www.deeplearningbook.org/

o Sebastian Raschka, Yuxi Lie, and Vahid Mirjalili, Machine Learning with PyTorch and Scikit-Learn at https://www.packtpub.com/product/machine-learning-with-pytorch-and-scikit-learn/9781801819312, see also https://sebastianraschka.com/blog/2022/ml-pytorch-book.html

o David Foster, Generative Deep Learning, https://www.oreilly.com/library/view/generative-deep-learning/9781098134174/

o Babcock and Gavras, Generative AI with Python and TensorFlow, https://github.com/PacktPublishing/Hands-On-Generative-AI-with-Python-and-TensorFlow-2