This repo contains open source material for Algorithmic Foundations of Data Science.
We thank the Robbins Family for their support! https://www.utsa.edu/alumni/news/2021/Retired%20Professors.html
Our main textbook was Kevin Muprhy's Into to Probabilistic ML: https://probml.github.io/pml-book/book1.html
This was complemented with my own notes for numerical analysis, optimization, JL Lemma, Concentration Inequalities, Streaming, and Sketching.
The lecture plan is as follows:
1-) Linear Algebra + Linear Algorithms -- QR, Cholesky, PCA, Linear Regression --
2-) Numerical Analysis, Conditioning, Stability
3-) Basics of Probability, Cross-Entropy distance, Naive Bayes Classifier, Logistic Regression
4-) Multivariate Probability, Convolutiions, CLT, Gaussian-Gaussian, PPCA, MLE. EM algorithm
5-) Basics of Decision Theory and Other Generalities (like Bias-Variance trade-off)
6-) Basics of Differentiable Programming
7-) Convolutional NNets
8-) NNets with memory: RNN, LSTM, Transformers, SSM
9-) Convexity Basics and Gradient Descent
10-) Stochastic, Accelerated, and Preconditioned Gradient Descent --- AdaGrad, Ada-Delta, RMSProp, ADAM --
11-) Convex Constrained Gradient Descent, Coordinate Descent -- Projected Gradient Descent, Frank-Wolfe, Graphical LASSO --
12-) Duality, KKT, Non-smooth Optimization -- Subgradient Methods, Proximal Point Method, Bundle Methods --
13-) Markov Chains and MCMC --- Metropolis-Hastings, Gibbs --
14-) Concentration Inequalities (Chernoff, Hoefding), Morris++, Hashing, Bloom Filters
15-) Limear Sketching, Johnson-Lindenstrauss
16-) Autoencoders, Manifold Learning
17-) Classical ML, Kernel Methods
18-) Trees, Forests, Bagging, Boosting
We also had industry speakers on: (1) Causal Inference, (2) Time-Series Analysis, (3) Reinforcement Learning, (4) Generative AI, (5) Recommendation and Search
Any feedback on the lecture plan is welcome.
Some of the lecture notes are available here in the repo, and some others are planned to be available in future. I did not have time to scrutinize the notes yet, so take it with your own risk.
We have wonderful notebooks prepped by Farhan Tajwar Romit on Matrix decompositions, PCA, PPCA, Linear Regression, Logistic Regression, Basics of Neural Nets - Backprop (after Karpathy).
Planned notebooks for future are: Why momentum works? and XGBoost/CatBoost.
Lecture recordings are available at https://www.youtube.com/watch?v=ME2n1WertBs&list=PLTgfuKRICqV9fYPB8ifXDnfVgv4MI9NrI