MachineLearningPlayground

Implementation of basic mathematical pattern recognition/machine learning techniques for fun

Setup

• Download with pip

pip install MLplayground

• Download with git

git clone https://github.com/lujiazho/MachineLearningPlayground.git

Basic Algorithms

• ⭐ Support Vector Classifier (SVC) - (Math Derivaton)
• 😃 Support Vector Regressor (SVR) - (Math Derivaton)
• 🔢 Ridge Regression - (Math Derivaton)
• 🎓 Nearest Mean - (Math Derivaton)
• 📕 K-Means - (Math Derivaton)
• 📗 K-Nearest Neighbors (KNN) - (Math Derivaton)
• 👀 Perceptron Learning - (Math Derivaton)
• 📷 MSE techniques (classification&Regression) - (Math Derivaton)
• 🀄 Density Estimation (Non-parametric) - (Math Derivaton)
• 👥 Density Estimation (parametric) - (Math Derivaton)
• 📊 ANN - (Math Derivaton)
• 🐍 PCA - (Math Derivaton)

Convolutional Neural Networks (CNN)

Training time on Colab of multiple implementation of CNN with parameters: epochs=20, batch=2.

Model / Dataset (imgs)	Loops CPU (s/epoch)	NumPy CPU (s/epoch)	CuPy GPU (s/epoch)	Loops+Numba CPU (s/epoch)	Img2col CPU (s/epoch)	Img2col+Numba CPU (s/epoch)
Baseline / Digits (1k)	255	24	19	2	2	1.5
Lenet / Digits (1k)	464	72	63	4.5	4	4
Lenet / Cifar-10 (100)	184.5	13.5	12	0.9	0.6	0.7

Junior versions

• 💻 CNN 1.0 - No Batch No Channel
• 🌱 CNN 2.0 - No Batch But Channel

Senior versions: Include both batch & channel

• 💬 CNN 3.0 - NumPy Array accelerated
• 🔭 CNN loops - Loops + Numba accelerated
• ✨ CNN img2col - Img2col Function accelerated

Math Derivation

• My version: CNN, Img2col
• A better tutorial from Microsoft

ML Playground

• 🏄 Digit Recognizer
• 🎆 Auto Encoder
• ✏️ Neural Network Language Model (NNLM) - (Model Structure)
• 💡 Word2Vec (Skip-gram) - (Model Structure)
• 📣 Autograd

Notes

• Learrning rate decay can be easily added to ANN while training because of the design of optimization operation

TODO

For ANN/MLP

• Relu derivative of D_relu(x==0) could be in [0, 1]
• add cross entropy loss (multi-class case)
• change default initialization from uniform(0, 1) to normal(0, 0.1)
• (Optional) weight initialization: He or Xavier normal

For CNN

• (Optional) Batch normalization

For decomposition

• Do X - np.mean(X, axis=0) to data before PCA transform (Align to origin, which is how sklearn did)

For New

• Add GMM(EM steps)
• Add CART(single decision tree)
• Add logistic regression
• Add Lasso regression
• (Optional) General Bridge Regression
• Add LMS for singal processing
• (Optional) Gaussian Process Regression