Machine Learning Practice

Some practices using statistical machine learning technique based on some dataset.

To see more detail or example about deep learning, you can checkout my Deep Learning repository.

Environment

• Using Python 3

(most of the relative path links are according to the repository root)

Dependencies

• numpy: For low-level math operations
• pandas: For data manipulation
• sklearn - Scikit Learn: For evaluation metrics, some data preprocessing

For comparison purpose

• sklearn: For machine learning models
• cvxopt: For convex optimization problem (for SVM)

NLP related

• gensim: Topic Modelling
• hmmlearn: Hidden Markov Models in Python, with scikit-learn like API
• jieba: Chinese text segementation library
• pyHanLP: Chinese NLP library (Python API)
• nltk: Natural Language Toolkit

Projects

Subject	Technique / Task	Dataset	Solution	Notes
Letter Recognition	kNN / Classification	Letter Recognition Datasets (File)	kNN From Scratch, kNN Scikit Learn	Notes
Page Blocks Classification	Decision Tree / Classification	Page Blocks Classification Data Set (File)	Decision Tree (CART) From Scratch, Decision Tree Scikit Learn	Notes
CSM	Linear Regression / Regression	CSM Dataset (2014 and 2015) (File)	Linear Regression From Scratch, Linear Regression Scikit Learn	Notes
Nursery	Naive Bayes / Classification	Nursery Data Set (File)	Gaussian Naive Bayes From Scratch, Gaussian Naive Bayes Scikit Learn	Notes
Post-Operative Patient	SVM (cvxopt) / Binary Classification	Post-Operative Patient Data Set (File, Simplified)	SVM From Scratch (using cvxopt and simplified dataset), SVM Scikit Learn	Notes
Student Performance	AdaBoost / Classification	Student Performance Data Set (File)	AdaBoost From Scratch, AdaBoost Scikit Learn	Notes
Sales Transactions	k-Means / Clustering	Sales Transactions Dataset Weekly (File)	k-Means From Scratch, k-Means Scikit Learn	Notes
Frequent Itemset Mining	FP-Growth / Frequent Itemsets Mining	Retail Market Basket Data Set (File)	FP-Growth From Scratch	Notes
Automobile	PCA / Dimensionality Reduction	Automobile Data Set (File)	PCA From Scratch, PCA Scikit Learn	Notes
Anonymous Microsoft Web Data	SVD / Recommendation System	Anonymous Microsoft Web Data Data Set (File, Ratings Matrix (by R))	SVD From Scratch, R Notebook - IBCF Recommender System	Notes
Handwriting Digit	SVM (SMO) / Binary & Multi-class Classification	MNIST (File)	Binary SVM From Scratch, Multi-class (OVR) SVM From Scratch	Notes
Chinese Text Segmentation	HMM (EM) / Text Segmentation & POS Tagging	File	HMM From Scratch, HMM hmmlearn, Compare with Jieba and HanLP	-
Document Similarity and LSI	VSM, SVD / LSI	Corpus of the People's Daily (File)	VSM From Scratch, VSM Gensim, SVD/LSI Gensim	Notes

Machine Learning Categories

Consider the learning task

• Surpervised Learning
  • Classification - Discrete
  • Regression - Continuous
• Unsupervised Learning
  • Clustering - Discrete
  • Dimensionality Reduction - Continuous
  • Association Rule Learning
• Semi-supervised Learning
• Reinforcement Learning

Cosider the desired output of a ML system

• Classification
  • Logistic Regression (optimization algo.)
  • k-Nearest Neighbors (kNN)
  • Support Vector Machine (SVM) - Deduction (optimization algo.)
  • Naive Bayes
  • Decision Tree (ID3, C4.5, CART)
• Regression
  • Linear Regression (optimization algo.)
  • Tree (CART)
• Clustering
  • k-Means
  • Hierarchical Clustering
• Association Rule Learning
  • Apriori
  • Eclat
  • FP-growth - Frequent itemsets mining
• Dimensionality Reduction
  • Principal Compnent Analysis (PCA)
  • Single Value Decomposition (SVD) - LSA, LSI, Recommendation System
  • Linear Discriminant Analysis (LDA)

Ensemble Method (Meta-algorithm)

• Bagging
  • Random Forests
• Boosting
  • AdaBoost <- With some basic boosting notes
  • Gradient Boosting
    • Gradient Boosting Decision Tree (GBDT) (aka. Multiple Additive Regression Tree (MART))
  • XGBoost

Others

• Hidden Markov Model (HMM)
• Probabilistic Latent Semantic Analysis (PLSA)
• Latent Dirichlet Allocation (LDA)
• Vector Space Model (VSM)

Heuristic Algorithm

• SMO --> SVM
• EM --> HMM

Machine Learning Concepts

General Case

• Data Preprocessing
  • Normalization
  • Training and Test Sets - Splitting Data
  • Missing Value
  • Dimensionality Reduction
• Model Expansion
  • Binary to Multi-class
• Fitting and Model Complexity
  • Overfitting
  • Underfitting
  • Generalization
  • Regularization
• Reducing Loss
  • Learning Rate
  • Gradient Descent
• Other Learning Method
  • Cost-sensitive Learning
  • Lazy Learning
  • Incremental Learning (Online Learning)
  • Multi-label Classification

Categorized

• Classification
  • Data Preprocessing
    • Label Encoding
  • Real-world Problem
    • Cost-sensitive Learning
    • Classification Imbalance
  • Evaluation Metrics
    • Classification Metrics
  • Binary to Multi-class Expension
• Regression
  • Evaluation Metrics
    • Regression Metrics
• Clustering
  • Evaluation Metrics
    • Clustering Metrics

Specific Field

• Recommendation System
  • Collaborative Filtering
• Information Retrieval - Topic Modelling
  • Latent Semantic Analysis (LSA/LSI/SVD)
  • Latent Dirichlet Allocation (LDA)
  • Random Projections (RP)
  • Hierarchical Dirichlet Process (HDP)
  • word2vec

Machine Learning Mathematics

Topic

• Kernel Usages
• Convex Optimization

Categories

• Linear Algebra
  • Orthogonality
  • Eigenvalues
  • Hessian Matrix
  • Quadratic Form
  • Markov Chain - HMM
• Calculus
  • Multivariable Deratives
    • Quadratic Approximations
    • Lagrange Multipliers and Constrained Optimization - SVM SMO
    • Lagrange Duality
• Probability and Statistics
  • Statistical Estimation
    • Maximum Likelihood Estimation (MLE)

Basics

• Algebra
• Trigonometry

Application

(from A to Z)

• Decision Tree
  • Entropy
• Naive Bayes
  • Bayes' Theorem
• PCA
  • Orthogonal Transformations
  • Eigenvalues
• SVD
  • Eigenvalues
• SVM
  • Convex Optimization
  • Constrained Optimization
  • Lagrange Multipliers
  • Kernel

Books Recommendation

Machine Learning

• Machine Learning in Action
  • Source Code
• 統計學習方法 (李航)
• 機器學習 (周志華)

Mathematics

• Linear Algebra with Applications (Steven Leon)
• Convex Optimization (Stephen Boyd & Lieven Vandenberghe)
• Numerical Linear Algebra (L. Trefethen & D. Bau III)

Resources

Tutorial

Videos

• Google - Machine Learning Recipes with Josh Gordon
  • Josh Gordon's Repository
• Youtube - Machine Learning Fun and Easy
• Siraj Raval - The Math of Intelligence
  • Siraj Raval's Repository
• bilibili - 機器學習 - 白板推導系列
• bilibili - 機器學習升級版

Documentations

• ApacheCN (ML, DL, NLP)
  • Github - AiLearning
  • Official Website - ApacheCN 中文開源組織
• Machine learning 101 (infographics)
  • Machine learning overview (infographic)
  • Machine learning methods (infographic)
  • Machine learning evolution (infographic)

Interactive Learning

• Google Machine Learning Crash Course
• Kaggle Learn Machine Learning
• Microsoft Professional Program - Artificial Intelligence track

MOOC

• Stanford Andrew Ng - CS229
  • Coursera

Github

• Machine Learning from Scratch (eriklindernoren/ML-From-Scratch)
• Jack-Cherish/Machine-Learning
• Dod-o/Statistical-Learning-Method_Code
  • blog

Datasets

• UCI Machine Learning Repository
• Awesome Public Datasets
• Kaggle Datasets
• The MNIST Database of handwritten digits
• 資料集平台 Data Market
• AI Challenger Datasets
• Peking University Open Research Data
• Open Images Dataset
  • github

Machine Learning Platform

• Kaggle
• OpenML

Machine Learning Tool

• AutoML
  • Auto-sklearn
    • github