Compares properties of some of the most important classes of supervised ML algorithms applicable to both regression and classification problems. We are not thinking about a specific implementation but rather about the "typical" implementation.
- GLM: Generalized linear model (e.g. logistic regression and the normal linear model) with optional L1/L2 penalties.
- Neural Net: Artificial neural net fitted by the backpropagation algorithm.
- Decision Trees: Recursive binary partitioning. Often called CART ("classification and regression trees").
- Boosting: A combination of sequentially fitted weak learners, usually shallow decision trees. Each learner tries to correct the "errors" from the previous ones. Well-known implementations are AdaBoost, XGBoost, LightGBM, and CatBoost.
- Random Forest: A combination of deep randomized decision trees fitted in parallel. There are two sources of randomness: (1) In each split, only a small subset of features are considered at random. (2) Each tree is fitted on a bootstrap sample.
- k-Nearest Neighbour
| Aspect | GLM | Neural Net | Decision Tree | Boosting | Random Forest | k-Nearest Neighbour |
|---|---|---|---|---|---|---|
| Scalable | 😍 | 😍 | 😃 | 😃 | 😐 | 😒 |
| Easy to tune | 😐 | 😐 | 😐 | 😐 | 😃 | 😐 |
| Flexible losses | 😃 | 😍 | 😃 | 😃 | 😐 | 😐 |
| Regularization | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| Case weights | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| Missing input allowed | 😒 | 😒 | ✔️ | ✔️ | 😒 | 😒 |
| Interpretation | 😍 | 😐 | 😍 | 😐 | 😐 | 😐 |
| Space on disk | 😍 | 😍 | 😍 | 😃 | 😒 | 😒 |
| Birth date (approx.) | 1972 (Nelder & Wedderburn) | 1974 Backprop (Werbos) | 1984 (Breiman et al.) | 1990 (Schapire) | 2001 (Breiman) | 1951 (Fix & Hodges) |
This compilation as per September 7, 2020 is neither complete nor does it claim to be correct.