Skip to content
An extensive machine learning library, made from scratch (Python).
Python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
ML
tests
.gitignore
.travis.yml
LICENSE
README.md
__init__.py
requirements.txt
setup.py

README.md

Overview

Travis Coverage Status

This is a machine learning library, made from scratch.

It uses:

  • numpy: for handling matrices/vectors
  • scipy: for various mathematical operations
  • cvxopt: for convex optimization
  • networkx: for handling graphs in decision trees

It contains the following functionality:

  • Supervised Learning:
    • Linear and Logistic regression
      • Regularization
      • Solvers
        • Gradient descent
        • Steepest descent
        • Newton's method
        • SGD
        • Backtracking line search
        • Closed form solutions
    • Support Vector Machines
      • Soft and hard margins
      • Kernels
    • Tree Methods
      • CART (classificiation and regression)
      • PRIM
      • AdaBoost
      • Gradient Boost
      • Random Forests
    • Kernel Smoothing Methods
      • Nadaraya average
      • Local linear regression
      • Local logistic regression
      • Kernel density classification
    • Discriminant Analysis
      • LDA, QDA, RDA
    • Naive Bayes Classification
      • Gaussian
      • Bernoulli
    • Prototype Methods
      • KNN
      • LVQ
      • DANN
    • Perceptron
  • Unsupervised Learning
    • K means/mediods clustering
    • PCA
    • Gaussian Mixtures
  • Model Selection and Validation

Examples

Examples are shown in two dimensions for visualisation purposes, however, all methods can handle high dimensional data.

Regression

  • Linear and logistic regression with regularization. Closed form, gradient descent, and SGD solvers.

Imgur

Imgur

Support Vector Machines

  • Support vector machines maximize the margins between classes

Imgur

  • Using kernels, support vector machines can produce non-linear decision boundries. The RBF kernel is shown below

Imgur

Imgur

  • An alternative learning algorithm, the perceptron, can linearly separate classes. It does not maximize the margin, and is severely limited.

Imgur

Tree Methods

  • The library contains a large collection of tree methods, the basis of which are decision trees for classification and regression

Imgur

These decision trees can be aggregated and the library supports the following ensemble methods:

  • AdaBoosting
  • Gradient Boosting
  • Random Forests

Kernel Methods

Kernel methods estimate the target function by fitting seperate functions at each point using local smoothing of training data

  • Nadaraya–Watson estimation uses a local weighted average

Imgur

  • Local linear regression uses weighted least squares to locally fit an affine function to the data

Imgur

  • The library also supports kernel density estimation (KDE) of data which is used for kernel density classification

Imgur

Discriminant Analysis

  • Linear Discriminant Analysis creates decision boundries by assuming classes have the same covariance matrix.
  • LDA can only form linear boundries

Imgur

  • Quadratic Discriminant Analysis creates deicion boundries by assuming classes have indepdent covariance matrices.
  • QDA can form non-linear boundries.

Imgur

  • Regularized Discriminant Analysis uses a combination of pooled and class covariance matrices to determine decision boundries.

Imgur

Prototype Methods

  • K-nearest neighbors determines target values by averaging the k-nearest data points. The library supports both regression and classification.

Imgur

  • Learning vector quantization is a prototype method where prototypes are iteratively repeled by out-of-class data, and attracted to in-class data

Imgur

  • Discriminant Adaptive Nearest Neighbors (DANN). DANN adaptively elongates neighborhoods along boundry regions.
  • Useful for high dimensional data.

Imgur

Unsupervised Learning

  • K means and K mediods clustering. Partitions data into K clusters.

Imgur

  • Gaussian Mixture Models. Assumes data are generated from a mixture of Gaussians and estimates those Gaussians via the EM algorithm. The decision boundry between two estimated Gaussians is shown below.

Imgur

  • Principal Component Analysis (PCA) Transforms given data set into orthonormal basis, maximizing variance.

Imgur

You can’t perform that action at this time.