Machine Learning Toolkit

The Machine Learning Toolkit is at the core of kdb+/q-centered machine-learning functionality. This library contains functions that cover the following areas:

• An implementation of the FRESH (FeatuRe Extraction and Scalable Hypothesis testing) algorithm for use in the extraction of features from time series data and the reduction in the number of features through statistical testing.
• Cross-validation and grid-search functions allowing for testing of the stability of models to changes in the volume of data or the specific subsets of data used in training.
• Clustering algorithms used to group data points and to identify patterns in their distributions. The algorithms make use of a k-dimensional tree to store points and scoring functions to analyze how well they performed.
• Statistical timeseries models and feature-extraction techniques used for the application of machine learning to timeseries problems. These models allow for the forecasting of the future behavior of a system under various conditions.
• Numerical techniques for calculating the optimal parameters for an objective function.
• A graphing and pipeline library for the creation of modularized executable workflow based on a structure described by a mathematical directed graph.
• Utility functions relating to areas including statistical analysis, data preprocessing and array manipulation.

These sections are explained in greater depth within the FRESH, cross validation, clustering, timeseries, optimization, graph/pipeline and utilities documentation.

Requirements

• embedPy

The Python packages required to allow successful execution of all functions within the machine learning toolkit can be installed via:

pip:

pip install -r requirements.txt

or via conda:

conda install --file requirements.txt

Installation

Place the ml folder in $QHOME and load into a q instance using ml/ml.q

The following will load all functionality into the .ml namespace

\l ml/ml.q
.ml.loadfile`:init.q

Examples

Examples showing implementations of several components of this toolkit can be found here. These notebooks include examples of the following sections of the toolkit.

• Pre-processing functions
• Implementations of the FRESH algorithm
• Cross validation and grid search capabilities
• Results Scoring functionality
• Clustering methods applied to datasets
• Timeseries modeling examples

Documentation

Documentation for all sections of the Machine Learning Toolkit:

📂 docs

Status

The Machine Learning Toolkit is provided here under an Apache 2.0 license.

If you find issues with the interface or have feature requests, please raise an issue.

To contribute to this project, please follow the contributing guide.