The identification of relevant features, i.e., the driving variables that determine a process or the property of a system, is an essential part of the analysis of data sets whose entries are described by a large number of variables. The preferred measure for quantifying the relevance of nonlinear statistical dependencies is mutual information, which requires as input probability distributions. Probability distributions cannot be reliably sampled and estimated from limited data, especially for real-valued data samples such as lengths or energies.
This interactive notebook introduces the concepts and the original implementation of total cumulative mutual information (TCMI) to reproduce the main results presented in the publication:
B. Regler, M. Scheffler, and L. M. Ghiringhelli: "TCMI: a non-parametric mutual-dependence estimator for multivariate continuous distributions". Data Mining and Knowledge Discovery (2022).
[DOI: 10.1007/s10618-022-00847-y (pdf)] [Preprint: arxiv:2001.11212 (pdf)]
TCMI is a measure of the relevance of mutual dependencies based on cumulative probability distributions. TCMI can be estimated directly from sample data and is a non-parametric, robust and deterministic measure that facilitates comparisons and rankings between feature sets with different cardinality. The ranking induced by TCMI allows for feature selection, i.e. the identification of the set of relevant features that are statistical related to the process or the property of a system, while taking into account the number of data samples as well as the cardinality of the feature subsets.
It is compared to Cumulative mutual information (CMI), Multivariate maximal correlation analysis (MAC), Universal dependency analysis (UDS), and Monte Carlo dependency estimation (MCDE).
This repository (notebook and code) is released under the Apache License, Version 2.0. Please see the LICENSE file.
Important notes:
- All comparisons have been computed with the Java package
MCDE v1.0written in Scala, which is not part of the repository. To download the package, please visit https://github.com/edouardfouche/MCDE-experiments. To build the package on your own, use thesbtbuild command (sbt compile, sbt package, sbt assembly). Then, copy the resulting java package into theassetsfolder, rename it tomcde.jar, and run all examples with 50,000 iterations. - For the sake of simplicity, all results have been cached. However, results can be recalculated after adjusting the respective test sections. Depending on the test, the calculation time ranges from minutes to days.
Maintainer: 👤 Benjamin Regler
Status: ✔ Actively maintained
Copyright (c) 2018+ Fritz Haber Institute of the Max Planck Society (Benjamin Regler).