Multi-label classification using the Classifier Chain framework

[andrewdalpino]

Unlike binary or multiclass classification, multi-label classification aims to predict the presence of more than a single class per sample. One approach to learning a model for this type of problem is to train a binary classifier to detect the presence of a single class label for each unique class label. This is sometimes called the "one vs. rest" approach. The simplest case of multi-label classification via this method is to assume that all labels are independent (the presence of one says nothing about the others). This case is referred to as Binary Relevance. A more complex approach is to model the correlations of labels using a directed acyclic graph (DAG) in which binary classifiers feed their predictions as features to downstream classifiers. This ticket is to research and develop a ensemble classifier that is capable of multi-label classification and covers the case where the indegree of each downstream node is at least 1 (Binary Relevance). If we can figure a way to greedily construct an (semi) optimal DAG using some heuristic approach then even better!

Related literature:

https://arxiv.org/pdf/1912.13405.pdf

Scikit Learn implemetation (Binary Relevance only):

https://scikit-learn.org/dev/modules/generated/sklearn.multiclass.OneVsRestClassifier.html

Other things to consider ...

• How expensive is it to construct a full-connected DAG in a real-world setting?
• Could cross validation methods be used to prune a fully-connected DAG for better inference performance?
• What changes to the Dataset API need to be made?
• Should we use probabilities, classes, or one of the other as derived features for downstream classifiers?

[andrewdalpino]

So the challenges so far for this one are in the necessity for an additional data type list as well as an additional estimator type multi-label. This may or may not be worth the trouble just yet.

Additional things to consider

• Will be ever do multi-label regression? Are there any literature on multi-label regression?
• How to stratify a multi-dimensional label?
• Is the behavior of possibleOutcomes() sane?

[andrewdalpino]

After some thinking, I now have the opinion that this solution is too opinionated and is better handled by providing the necessary primitives to build Classification Chains on a per case basis and add a section to the documentation with instructions and example of how to build such systems for multi-label classification. Perhaps we can discover some features to implement that will aid in this process along the way!

See here for prototype https://github.com/RubixML/ML/compare/1.1...one-v-rest?expand=1