[TEST PR] Adding oblique trees (i.e. Forest-RC) to cythonized tree module

[adam2392]

Reference Issues/PRs

Fixes:

What does this implement/fix? Explain your changes.

Adds cythonized oblique trees to the tree module. This is known as Forest-RC in the Breiman 2001 paper.

• _oblique_tree.pxd/pyx: This file implements i) the ObliqueTree(Tree), which defines a few additional class members for storing the projection weight and indices and a new function for adding and oblique node and then ii) the ObliqueTreeBuilder(TreeBuilder), which defines how to build the oblique tree.
• _oblique_splitter.pxd/pyx: This is the main change, which i) defines an ObliqueSplitRecord for keeping track of oblique splits, and ii) defines an ObliqueSplitter(Splitter) which gets oblique node splits and samples projection matrices, while also storing additional hyperparameters.
• _classes.py: Defines new Python interfaces for the Oblique trees and forests

Any other comments?

I'm not an expert in cython and c++ interplay, but I suspect that if we can "generalize" the Node struct to carry projection vector and weight information (not used in Forest-RI, or axis-aligned Random Forest), then much of the tree, tree building code is not even necessary. The only thing that is different at a fundamental level is the idea of a sample_proj_mat at each node of the tree, which samples sparse combinations.

Another missing component currently is the implementation on sparse data, but this should be easily added in I presume.

Code That Can Be Shortened

New data structures and classes, ObliqueSplitRecord, ObliqueSplitter, ObliqueTree are defined. However, if the existing SplitRecord, Splitter, Tree can be generalized, then the existing functions can just be used to build Oblique Trees too.

However, I'm not sure if the scikit-learn devs would want that, rather then just replicating some code across these two cases?

[thomasjpfan]

Thanks for the prototype, I think I see what is required to make it easier for you to extend scikit-learn's trees.

In the redesign, I think methods such as tree._apply_dense and DenseObliqueSplitter.node_split would still need to exist to perform your custom logic + data structures.

[thomasjpfan]

This is a different API than the scikit-learn. As you mentioned in office hours, to better support this use case, tree._add_node should accept a SpiltRecord and SplitRecord is a class that can be subclassed. WDYT?

[adam2392]

Yes if SplitRecord can become a class that can be subclassed, then this becomes easier. However, I wasn't sure if the struct was there for performance, or maintenance reasons.

[thomasjpfan]

This is the same API as scikit-learn. As long as SplitRecord can be subclassed, I think this should just work with your custom splitter.

[adam2392]

Yes, what I specced out at least initially is that almost all of the Tree class currently in scikit-learn can be used for any "new tree" that only changes the splitter function as long as SplitRecord and the type of splitter can be subclassed.

Since the tree doesn't do anything fancy except call Splitter functions.

[adam2392]

Some notes as of 3/10/22

Have been working on a refactor that would make adding oblique trees to the existing codebase a lot simpler.

Some notes for further improvement:

• To enable fusedtype member in a class, use tempita: Allow fused types in extension type attributes. cython/cython#3283 (comment)
• To migrate the proj_vec_indices and weights to a struct ObliqueNode, requires converting a vector pointer to numpy array to get the DTYPE (see: https://stackoverflow.com/questions/58476007/creating-a-numpy-array-from-a-pointer-in-cython)