Alex's Crazy Random Forest Idea =============================== Random Forests typically promote diversity between trees by using a number of predictors to consider at each node (M) which is less than the total number of predictors (P). M seems to range from about 0.3 P to about 0.6 P, but is constant in the entire forest. This implementation chooses M from a distribution. In an attempt to increase diversity without losing accuracy (for each of the trees) this code draws M from a distribution before choosing the predictor variables to use at each node. My thought is that this will allow some nodes in each tree to be much more accurate than other nodes. Since the trees are grown out completely (i.e. not pruned) this might make a difference. Tests indicate nothing conclusive. This implementation was a little better for some data sets and a little worse for others. All the data and tests are in this repo. The professor of the class and I both agree it is worth exploring further with more data and a faster implementation perhaps in something like C that can then be hooked into R. If I wind up doing that, I'll add a link here. The main code (building and traversing of the trees) is in randomForestM.R and the test scripts are in the files that begin with overnight. There are comparisons with the standard random forest code in those test scripts.
The accuracy of a Random Forest is dependent upon the accuracy of the individual trees as well as the diversity between them. This is an attempt to add an additional element of diversity through randomness between the trees without sacrificing their accuracy.