HoeffdingAdaptiveTreeClassifier does not draw feature engineered variables

[HarshaAsh]

I have created a model with feature engineering. For validating how it works, I have created a sample dataset with y being defined by x1x2. I want to validate if polynomial extender creates x1x2 variable and if HoeffdingAdaptiveTreeClassifier plots this.
This dataset is almost balanced with 52% of the data having class 1. This should indicate that any simple models which don't build a tree should have at most 52% accuracy.
I am creating a model with

feature engineering. I can observe that the accuracy increases to about 91% while running this model for 100 data points. But from the drawing of the tree, I cannot find any tree being built.
In the last piece of code, I run the same example without feature engineering, I can find the trees being built, and accuracy increases as expected. Therefore I believe there is a bug within the draw statement of tree classifiers that do not plot variables created using feature engineering.

Please refer to the attached notebook.
HoeffdingAdaptiveTreeClassifier with polynomial extender.pdf

[smastelini]

Hi @HarshaAsh. Thanks for reporting that. Did you try to increase the value of split_confidence? In your toy example, although the grace period is set to 5, the number of features changes when using the polynomial extender. Therefore, I believe 30 observations might not be enough to trigger a split. The tree might not be confident enough about the effectiveness of the observed values so far.

You could either:

• Process more samples.
• Increase split_confidence to, let say, 0.01 or even 0.1. Hence, the tree will perform splits even when it's "not sure" about how good they are.

Please, confirm if splits are triggered under these conditions. If not, it may be indeed something wrong going on.

[smastelini]

I ran your code and can confirm that the splits happen if more instances are processed. :D

[smastelini]

By the way, the bottleneck in your example is the prediction portion. The model is applied to the whole dataset after each processed instance (both past and upcoming samples). This leads to data leakage (instances utilized for learning are also used for prediction) and slows down your code.

I suggest using our suite of incremental metrics as well as river.stream.iter_pandas (or other utilities from the stream submodule). Probabilities and predictions for a given sample can be retrieved directly by using model.predict_proba_one and model.predict_one, respectively :)

[HarshaAsh]

Hi @smastelini , thanks for the reply. Could you also kindly explain the increase in accuracy even when no tree is built? I am unable to understand that. I was under the assumption that some trees were being built (but not displayed) as accuracy was improving.

[HarshaAsh]

@smastelini , also I was able to get the tree built by increasing the split_confidence. Thanks for the help. Even in this scenario, I can see that the accuracy improves significantly even before the tree is built.

[smastelini]

Hi @smastelini , thanks for the reply. Could you also kindly explain the increase in accuracy even when no tree is built? I am unable to understand that. I was under the assumption that some trees were being built (but not displayed) as accuracy was improving.

Hi @HarshaAsh.The tree is built in all the cases! Differently from batch decision trees, Hoeffding Trees are constructed incrementally. They start with a single leaf that is constantly updated until enough evidence is gathered to enable splits. The accuracy improves because the leaf statistics are updated with each new observed instance. By default, the classification trees use adaptive Naive Bayes (ANB) models at the leaves to provide predictions. The ANB prediction works as follows:

• The number of "hits" and "misses" for both the Naive Bayes predictor and the majority class predictor is kept by each leaf.
• Each time predict[_proba]_one is called, the reached leaf checks:
  • if Naive Bayes correctly classified more instances the majority class estimator, then Naive Bayes is applied
  • the Majority class predictor is used otherwise

You can also set up the trees to always use either Naive Bayes or the majority class by changing the parameter leaf_prediction.

The majority class is defined by simply keeping class counts. The Naive Bayes predictor, on the other hand, relies on the attribute splitters to calculate the probability estimates. You can check the documentation on tree splitters to get more information about these crucial tree components :D