Double checking min_split_loss in bag and out of bag

[jaguerrerod]

I'm interested in filtering split candidates based on their split gain out of sample, assuming a subsample less than 1. The aim is to consider only those splits where both the gain in sample and out of sample are greater than the specified minimum split loss (gamma). Can anyone provide guidance on how to achieve this?

[trivialfis]

I don't think we have a way to evaluate based on out-of-bag samples at the moment.

[jaguerrerod]

I think the way to do this is to simultaneously build a tree with the out-of-sample data and apply each split to it. For the new split candidates in the tree built with the sample data, evaluate gain reduction in the out-of-sample tree and choose the candidate that maximally improves in-sample, among those for which gamma is also greater than X out-of-sample.
I think this double-checking is very useful in high noise / low signal problems.
I don't know how hold predictions are done, if doing something similar (building hold tree in parallel) or predicting hold with each new tree.

[trivialfis]

For statistical inference algorithms like gradient boosting, the standard way is simply to run cross-validation, which is reasonably effective. GBM is built upon the concept that we can stack many weak models to obtain a strong model, instead of preventing the tree from splitting, you can simply train small/shallow trees and try to prevent XGBoost from adding more trees, at least in theory this should have the similar effect as your feature request.

[jaguerrerod]

Yes, that's the theory behind gradient boosting and it works very well in almost all contexts. The scenario where there is an extremely low signal-to-noise ratio is different. Let me justify why I am interested in this functionality.

I am referring to datasets for stock market prediction, with level of rank correlation is on the order of 0.02 - 0.03.

At the same time, these predictions exhibit extreme variability. For example, changing the starting seed without changing any other parameter produces changes on the order of 15% in Spearman correlation. This can give us the noise level.

In this scenario, every time a model fits signal with a weak model, that signal becomes unrecoverable later by another weak model because its level is so low that once the information is fragmented, it dissipates into noise.

That's why I believe that a double-check is very important before using ('destroying') the signal in a tree (weak model). I may be wrong, but my intuition tells me that by preserving the signal and using it only in splits that are useful in both a sample and out-of-sample double-check, better results will be achieved than by adding new trees that won't be able to extract the remaining little signal diluted in the noise.

[trivialfis]

Hi, this is the line for split enumeration on CPU

xgboost/src/tree/hist/evaluate_splits.h

Line 304 in 1450aeb

static_cast<float>(evaluator.CalcSplitGain(*param_, nidx, fidx, GradStats{left_sum},

It might take some time to get around it and enable the feature you want. We implement sampling by setting the hessian to zero, which is quite different from your feature request.