Loading…
Addressing issue #1802 (Gradient Boosting Out-of-bag Estimates) #1806
5 participants
Do you have a better idea?
Don't take it the wrong way. Thank you for your contribution!
The thing is, I am not sure we can have non-biased oob estimates at all from boosting algorithms. We should dig in the literature to see whether it can be done.
No worries, no offence taken :-)
I completely agree that if there is a better way to generate OOB estimates, it should be used. I couldn't find anything in the literature, but admittedly I mostly skimmed the relevant (major) publications.
In any case, with the current OOB score implementation, it seems like it improves indefinitely (like the in-bag scores), at least based on my limited testing. So it's not very useful as a way of tuning n_estimators.
I think the current implementation should definitely be removed. There are no "out of bag" estimates here as there are no bags imho. But maybe there is a way to get some estimate via the subsampling.
I think the current implementation should definitely be removed. There are no "out of bag" estimates here as there are no bags imho. But maybe there is a way to get some estimate via the subsampling.
Agree.
@amueller I disagree, there are "bags" - still the oob estimates might be fundamentally flawed due to the fact that we already peeked at the out-of-bag samples (via our old iterations).
thinking aloud: I totally see a problem when using OOB estimates as a proxy for generalization error - but they might be interesting for model selection -- in particular, for selecting the number of boosting iteration or for early stopping.
as @yanirs already said, there is no account for out-of-bag estimates in the primary literature but its featured in R's gbm and ada vignettes.
even Wikipedia discusses OOB for gradient boosting; it references the Stochastic Gradient Boosting paper by Friedman which doesn't discuss OOB as far as I know.
@pprett Maybe I should have said "gradient boosting is not bagging".
Bagging = bootstrap aggregation, i.e. averaging over bootstrap samples.
If you don't do bagging, I don't think is is appropriate to talk about bags ;)
(apparently the author of gbm disagrees with me on the last point and calls the individual samples in each round bag).
So how is it implemented in gbm? Just using the trees that don't use the sample?
in gbm, the oob estimate of the i-th iteration is just the loss (=deviance) of the out-of-bag samples of the current (the i-th) tree. this is the same what we have in sklearn.
That sounds like a horrible idea. Interesting.
Seriously? After agreeing that the current OOB implementation is broken you're just going to leave it as it is? At the very least, it's worth noting this issue in the documentation to avoid misleading users. I spent quite a bit of time digging through the code to figure out why the OOB scores never stop improving, and I would like other users to avoid going down the same path.
If I had time, I would run some more experiments to demonstrate that my solution works (I only tested it on one dataset: https://www.kaggle.com/c/bluebook-for-bulldozers/forums/t/4368/congratulations-to-the-preliminary-winners/23125#post23125), which I think is a more reasonable way forward than copying the bugs in R's gbm implementation.
This still sounds as a terrible idea to me to skip trees, in the sense that I really don't understand what your "oob estimates" actually represent. Even if it this biased, the current oob implementation at least makes sense.
But okay, let's reopen this and continue the discussion.
I think the best way to move forward is to compile a test suite with benchmark datasets (e.g. hastie 10_2, spam, cal_housing) and compare the OOB estimator for n_estimators against a CV estimator.
As far as I understand @yanirs correctly, the issue is that OOB score always improves and thus over-estimates the optimal number of boosting iterations. G. Ridgeway actually claims the opposite: "OOB underestimates the optimal number of iterations".
In boosting, the only reliable way I see to have a non-biased approximation of the generalization error is to put aside a validation set and evaluate the model on it after training.
For a given sample x_i, it makes no sense in my opinion to only consider those trees that have not been trained on x_i. The prediction is a sequential process, not a vote. I don't understand why you could skip steps in that process.
Yes @pprett, from my testing oob_score_ behaves like train_score_, and having oob_score_ is misleading because it's not really oob.
I agree with @glouppe that the best approach is to use an external validation set, but this is not always practical. Without further testing, I'm not confident that the approach of skipping trees is always correct, but it makes more sense to me than the current oob_score_ implementation. So I think that the minimal solution is to note this issue in the documentation until further testing is performed (I may end up running more experiments, but I'm a flooded with work at the moment).
So I think that the minimal solution is to note this issue in the documentation
I agree. We should indeed at least says that it is heavily biased, nearly as much as the training error.
I had a closer look at the issue - in particular, the GBM implementation of OOB estimates to compute the "optimal" number of iterations.
GBM does not record the OOB score per se but the improvement in deviance on the OOB samples by adding the predictions of the current tree to the predictions we have so far.
In order to estimate the optimal number of iterations GBM smooths the improvement scores using LOESS, computes the cumulative sum and picks the max**
The relevant functions are src/distribution.h:BagImprovement and gbm.perf.R:gbm.perf#20 .
** GBM actually picks the min of the negative cumsum (see code referenced above)
We should definitely warn. Maybe we should even rename? This really is no oob estimate (with either "method").
I have to read up on what Peter said to understand what is going on in gbm.
I finally got around to testing my tree-skipping approach on multiple datasets: http://yanirseroussi.com/2014/12/29/stochastic-gradient-boosting-choosing-the-best-number-of-iterations/
I agree with the decision not to merge the original pull request, but I was thinking that it may be useful to others if I contributed my experimental code to extend the example in http://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_oob.html to real datasets. Thoughts?
Thanks Peter! Good point about adding a notebook to the post. Notebooks do tend to be more readable than plain Python files.
Your blog engine might also have a plugin to directly turn notebooks into blog posts:
Or you can use nbconvert to script the conversion:
-
…ged oob_score_ to be calculated based only on trees where the OOB instances weren't used for training
Changed oob_score_ to be calculated based only on trees where the OOB instances weren't used for training.
Not entirely sure about the correctness of doing it this way, but it seems to make the OOB scores more reliable (based on local testing).