FEA Add staged_predict to HistGradientBoosting

[haochunchang]

Reference Issues/PRs

Try Fixes #16063

What does this implement/fix? Explain your changes.

Add staged_predict by emulating BaseGradientBoosting. Including private functions.

• Add associated tests
• Update documentation (Currently, it was copied from BaseGradientBoosting.)
• Also add to HistGradientBoostingClassifier

Any other comments?

This is my first PR for a feature request.
I would be very appreciated for any advices or collaborations.

[NicolasHug]

thanks for the PR @haochunchang , please ping me when you are ready to get reviews or if you need any help

[haochunchang]

I have also checked with an example: test_staged_predict_example.txt (adopted from here).
Although I did not convert it into a jupyter notebook with binder, the plot seems reasonable.

In short, I moved part of the original _raw_predict to _predict_stages.
And staged_predict calls _staged_raw_predict that uses _predict_stages by supplying part of predictors_.

I wasn't sure whether the change is what staged_predict want, but it seems it can incrementally predict the target.

I would be very appreciated for @NicolasHug or anyone who reviews this code and gives any directions for checking :)

[NicolasHug]

Thanks @haochunchang , it looks in good shape!

I made a few comments, mostly on the style.

Do you also want to add support for the classifier?

[NicolasHug]

About the tests, should we add separated tests like

hmm, use your best judgement. It'd be fine eitherway. Also note that we don't need the first one if you do the second

[haochunchang]

For simplicity, I added only one test for testing staged_predict.
It tests both Classifier and Regressor by parametrize:

• Predictions at each stage equals newly-trained predictions (also test max_iter = 1).
• The number of iterations in staged_predict is equal to gb.n_iter_.

[haochunchang]

I would be very appreciated for @NicolasHug or anyone who reviews this code and gives any directions for checking :)

[NicolasHug]

thanks for your patience @haochunchang , we've been busy with the release

This mostly looks good, I'd just suggest to also add staged_predict_proba and staged_decision_function which should be pretty easy now that you have it all laid out.

[haochunchang]

Good effort for the release! @NicolasHug
I have also added staged_predict_proba and staged_decision_function and applied your valuable suggestions.
Thanks for your time :)

[NicolasHug]

thanks @haochunchang , almost there

[NicolasHug]

thanks @haochunchang

[NicolasHug]

maybe @amueller @thomasjpfan can review this one?

[NicolasHug]

Thanks @haochunchang !

Sorry we forgot to mention earlier: we'll need a what's new entry:

Please add an entry to the change log at doc/whats_new/v024.rst. Like the other entries there, please reference this pull request with :issue: and credit yourself with :user:. Also since it's new you'll need to create a :mod:sklearn.ensemble section.

You can take inspiration from the 0.23 what'snew if you need

[NicolasHug]

Thanks for the work and for your patience @haochunchang !

Just a side note for you next contributions: you don't need to force-push changes, we're always going to squash and merge anyway. When you force push it makes things a little bit harder for us if we have already checked-out your branch ;)

[haochunchang]

Got it!
Thanks @NicolasHug and @thomasjpfan for the review and all the suggestions :)