FIX several bugs in initial predictions for GBDT

[NicolasHug]

Reference Issues/PRs

Closes #12436 , continuation of the work from @jeremiedbb

What does this implement/fix? Explain your changes.

This PR:

• Fixes the use of the init estimator parameter in GBDTs
• exposes init='zero' which was supported but undocumented.
• deprecates all the losses and init estimators in gradient_boosting.py
  • the loss classes have been moved to a losses.py file and have a new get_init_raw_predictions method
  • the init estimators are replaced by an equivalent instance of DummyClassfier or DummyRegressor
• renames y_pred into raw_predictions, and relevant (private) methods as well.
• fixes some the inaccurate docstrings
• "Fixes" the initial default prediction for multiclass classification: the raw prediction used to be the classes priors, but it should be the log of the priors. In practice the difference in accuracy is very minimal: over 100 runs, I observed an avg improvement in test accuracy of 0.000388 (std = 0.008266) between the 2 methods, with n_estimators=1, n_samples=10000.

The loss.get_init_raw_predictions(X, predictor) methods return a raw_prediction with the correct shape (n_samples, K) where K is the number of classes in multiclass classification, else 1.

Those raw_predictions are homogeneous to what the trees predict, not homogeneous to what the ensemble predicts. For regression GBDT this is the same thing, but for e.g. binary classification raw_prediction is homogeneous to a log-odds ratio.

For more, see #12436 (comment) and following discussion (not sure if this is much clearer ^^)

Any other comments?

[glemaitre]

Partial review

[glemaitre]

You can add a FIXME:

[glemaitre]

I think that we are using something like that to check the support for sample_weight:

support_sample_weight = has_fit_parameter(self.init_, "sample_weight")
    if not support_sample_weight and not sample_weight_is_none:
        raise ValueError("xxxx")

[glemaitre]

are we using the double or single backticks?

[NicolasHug]

I think we use single backticks for anything that has a glossary entry (that will automatically link it) and double backticks for everything else (to avoid shpinx warnings)

[thomasjpfan]

@glemaitre Once #13000 gets merged, we do not need to worry about this anymore! Single backticks without a reference will automatically be treated as double backticks!

[glemaitre]

I think that you can remove this comment. Strategy quantile with 0.5 should be self explicit (and this is the Least Absolute Error) as well

[glemaitre]

Another partial review

[glemaitre]

is it array-lie?

[glemaitre]

self.class.name, this might be the same actually

[NicolasHug]

Is the problem merely that you don't know how to align the prediction to the validation data?

No the problem isn't related to the validation data. It's related to the training data.

---

Basically the raised ValueErrors in e2e1a5c may happen in 2 scenarios:

• User has passed an init estimator that was trained on a different number of classes than what the GBDT. In this case the error message is appropriate, since this is the user's fault.
• User has passed an init estimator, or is using the default init estimator, AND uses early stopping. If one of the classes is not represented in the actual training data (after train_test_split), the init predictor is going to be trained on this data and init_predictor.predict_proba() will be of shape (n_samples, n_classes - 1), but the GBDT expects it to be of shape (n_samples, n_classes). In this case the error message is not helpful because it doesn't tell the user what to do. In this case we can either:
  • Raise an error with a more helpful message (I think that using stratify in train_test_split would be the right solution)
  • Not raise an error, try to find which of the classes has been ignored, and reconstruct a valid probas array. That might be tricky and will definitely make the code more complex than it needs to be.

I'm sorry I find it hard to explain clearly and concisely.

I think that using stratify in train_test_split would solve the issue because train_test_split raises an error when one of the classes are missing in y_train. Regardless of this issue, I think that early stopping splits should be stratified anyway. Would that be OK to use it and consider it as a bugfix?

[jnothman]

The init predictor's output can be fixed by aligning it up through classes_ I think. But stratifying does seem a reasonable fix. Does this happen elsewhere in the library with validation splits?

[NicolasHug]

The init predictor's output can be fixed by aligning it up through classes_

Yes but we still need to identify which of the classes are missing in y_train, and the loss is not supposed to know anything about y_pred at this point (only its init estimator knows about it, but itself cannot know which classes are missing). That's what I mean by tricky / overly complicated.

Does this happen elsewhere in the library with validation splits?

I don't think the same issues happens in other estimators. This issue is happening because our main estimator (GBDT) relies on another estimator (the init estimator) which is passed only a subset of the data where some classes might be missing, because of early stopping. GBDTs are the only estimators that use early stopping + another estimator, as far as I know.

I took a look at different early stopping implementations:

• All estimators inheriting from BaseSGD (perceptron, passiveagressive, SGDClassifier) use stratified splits.
• Multilayer perceptron and gradient boosting do not use stratifies splits.

---

Would it be acceptable that I open a new PR to "bugfix" multiplayer perceptron and gradient boosting by making the splits stratified, and wait for it to be merged?

[amueller]

Would it be acceptable that I open a new PR to "bugfix" multiplayer perceptron and gradient boosting by making the splits stratified, and wait for it to be merged?

fine with me.

[glemaitre]

I am still +1 here

[jnothman]

Thanks for a phenomenal effort, @NicolasHug!

[agramfort]

awesome ! thanks heaps @NicolasHug <https://github.com/NicolasHug>!

…