[MRG] FIX solve consistency between predict and predict_proba in AdaBoost

[glemaitre]

closes #14084
closes #2974

compute the probabilities in AdaBoostClassifier as specified in "Multiclass AdaBoost"

[glemaitre]

@NicolasHug @amueller I am playing with something which is not my strong suit. Could you have a look at the PR and if the fix proposed seems theoretically reasonable?

[NicolasHug]

As far as I can tell the changes are correct.

What is strange to me is that the previous implementation is also a softmax of the decision function, so other than the fact that the new version is much clearer, I don't understand where the fix is.

(to be honest this isn't really my specialty either)

[NicolasHug]

can you explain this? is it in the paper too?

[glemaitre]

My original thought was that we were keeping one of the 2 columns when computing the decision function. Actually, we do something a bit different:

        if n_classes == 2:
            pred[:, 0] *= -1
            return pred.sum(axis=1)

You originally have symmetry between both classes. However, I should divide decision by 2 since that we are summing.

[glemaitre]

What is strange to me is that the previous implementation is also a softmax of the decision function, so other than the fact that the new version is much clearer, I don't understand where the fix is.

It uses the predict_proba of the underlying classifier instead of the predict (used in the decision function). It is the main difference.

[glemaitre]

@NicolasHug do you have any other comments?

Any second reviewer? @rth @thomasjpfan

[thomasjpfan]

Strange how master's implementation used predict_proba instead of predict which is more inline with the paper's use of missclassification error rate.

[glemaitre]

@thomasjpfan I addressed the comments.

[thomasjpfan]

LGTM

[agramfort]

thx @glemaitre