-
-
Notifications
You must be signed in to change notification settings - Fork 8.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Base score for custom multi-output objects is hard-coded at 0.5 #9840
Comments
Indeed, when a custom objective is used, the intercept cannot be fitted by XGBoost since the intercept is fitted according to the objective. For instance, there is a close solution for MAE (the median). Users need to set the |
But in that case, why not set it to zero instead of 0.5? Or why not add a parameter (ideally turned on by default) to try to obtain it through Newton steps if |
I think the 0.5 choice was due to logistic regression with sigmoid. It's an old default that nobody touches. I added the Newton step in the recent releases, but did not change the default values when the Newton step is not applicable. In summary, "historical reasons". Your suggestion is interesting, I can look into intercept fitting with custom objective by using one step Newton in the future. |
As a quick workaround, wouldn't it be better to at least leave it at zero? In the case of logistic regression with sigmoid, that's what would output a probability of 0.5, as opposed to a raw score of 0.5 which is a probability of 0.62. |
The inverse of the sigmoid is logit, which turns 0.5 into 0.0. That's my guess anyway.
I think the default value probably has very little meaning for regression output where the mean can be anything. I can work on custom objectives. |
EDIT: sorry, first version of this issue was wrong, I've now edited it.
When using multi-output objectives with multi-output trees, the intercept / base score / first tree seems to start at 0.5 regardless of what the labels are.
In this example, I am passing a multi-output label in which the second column is the negative of the first one:
Both have the same score of 0.5, which doesn't look like it'd be a better choice than zero, for example.
The text was updated successfully, but these errors were encountered: