-
-
Notifications
You must be signed in to change notification settings - Fork 8.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Intuition behind hessian and min_child_weight #2483
Comments
benoitdescamps, I read this article and know these formulas but they don't answer my questions. |
@sergun while I agree that the PDF does not magically give all the answers, the answer to the (3) is there, see eq. 7. |
@superboby right, I saw the equation bu exact role is not clear. We can have two split candidates. Both with the same Gleft^2, Gright^2, (Gright+Gleft)^2 but differ in Hleft and Hright so one split will be better. It's not clear what is the meaning of such normalization by hessian during split find. It's complitely clear for RMSE loss when H is equal to number of observations but not clear for logloss when H is direvative of sigmoid |
@khotilov Thanks! |
In the case of binary classification, you might assume that each observations are i.i.d. Bernouilli B(1-p). Recall that Bernouilli has mean 1-p and variance p(1-p). In the case of logloss notice that these are proportional to the gradient and Hessian respectively. Have a look at equation(5) in the paper. The weight is the ratio of the averaged probability of each observation, hence roughly the average probability of the leaf up to a factor, divided by the sum of the variance of each observation, hence the variance of the probability of the leaf up to a factor. So basically when boosting, you are updating the probabilities of each observations with the Z-score of the new partition they belong to. I do not know, if this helps, but if your intuition for RMSE satisfies you, then this should be enough. |
I tried to understand the role of hessian and min_child_weight but not everything is clear from the documentation.
The text was updated successfully, but these errors were encountered: