-
-
Notifications
You must be signed in to change notification settings - Fork 8.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How does xgboost handle instance weights #144
Comments
This is a good question:) The instance weights are blended into gradient and second order gradients. Think about how will you calculate the g and h when you have weighted loss functions. |
Think in case weights as repeated cases, so calcGain = Sqr(case_weight * sum_grad) / (case_weight * sum_hess + reg_lambda); See the R gbm implementation (point 4.3, terminal node estimates): |
In xgboost, when you feed things into booster, for each instance i grad_xgb[i] = grad[i] * case_weight[i] grad_xgb, and hess_xgb is the statistics you feed into tree constructor. In another word, case_weight is "blended" into the gradient statistics already |
In your simplified case, sum_grad *case_weight is the actual sum_grad the calcGain function will see |
Interestingly, this question is also covered in this slide http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf See the last few slide about questions on weigthted training |
I've searched in code and didn't found the weights in grad and hessian. I'll take a second look. |
You can checkout here |
Thank you. Seems OK and your terminal node estimates are congruent with given in R gbm document I posted yesterday. A last thing: |
sum_hess is also computed with the weights. |
@tqchen |
Hi,
I think case weights should be present in calcGain and calcWeight functions in any way (multiplying to the grad and hess).
Where are used the case weights?
The text was updated successfully, but these errors were encountered: