How does xgboost handle instance weights

[BlindApe]

Hi,

I think case weights should be present in calcGain and calcWeight functions in any way (multiplying to the grad and hess).
Where are used the case weights?

[tqchen]

This is a good question:) The instance weights are blended into gradient and second order gradients. Think about how will you calculate the g and h when you have weighted loss functions.

[BlindApe]

Think in case weights as repeated cases, so

calcGain = Sqr(case_weight * sum_grad) / (case_weight * sum_hess + reg_lambda);
calcWeight = - case_weight * sum_grad / (case_weight * sum_hess + reg_lambda);

See the R gbm implementation (point 4.3, terminal node estimates):

https://r-forge.r-project.org/scm/viewvc.php/*checkout*/pkg/inst/doc/gbm.pdf?revision=18&root=gbm&pathrev=22

[tqchen]

In xgboost, when you feed things into booster, for each instance i

grad_xgb[i] = grad[i] * case_weight[i]
hess_xgb[i] = hess[i] * case_weight[i]

grad_xgb, and hess_xgb is the statistics you feed into tree constructor. In another word, case_weight is "blended" into the gradient statistics already

[tqchen]

In your simplified case, sum_grad *case_weight is the actual sum_grad the calcGain function will see

[tqchen]

Interestingly, this question is also covered in this slide http://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf

See the last few slide about questions on weigthted training

[BlindApe]

I've searched in code and didn't found the weights in grad and hessian. I'll take a second look.

[tqchen]

You can checkout here
https://github.com/tqchen/xgboost/blob/master/src/learner/objective-inl.hpp#L149

[BlindApe]

Thank you. Seems OK and your terminal node estimates are congruent with given in R gbm document I posted yesterday.
'j = i % nstep' line I suppose is for multinomial loss where there are k preds for each label, ok?

A last thing:
sum_hess in weighted instances is computed in same way? How does this affect the min_child_weight restriction? If all instances have big weights this restriction should be manually incresed or this sum_hess is computed without the weights?

[tqchen]

sum_hess is also computed with the weights.

[vatsan]

@tqchen
For a user supplied objective, should we manually blend the weights with the gradients & hessian? Sorry i am not able to verify this myself as the link you pointed to doesn't work anymore: (https://github.com/tqchen/xgboost/blob/master/src/learner/objective-inl.hpp#L149).