Fix huber loss

[henry0312]

TODO:

• merge and rebase Add L1 objective function #175

[msftclas]

Hi @henry0312, I'm your friendly neighborhood Microsoft Pull Request Bot (You can call me MSBOT). Thanks for your contribution!
You've already signed the contribution license agreement. Thanks!

The agreement was validated by Microsoft and real humans are currently evaluating your PR.

TTYL, MSBOT;

[guolinke]

merge these two if?

[henry0312]

I'm sorry, I don't understand what you mean 😢
This PR will be merged after #175

[guolinke]

change

if () 
...
else if()
...

to

if()
...

. They are all regression objective, right?

[henry0312]

yes, they are.

you mean,

  if (type == std::string("regression") || type == std::string("regression_l2") || type == std::string("mean_squared_error") || type == std::string("mse")) {
    return new RegressionL2loss(config);
  }

right?

[guolinke]

yes. and you can break if() to two line:

  if (type == std::string("regression") || type == std::string("regression_l2") 
      || type == std::string("mean_squared_error") || type == std::string("mse")) {
    return new RegressionL2loss(config);
  }

[guolinke]

I think you can use an inline function to calculate hessian and avoid these repeated code.

[guolinke]

same as above, and you can put this function into utils/common.h

[henry0312]

ok!

[henry0312]

rebased master.
However, there may be something wrong with ApproximateHessianWithGaussian.
I'll check.

[Laurae2]

@henry0312 This topic has a bunch of Gradient/Hessian functions for helping to converge for a L1 loss function when a hessian is mandatory for convergence.

They were all tested in xgboost and are all converging fast. Without a hessian, the convergence is slow (which seems to be the case in LightGBM also as @guolinke mentioned in the previous PR for L1 loss). It would be good to have an example to check the convergence speed of the L1 loss.

The short "explanation" is that when a hessian is mandatory for good convergence and we want to use a L1 loss function, we should avoid at any cost a constant hessian. To fix this issue, we smooth the linear function using a curve so the hessian is never zeroed out (better and faster convergence speed). For instance, the Fair objective (from a Microsoft research) is used to approximate such L1 function requiring a hessian.

The R versions with their gradient/hessian formulas are below:

ln_cosh_obj <- function(preds, dtrain) {
  labels <- getinfo(dtrain, "label")
  grad <- tanh(preds-labels)
  hess <- 1-grad*grad
  return(list(grad = grad, hess = hess))
}

ln_expexp_obj <- function(preds, dtrain) {
  labels <- getinfo(dtrain, "label")
  x <- preds-labels
  grad <-  (exp(2*x)-1) / (exp(2*x)+1)
  hess <-  (4*exp(2*x)) / (exp(2*x) + 1)^2 
  return(list(grad = grad, hess = hess))
}

cauchyobj <- function(preds, dtrain) {
  labels <- getinfo(dtrain, "label")
  c <- 2  #the lower the "slower/smoother" the loss is
  x <-  preds-labels
  grad <- x / (x^2/c^2+1)
  hess <- -c^2*(x^2-c^2)/(x^2+c^2)^2
  return(list(grad = grad, hess = hess))
}

fairobj <- function(preds, dtrain) {
  labels <- getinfo(dtrain, "label")
  c <- 2  #the lower the "slower/smoother" the loss is
  x <-  preds-labels
  grad <- c*x / (abs(x)+c)
  hess <- c^2 / (abs(x)+c)^2
  return(list(grad = grad, hess = hess))
}

[henry0312]

@Laurae2 the hessian of mae is delta function, therefore, if we can approximate it with something appropriate (e.g. gaussian function), converge and performance will be good, I guess.

[guolinke]

where is the define of M_PI ? I cannot compile it on windows.

[henry0312]

Could you add #include<cmath>?

[guolinke]

it seems need to add #define _USE_MATH_DEFINES in windows:
http://stackoverflow.com/questions/6563810/m-pi-works-with-math-h-but-not-with-cmath-in-visual-studio

[henry0312]

probably fixed with 05e97f1

[henry0312]

I finished fixing ApproximateHessianWithGaussian (also Huber loss).
L1 loss is approximated with Gaussian function like http://www.wolframalpha.com/input/?i=Integrate%5BErf%5Bx%5D%5D,+Integrate%5BErf%5Bx%2F10%5D%5D.

[henry0312]

Next, I will create Fair loss in an another PR, which @Laurae2 tells me.