[R-package] Fix best_iter and best_score

[Laurae2]

This should fix #2158 and #2029.

"best" rule:

• Take the first iteration where the best score is attained, even if the further iterations have identical scores (do not take the last)
• When there is no scoring done, the best score becomes NA and the best iteration is -1 (currently)

Later, we should enforce the metric used for early stopping should be only the first one, or at worst give the user the ability to choose the metric (best is the first).

note: @jameslamb spaces from RStudio to fix

Example (change to metric = "auc" and max_depth = 3 to test maximization):

library(lightgbm)
data(agaricus.train, package = "lightgbm")
train <- agaricus.train
dtrain <- lgb.Dataset(train$data, label = train$label)
data(agaricus.test, package = "lightgbm")
test <- agaricus.test
dtest <- lgb.Dataset.create.valid(dtrain, test$data, label = test$label)
params <- list(objective = "regression", metric = "l2")
valids <- list(test = dtest)
model <- lgb.train(params,
                   dtrain,
                   100,
                   min_data = 1,
                   learning_rate = 0.5)
model$best_score
model$best_iter
# > model$best_score
# [1] NA
# > model$best_iter
# [1] -1

model <- lgb.train(params,
                   dtrain,
                   100,
                   valids,
                   min_data = 1,
                   learning_rate = 0.5)
model$best_score
model$best_iter
# > model$best_score
# [1] 7.497024e-62
# > model$best_iter
# [1] 100

model <- lgb.train(params,
                   dtrain,
                   100,
                   valids,
                   min_data = 1,
                   learning_rate = 0.5,
                   early_stopping_rounds = 10)
model$best_score
model$best_iter
# > model$best_score
# [1] 7.497024e-62
# > model$best_iter
# [1] 100

model <- lgb.cv(params,
             dtrain,
             10,
             nfold = 5,
             min_data = 1,
             learning_rate = 1)
model$best_score
model$best_iter
# > model$best_score
# [1] 0.0003072197
# > model$best_iter
# [1] 1

model <- lgb.cv(params,
             dtrain,
             10,
             nfold = 5,
             min_data = 1,
             learning_rate = 1,
             early_stopping_rounds = 5)
model$best_score
model$best_iter
# > model$best_score
# [1] 5.369631e-18
# > model$best_iter
# [1] 3

[StrikerRUS]

Later, we should enforce the metric used for early stopping should be only the first one, or at worst give the user the ability to choose the metric (best is the first).

There are some problems with metrics' order in Python: #2127. I suppose the same is true and for R too.

[Laurae2]

@StrikerRUS In R lists, it is fixed. The first element always remains the first element. You may even have two elements having the same exact names without any conflict.

However users may provide the same metric multiple times by mistake, in that case we deduplicate them.

Example:

library(lightgbm)
data(agaricus.train, package = "lightgbm")
train <- agaricus.train
dtrain <- lgb.Dataset(train$data, label = train$label)
data(agaricus.test, package = "lightgbm")
test <- agaricus.test
dtest <- lgb.Dataset.create.valid(dtrain, test$data, label = test$label)
params <- list(objective = "regression", metric = c("l2", "l1", "l2"))
valids <- list(test = dtest)

model <- lgb.train(params,
                   dtrain,
                   100,
                   valids,
                   min_data = 1,
                   learning_rate = 0.5)
str(model$record_evals$test, max.level = 1)
# > str(model$record_evals$test, max.level = 1)
# List of 2
#  $ l2:List of 2
#  $ l1:List of 2

[guolinke]

@Laurae2 will it be better to have a parameter named "first_metric_only" in R as well?

[guolinke]

BTW, if both R and python have "first_metric_only" option, I think we should have the same option in CLI version.

[Laurae2]

@guolinke Yes, I think we should have first_metric_only option for CLI, R, and Python.

As the handling would be different for each wrapper, R and Python would have their own implementations using callback.

[Laurae2]

@StrikerRUS do you know why Travis MPI/Python jobs are failing?

[StrikerRUS]

@Laurae2 gcc 9 has been released recently. Hotfix is already in master: abbbbd7.

[guolinke]

Later, we should enforce the metric used for early stopping should be only the first one, or at worst give the user the ability to choose the metric (best is the first).

@Laurae2 is this implemented in this PR?

[guolinke]

also refer to this:
#2127 (comment)
we should ensure that cli, python and R have the same behavior with first_metric_only.

[Laurae2]

@guolinke This PR uses all metrics. We can do first_metric_only in another PR.

Note that the best score / iteration is taken from the first metric when it was not computed by the model.

[Laurae2]

Closing/reopening for CI

[StrikerRUS]

@Laurae2

Closing/reopening for CI

This doesn't work for branch's CIs, it works only for PR's CIs.

[Laurae2]

@jameslamb do we merge it as is for the moment?

[StrikerRUS]

I've updated the branch. Now all checks should be OK.

[jameslamb]

Looks good to me! Thank you @Laurae2 , apologies for my delayed review