handle NULL objective in xgb_predict()

[simonpcouch]

This PR ought not to be merged before the upcoming 1.0.4 release.

Closes #873!

This PR:

• Handles NULL objectives in xgb_predict()
• Tests an additional type of alternative objective and tests each alternative objective not only for fitting but for predicting

The reason we need to be able to handle NULL objectives is because the xgb.Booster object will not carry along the objective argument in its params slot if's not one of the pre-defined options. Using the example fit and custom objective fn from the xgb.train() examples:

library(tidymodels)
library(xgboost)

data(agaricus.train, package = "xgboost")
data(agaricus.test, package = "xgboost")

dtrain <- with(agaricus.train, xgb.DMatrix(data, label = label, nthread = 2))
dtest <- with(agaricus.test, xgb.DMatrix(data, label = label, nthread = 2))
watchlist <- list(train = dtrain, eval = dtest)

The objective parameter is retained, usually:

param <- list(max_depth = 2, eta = 1, nthread = 2,
              objective = "binary:logistic", eval_metric = "auc")
bst <- xgb.train(param, dtrain, nrounds = 2, watchlist)
#> [1]  train-auc:0.958228  eval-auc:0.960373 
#> [2]  train-auc:0.981413  eval-auc:0.979930

bst$params$objective
#> [1] "binary:logistic"

# But is not brought along when a custom objective function is used:
logregobj <- function(preds, dtrain) {
  labels <- getinfo(dtrain, "label")
  preds <- 1/(1 + exp(-preds))
  grad <- preds - labels
  hess <- preds * (1 - preds)
  return(list(grad = grad, hess = hess))
}

param$objective <- logregobj

bst <- xgb.train(param, dtrain, nrounds = 2, watchlist)
#> [1]  train-auc:0.958228  eval-auc:0.960373 
#> [2]  train-auc:0.981413  eval-auc:0.979930

bst$params$objective
#> NULL

The "more minimal" reprex from the original issue, with this PR installed:

res <-
  fit_resamples(
    boost_tree("classification") %>% set_engine("xgboost", objective = logregobj),
    Class ~ .,
    bootstraps(two_class_dat, 5)
  )

collect_metrics(res)
#> # A tibble: 2 × 6
#>   .metric  .estimator  mean     n std_err .config             
#>   <chr>    <chr>      <dbl> <int>   <dbl> <chr>               
#> 1 accuracy binary     0.805     5 0.00676 Preprocessor1_Model1
#> 2 roc_auc  binary     0.865     5 0.00684 Preprocessor1_Model1

^{Created on 2023-02-21 with reprex v2.0.2}

[simonpcouch]

Does this do the trick for you, @SHo-JANG? You can install and test by running pak::pak("tidymodels/parsnip#875"), restarting R, and running your reprex.

[SHo-JANG]

It works surprisingly well.
Could you modify this method so that it can be applied to other models? such as Lightgbm and so on.

Thank you so much!😂

[simonpcouch]

Glad this is helpful for you! xgb_predict() is intended as a light wrapper around xgboost's predict method, so I think the scope of usage for this function will stay the same for now.

[hfrick]

nice!

[github-actions]

This pull request has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.