Sparse matrix conversion not working with XGboost

[tolliam]

I have attempted to alter the example in this tidyverse blog post to use boost_tree with xgboost rather than logistic_reg with glmnet. I am keen to get the speed boost which comes with passing to XGBoost a sparse matrix rather than tibble. However what I find is that the sparse matrix version of the workflow is not returning the same predictions as the tibble version, unlike in the lasso example. The predictions coming out of the sparse matrix version are worse and I suspect systematically wrong.

I wonder if this problem relates to translating factor levels or something like that? I actually discovered this problem first using my own data set but it also occurs with the small_fine_foods dataset.

The code below is very similar to what is in the blog.

library(tidyverse)
library(tidymodels)

data("small_fine_foods")
training_data
#> # A tibble: 4,000 × 3
#>    product    review                                                       score
#>    <chr>      <chr>                                                        <fct>
#>  1 B000J0LSBG "this stuff is  not stuffing  its  not good at all  save yo… other
#>  2 B000EYLDYE "I absolutely LOVE this dried fruit.  LOVE IT.  Whenever I … great
#>  3 B0026LIO9A "GREAT DEAL, CONVENIENT TOO.  Much cheaper than WalMart and… great
#>  4 B00473P8SK "Great flavor, we go through a ton of this sauce! I discove… great
#>  5 B001SAWTNM "This is excellent salsa/hot sauce, but you can get it for … great
#>  6 B000FAG90U "Again, this is the best dogfood out there.  One suggestion… great
#>  7 B006BXTCEK "The box I received was filled with teas, hot chocolates, a… other
#>  8 B002GWH5OY "This is delicious coffee which compares favorably with muc… great
#>  9 B003R0MFYY "Don't let these little tiny cans fool you.  They pack a lo… great
#> 10 B001EO5ZXI "One of the nicest, smoothest cup of chai I've made. Nice m… great
#> # … with 3,990 more rows

library(hardhat)
#> Warning: package 'hardhat' was built under R version 4.1.2
sparse_bp <- default_recipe_blueprint(composition = "dgCMatrix")

library(textrecipes)

text_rec <-
  recipe(score ~ review, data = training_data) %>%
  step_tokenize(review)  %>%
  step_tokenfilter(review, max_tokens = 1e3) %>%
  step_tfidf(review)

xgb_spec <-
  boost_tree(mode = "classification") %>%
  set_engine("xgboost")

wf_sparse <- 
  workflow() %>%
  add_recipe(text_rec, blueprint = sparse_bp) %>%
  add_model(xgb_spec)

wf_default <- 
  workflow() %>%
  add_recipe(text_rec) %>%
  add_model(xgb_spec)

set.seed(123)
food_folds <- vfold_cv(training_data, v = 3)

results <- bench::mark(
  iterations = 1, check = FALSE,
  sparse = fit_resamples(wf_sparse, food_folds),  
  default = fit_resamples(wf_default, food_folds), 
)
#> Warning: Some expressions had a GC in every iteration; so filtering is disabled.

fit_resamples(wf_sparse, food_folds) %>%
  collect_metrics()
#> # A tibble: 2 × 6
#>   .metric  .estimator  mean     n std_err .config             
#>   <chr>    <chr>      <dbl> <int>   <dbl> <chr>               
#> 1 accuracy binary     0.474     3  0.0492 Preprocessor1_Model1
#> 2 roc_auc  binary     0.533     3  0.0426 Preprocessor1_Model1

fit_resamples(wf_default, food_folds) %>%
  collect_metrics()
#> # A tibble: 2 × 6
#>   .metric  .estimator  mean     n std_err .config             
#>   <chr>    <chr>      <dbl> <int>   <dbl> <chr>               
#> 1 accuracy binary     0.743     3 0.00416 Preprocessor1_Model1
#> 2 roc_auc  binary     0.780     3 0.00521 Preprocessor1_Model1

[hfrick]

Can you please provide a minimal reprex (reproducible example)? The goal of a reprex is to make it as easy as possible for us to recreate your problem so that we can fix it: please help us help you!

If you've never heard of a reprex before, start by reading "What is a reprex", and follow the advice further down that page.

[tolliam]

@hfrick I've updated it, hope that should make it easier. Many thanks

[juliasilge]

I can't reproduce this with a simpler dataset:

library(tidymodels)
data(two_class_dat)

data_train <- two_class_dat[-(1:10), ]
data_test  <- two_class_dat[  1:10 , ]

library(hardhat)
sparse_bp <- default_recipe_blueprint(composition = "dgCMatrix")

simple_rec <- recipe(Class ~ ., data = data_train)

xgb_spec <-
  boost_tree(mode = "classification") %>%
  set_engine("xgboost")

wf_sparse <- 
  workflow() %>%
  add_recipe(simple_rec, blueprint = sparse_bp) %>%
  add_model(xgb_spec)

wf_default <- 
  workflow() %>%
  add_recipe(simple_rec) %>%
  add_model(xgb_spec)

set.seed(123)
folds <- vfold_cv(data_train, v = 3)

fit_resamples(wf_sparse, folds) %>%
  collect_metrics()
#> # A tibble: 2 × 6
#>   .metric  .estimator  mean     n std_err .config             
#>   <chr>    <chr>      <dbl> <int>   <dbl> <chr>               
#> 1 accuracy binary     0.787     3 0.0100  Preprocessor1_Model1
#> 2 roc_auc  binary     0.847     3 0.00469 Preprocessor1_Model1

fit_resamples(wf_default, folds) %>%
  collect_metrics()
#> # A tibble: 2 × 6
#>   .metric  .estimator  mean     n std_err .config             
#>   <chr>    <chr>      <dbl> <int>   <dbl> <chr>               
#> 1 accuracy binary     0.787     3 0.0100  Preprocessor1_Model1
#> 2 roc_auc  binary     0.847     3 0.00469 Preprocessor1_Model1

^{Created on 2022-03-22 by the reprex package (v2.0.1)}

I think the next step is to look into whether this is textrecipes. What do you think @EmilHvitfeldt?

[EmilHvitfeldt]

After fixing a bug in {textrecipes}, I can reproduce the error, but it is not happening in {textrecipes}. Below is a reprex. They the fitted workflows produce different predictions (quite different, around 70% are different), but the recipes for the two workflows produces identical preprocessing

library(tidyverse)
library(tidymodels)

data("small_fine_foods")

library(hardhat)

sparse_bp <- default_recipe_blueprint(composition = "dgCMatrix")

library(textrecipes)

text_rec <-
  recipe(score ~ review, data = training_data) %>%
  step_tokenize(review)  %>%
  step_tokenfilter(review, max_tokens = 1e3) %>%
  step_tfidf(review)

xgb_spec <-
  boost_tree(mode = "classification") %>%
  set_engine("xgboost")

wf_sparse <- 
  workflow() %>%
  add_recipe(text_rec, blueprint = sparse_bp) %>%
  add_model(xgb_spec)

wf_default <- 
  workflow() %>%
  add_recipe(text_rec) %>%
  add_model(xgb_spec)

fit_sparse <- fit(wf_sparse, training_data)
fit_default <- fit(wf_default, training_data)

tibble(
  sparse = predict(fit_sparse, new_data = training_data)$.pred_class,
  default = predict(fit_default, new_data = training_data)$.pred_class
) %>%
  count(sparse == default)
#> # A tibble: 2 × 2
#>   `sparse == default`     n
#>   <lgl>               <int>
#> 1 FALSE                2743
#> 2 TRUE                 1257

identical(
  fit_sparse %>% extract_recipe() %>% bake(training_data),
  fit_default %>% extract_recipe() %>% bake(training_data)
)
#> [1] TRUE

[EmilHvitfeldt]

This looks like it might be a xgboost thing, and is "expected behavior" dmlc/xgboost#1634, https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/40835

[tolliam]

Many thanks @EmilHvitfeldt for getting to the bottom of this. Essentially then using the sparse matrix representation with XGBoost is not something tidymodels can support. I can't think of a scenario where you'd want 0s in a bag of words to be treated as NAs but might be wrong. I've found a separate issue with ranger not working too which I'll raise another issue for. I do appreciate you folks have done a fantastic job with tidymodels and textrecipes so any limitations with it are really just people like me being greedy!

[EmilHvitfeldt]

I would like to clarify that using the sparse matrix representation with XGBoost is something tidymodels fully supports. It just happens to be that xgboost is designed to behave differently when supplied sparse data.

Below is a small reprex to show what I mean. I created a sparse and dense version of the tfidf'd data set. Then I fit a xgboost model using the same data both in sparse and dense representation, and we see a quite striking different between the two.

I fully agree that this is weird behavior and properly not something one would expect.

library(textrecipes)
library(dplyr)
library(xgboost)

data("small_fine_foods", package = "modeldata")

text_rec <-
  recipe(score ~ review, data = training_data) %>%
  step_tokenize(review)  %>%
  step_tokenfilter(review, max_tokens = 1e3) %>%
  step_tfidf(review) %>%
  prep()

foods_sparse <- text_rec %>%
  bake(new_data = NULL, -score, composition = "dgCMatrix")

foods_dense <- text_rec %>%
  bake(new_data = NULL, -score, composition = "matrix")

foods_label <- text_rec %>%
  bake(new_data = NULL, score) %>%
  mutate(score = as.numeric(score) - 1) %>%
  pull(score)

xgboost_sparse <- xgb.train(
  data = xgb.DMatrix(foods_sparse, label = foods_label),
  params = list(
    eta = 0.3, max_depth = 6, gamma = 0, colsample_bytree = 1,
    colsample_bynode = 1, min_child_weight = 1, subsample = 1,
    objective = "binary:logistic"
  ),
  nrounds = 15
)

xgboost_dense <- xgb.train(
  data = xgb.DMatrix(foods_dense, label = foods_label),
  params = list(
    eta = 0.3, max_depth = 6, gamma = 0, colsample_bytree = 1,
    colsample_bynode = 1, min_child_weight = 1, subsample = 1,
    objective = "binary:logistic"
  ),
  nrounds = 15
)

pred_sparse <- predict(xgboost_sparse, foods_sparse)
pred_dense <- predict(xgboost_dense, foods_sparse)

plot(pred_sparse, pred_dense)

^{Created on 2022-03-26 by the reprex package (v2.0.1)}

[hfrick]

maybe closed by #950?

[EmilHvitfeldt]

Yes it is!