Thanks for using LightGBM and for the clear writeup with a reproducible example!

I've edited the text a bit to make it clearer which parts are your own words, logs printed by LightGBM, and the code you ran. If you're unfamiliar with how to do that, please see https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax.

I'll investigate this today.

I ran this code on {lightgbm} v3.3.5 and v4.0.0 (built from source here, since that isn't on CRAN yet), and got the same error both times. Providing a bit more error message to help:

Error in valid_data$set_reference(data) :
set_reference: cannot set reference after freeing raw data,
please set ‘free_raw_data = FALSE’ when you construct lgb.Dataset

That looks like a bug to me. Maybe because the raw data isn't being passed through in Dataset$slice()?

Sorry about that! Are you interested in trying to contribute a fix?

If not, someone here will pick it up when we have time.

I opened a PR for this as a possible fix. I reproduced this using the lgb.slice.Dataset interface, but got the same error.

library(lightgbm)

boston = MASS::Boston
str(boston)
dim(boston)

set.seed(12)
boston_lgb_dataset = lgb.Dataset(scale(boston[, -14]), label = boston[,  14] ,free_raw_data = FALSE)

dtrain = lgb.slice.Dataset(boston_lgb_dataset, c(1:350))
dtest = lgb.slice.Dataset(boston_lgb_dataset, c(351:506))

params = list(
  objective = "regression"
  , metric = "l2"
  , min_data = 1L
  , learning_rate = .3
)
 
model = lgb.train( 
  params = params
  , data = dtrain
  , nrounds = 20L
   , valids = list( test  = dtest) 
 )