You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
mse() returns unexpected result when applied to a certain part of inner resampling (extended_archive$...prediction("test")), whereas returns expected result when applied to more detailed part of same inner resampling (...predictions("test")[[1]])
Date: 2022-02-09. R Version: 4.0.3 (2020-10-10).
Platform: x86_64-apple-darwin17.0 (64-bit)
Setup:
Using nested resampling (inner: rsmp("cv", 4), outer: rsmp("rep_cv", 2, 3)) and regr.glmnet as only learner, auto-tuning s (random search, terminate n_evals = 7), predict_sets = c("train", "test"), performance measure regr.mse. Execute: rrMap <- mlr3misc::map(as.data.table(rr)$learner, "model")
Replicable example with dummy data.
set.seed(123)
for(iin1:9) {
assign(paste0("x", i), rnorm(n=100, mean= sample(50:100,1), sd= sample(5:60,1)))
} # Make predictorserr<- rnorm(n=100, 1000, 100) # error termy<-42- (2*x1) +3*x2-.66*x3+.03*x4+1.7*x5+.085*x6+.1*x7-.008*x8+x9+errdat<-data.frame(y, x1, x2, x3, x4, x5, x6, x7, x8, x9)
library(mlr3verse)
tskReg<-TaskRegr$new(id="Check", backend=dat, target="y")
LASSO<- lrn("regr.glmnet", alpha=1, predict_sets=c("train", "test"))
inner_rsmp= rsmp("cv", folds=4)
measure= msr("regr.mse")
search_space= ps(s= p_dbl(lower=0.001, upper=0.8))
terminator= trm("evals", n_evals=7)
tuner= tnr("random_search")
at=AutoTuner$new(LASSO, inner_rsmp, measure, terminator, tuner, search_space, store_models=TRUE) # at = auto tuningouter_rsmp<- rsmp("repeated_cv", repeats=2, folds=3)
rr= resample(tskReg, at, outer_rsmp, store_models=TRUE)
rrMap<-mlr3misc::map(as.data.table(rr)$learner, "model")
rrMap[[1]]$tuning_instance$archive$extended_archive# OverviewrrMap[[1]]$tuning_instance$archive$best() # Best predictionrrMap[[1]]$tuning_instance$archive$extended_archive$resample_result[[3]]$prediction("test") # Best regr.mse = 13083.91 -> resample_result[[3]] -> prediction resulttruthVals<-rrMap[[1]]$tuning_instance$archive$extended_archive$resample_result[[3]]$prediction("test")$truthrespVals<-rrMap[[1]]$tuning_instance$archive$extended_archive$resample_result[[3]]$prediction("test")$response# Extract truth and response values### Unexpected result - why?mlr3measures::mse(truth=truthVals, response=respVals) # Expected result = 13083.91, actual result = 13009.43rrMap[[1]]$tuning_instance$archive$extended_archive$resample_result[[3]]$score()[1,] # The same does not happen when using a single inner fold, instead of all four folds, per tuning parameter s.truthVals<-rrMap[[1]]$tuning_instance$archive$extended_archive$resample_result[[3]]$predictions("test")[[1]]$truthrespVals<-rrMap[[1]]$tuning_instance$archive$extended_archive$resample_result[[3]]$predictions("test")[[1]]$responsemlr3measures::mse(truth=truthVals, response=respVals) # Expected and actual result of 15541.3 agree.
Thank you very much in advance, not just for answering, but also for your efforts put into mlr3.
The text was updated successfully, but these errors were encountered:
Internally, we call rr$aggregate() which calls mlr3measures::mse() on each Prediction object and then calculates the mean of the four regr.mse scores. The result 13083.91 is logged to the archive.
You used the rr$prediction("test") object which is the combined Prediction object of the four resampling iterations and then you called mlr3measures::mse() on the combined prediction result.
mse() returns unexpected result when applied to a certain part of inner resampling (extended_archive$...prediction("test")), whereas returns expected result when applied to more detailed part of same inner resampling (...predictions("test")[[1]])
Date: 2022-02-09. R Version: 4.0.3 (2020-10-10).
Platform: x86_64-apple-darwin17.0 (64-bit)
Setup:
Using nested resampling (inner: rsmp("cv", 4), outer: rsmp("rep_cv", 2, 3)) and regr.glmnet as only learner, auto-tuning s (random search, terminate n_evals = 7), predict_sets = c("train", "test"), performance measure regr.mse. Execute: rrMap <- mlr3misc::map(as.data.table(rr)$learner, "model")
Replicable example with dummy data.
Thank you very much in advance, not just for answering, but also for your efforts put into mlr3.
The text was updated successfully, but these errors were encountered: