Multi-objective Tuning basics

[aminevsaziz]

after using @ja-thomas snippet in #395 & reading carefully the literature that comes with mlrMBO specially " Model-Based Multi-objective Optimization: Taxonomy, Multi-Point Proposal, Toolbox and Benchmark"; i made an example where i want to understand how things works :

library(mlr,quietly = T)
library(mlrMBO,quietly = T)
library(parallelMap,quietly = T)

mlr::configureMlr(show.info = F, show.learner.output = F,on.learner.warning = "quiet")

## Perparing outer & inter resampling 
Outer = makeResampleInstance(desc = makeResampleDesc(method = "Holdout",predict = "both",split=0.70),task = mlr::iris.task)
Inner = makeResampleDesc(method = "CV",predict = "both",iters=3L)
Tun_Task <- mlr::subsetTask(mlr::iris.task,subset = Outer$train.inds[[1]])

cpus = 2L
itr = 5L

## hyperparamter to tune in
gbm.ps <- makeParamSet(makeDiscreteParam("bag.fraction",values = 1 ),
                       makeNumericParam("shrinkage",1e-3,0.5 ),
                       makeNumericParam("train.fraction",0.5,1 ),
                       makeIntegerParam("n.trees",25L,100L) ,
                       makeIntegerParam("interaction.depth",1L,4L ),
                       makeDiscreteParam("n.minobsinnode",values = 10L ))
## setup MBO control
ctrl = makeMBOControl(n.objectives = 2L, propose.points = 2L,y.name = c("mmce","kappa"))
ctrl = setMBOControlTermination(ctrl, iters = 10L)

## here is little twist
ctrl = setMBOControlMultiObj(ctrl, method = "dib",dib.indicator = "eps")
ctrl = setMBOControlInfill(ctrl, crit = makeMBOInfillCritDIB(cb.lambda = 2L))

gbm = makeMultiObjectiveFunction(name = "gbm.tuning",
                                 fn =  function(x) {
                                   lrn = mlr::makeLearner("classif.gbm",predict.type = "prob", par.vals = x)
                                   mlr::resample(lrn, Tun_Task, Inner, measures = list(mmce,kappa), show.info = F)$aggr
                                 },
                                 par.set = gbm.ps,has.simple.signature = FALSE,minimize = c(TRUE, FALSE),n.objectives = 2L)
## initial Design
gbm.des = ParamHelpers::generateDesign(4L, getParamSet(gbm), fun = lhs::maximinLHS)

## Perform Tuning 
parallelMap::parallelStartMulticore(cpus = cpus,level = "mlrMBO.propose.points")
tune.gbm = mlrMBO::mbo(gbm, gbm.des, control = ctrl, show.info = TRUE)
parallelMap::parallelStop()


## Train Whole model using the outer resampling based on the hyperparamters 
## obtain from tuning the multi-objective function 

discrete_ps <- makeParamSet(makeDiscreteParam("shrinkage",values = tune.gbm$pareto.set[[1]]$shrinkage ),
                            makeDiscreteParam("bag.fraction",values = tune.gbm$pareto.set[[1]]$bag.fraction ),
                            makeDiscreteParam("train.fraction",values = tune.gbm$pareto.set[[1]]$train.fraction ),
                            makeDiscreteParam("n.trees",values = tune.gbm$pareto.set[[1]]$n.trees ) ,
                            makeDiscreteParam("interaction.depth",values = tune.gbm$pareto.set[[1]]$interaction.depth ),
                            makeDiscreteParam("n.minobsinnode",values = tune.gbm$pareto.set[[1]]$n.minobsinnode ))

final.gbm = tuneParams(makeLearner("classif.gbm",predict.type = "prob"), task = mlr::iris.task, resampling = Outer,measures = list(mmce,kappa),
                       par.set = discrete_ps, control = makeTuneControlGrid(resolution = 1),show.info = T)

here are my questions :

1. how to determine or even calculate lambda for both LCB/DIB for example in mix space optimization section in mlrMBO tuturials lambda was set to 5 but in package vignette 1 for purely numeric space & 2 for otherwise(mixed space) so from where 5 is coming in that example & in our example above is correct to pass lambda to 2 .
2. is there away to tune a nested level resampling with multi objective since i do care for final performance of the test set in the outer resampling instance.however i find it clunky to do so specially when tuning the inner resampling like the example above it gives 3 sometimes 2 best hyperparmeter combination so to obtain performance on the outer test set would require the shady tune-grid.

3.in the paper " Model-Based Multi-objective Optimization: Taxonomy, Multi-Point Proposal, Toolbox and Benchmark" epsilon-EGO was found to have less deterioration in multi-point proposal than sms-EGO than why sms-EGO is the default for DIB.

4.any advice to enhance that example would be great.

[Steviey]

+1

[jakob-r]

1. There is no true best lambda value. In some papers some values work better, but this is just due to the specific problems they looked at. If you have totally different functions another value might work better. If you know that your function really is a realization of a certain Gauss process then there certainly is an optimal lambda, but this assumption usually does not hold. My gut feeling tells me that a value between 2 and 5 seems fair. After all the difference of how fast you get close to true optimum (Pareto front) should just be small.
2. You do nested resampling to estimate the performance you can get if you tune your hyperparameters. Therefore, we need one optimal result. Multi-objective optimization gives you a Pareto set instead of a single best result. You could come up with a strategy or always select one setting yourself. Afterwards you could do the nested resampling to validate the tuning + selection process. Multi-objective optimization is not for tuning hyperparameters, it's for exploring the possible trade of between to or more objectives.
3. I guess the choice was made before the benchmark showed that eps-EGO is better. (Also @danielhorn ?)
4. Sorry, no (but it's probably too late anyhow)

Closing, because there is no bug (just questionable defaults)