Create search_space from TuneTokens #285

be-marc · 2020-11-25T13:21:41Z

No description provided.

jakob-r · 2020-11-26T08:08:57Z

R/TuningInstanceMulticrit.R

+      search_space = NULL, terminator, store_models = FALSE, 
+      check_values = FALSE, store_benchmark_result = TRUE) {
+
+      if (!is.null(search_space) && length(learner$param_set$get_values(tune_token = "only")) > 0) {


$get_values(tune_token = "only") is not the final API, right?

R/TuningInstanceMulticrit.R

jakob-r · 2020-11-26T08:39:32Z

R/TuningInstanceSingleCrit.R

-          codomain = self$objective$codomain, check_values = check_values)
-        self$objective$archive = self$archive
+    initialize = function(task, learner, resampling, measure,
+      search_space = NULL, terminator, store_benchmark_result = TRUE,


Now that the search space is optional it would make sense to put it after the terminator? Does that break a lot of stuff?

I changed it.

R/TuningInstanceSingleCrit.R

Co-authored-by: Jakob Richter <code@jakob-r.de>

jakob-r · 2020-12-01T09:09:30Z

@be-marc Can you check if adaptation are needed here: mlr-org/mlr3book#209 (and in case make a PR to this branch or talk to @mb706 and edit directly)

be-marc · 2020-12-02T15:57:48Z

learners = list(
  lrn("classif.kknn", id = "kknn"),
  lrn("classif.svm", id = "svm", type = "C-classification")
)

graph =
  po("branch", options = c("kknn", "svm")) %>>%
  gunion(lapply(learners, po)) %>>%
  po("unbranch")

graph_learner = GraphLearner$new(graph)

graph_learner$param_set$values$branch.selection = to_tune(c("kknn", "svm"))

instance = TuningInstanceSingleCrit$new(
  task = tsk("iris"),
  learner = graph_learner,
  resampling = rsmp("cv", folds = 3),
  measure = msr("classif.ce"),
  terminator = trm("evals", n_evals = 60)
)

Fails since we try to remove the TuneToken from the parameter set but branch.selection is a required parameter (TuningInstanceSingleCrit.R - 128).

mb706

I think the autotuner searchspace from paramset code is not tested yet? Also see my comments on how I think the "requires" problem should be solved.

mb706 · 2020-12-02T20:40:44Z

R/AutoTuner.R

+#'   terminator = terminator,
+#'   tuner = tuner,
+#'   search_space = search_space,
+#'   store_tuning_instance = TRUE)


imho this is a bit verbose for example code where every argument has exactly the right name already. Are you adding this for the sake of consistency with something? I guess its good practice to call functions with many arguments like this, but I think it subtracts from the example code qua example code.

mb706 · 2020-12-02T20:44:04Z

R/AutoTuner.R

+      learner = assert_learner(learner)$clone(deep = TRUE)
+
+      if (!is.null(search_space) && length(learner$param_set$get_values(type = "only_token")) > 0) {
+        stop("If the values of the ParamSet of the Learner contain TuneTokens you cannot supply a search_space.")


I know this is what the committee decided in the end so you can ignore this comment, but just for the record I am against this.

mb706 · 2020-12-02T20:47:29Z

R/AutoTuner.R

+      }
+      if (is.null(search_space)) {
+        search_space = learner$param_set$search_space()
+        learner$param_set$values = learner$param_set$get_values(type = "without_token")


you cannot do this here because this unsets parameters that may be required. Instead you should be doing this inside the tuning code.

One option here would be to leave search_space NULL, to leave the learner$param_set as it is, and trust the TuningInstance initialization code to take care of this.

Another way would be to have a "placeholder" token, similar to TuneToken, that prevents the "requires" check from triggering but has no other functionality. TuneToken could even inherit from PlaceholderToken. The semantics would be that whenever a ParamSet$values slot contains a PlaceholderToken, then its "requires" are fulfilled, its dependencies are fulfilled, but calling Learner$train() should throw an error (because the placeholder would be replaced by a concrete value in the tuning code). I.e. PlaceholderToken would be doing some part of what TuneToken is doing already, with the exception that PlaceholderToken does not generate a search_space.

mb706 · 2020-12-02T20:51:37Z

R/AutoTuner.R

-        stop("param_set is read-only.")
-      }
-      private$.param_set
-    },


sic transit gloria mundi

mb706 · 2020-12-02T20:54:24Z

R/TuningInstanceMulticrit.R

+      }
+      if (is.null(search_space)) {
+        search_space = learner$param_set$search_space()
+        learner$param_set$values = learner$param_set$get_values(type = "without_token")


don't do this here, gives a problem when there are requirements. instead do it in the evaluation code in ObjectiveTuning$eval_many like here

Wait, if this is not possible here how was it even possible before we introduced the tune_tokens? It was'nt right? So your example is the following:

We tune over Param A which can have values aX and aY.
We set Param X that needs A to be aX in the param_set$values to a fixed value - as well we set Param Y that needs A to ba aY to a fixed value.... right?

This was not possible before, and should be possible now?

pipeop = po("branch", options = c("kknn", "svm", "ranger")) pipeop$param_set <ParamSet:branch> id class lower upper levels default value 1: selection ParamFct NA NA kknn,svm,ranger <NoDefault[3]> kknn

selection has the tag required. PipeOpBranch automatically sets the required value to the first option. Currently we try to remove this values / TuneToken in TuningInstanceSingleCrit.

This was possible before because we just didn't touch the param_set$values before, and in the tuning loop we just overwrite param_set$values without clearing values.

be-marc · 2020-12-03T15:06:12Z

I am dissatisfied with the solution to store a learner with TuneToken and the associated search space in ObjectiveTuning. We store the same information and have an unsable learner again. We should discuss a clean solution in the future #286. For now,
TuneTokens are removed in ObjectiveTuning$.eval_many to avoid errors with required parameters.

be-marc added 5 commits November 25, 2020 13:48

Create search_space from TuneTokens

67efb24

Merge branch 'master' into tune_token

018b9f8

Add unit tests

22a5c22

Add multi-crit unit tests

d9c38d3

Docs and fix remotes

cbcf096

be-marc requested a review from jakob-r November 25, 2020 13:29

Remove Learner ParamSet from AutoTuner ParamSet

3859e89

jakob-r reviewed Nov 26, 2020

View reviewed changes

R/TuningInstanceMulticrit.R Outdated Show resolved Hide resolved

jakob-r reviewed Nov 26, 2020

View reviewed changes

R/TuningInstanceSingleCrit.R Outdated Show resolved Hide resolved

be-marc and others added 6 commits November 26, 2020 13:20

Update R/TuningInstanceMulticrit.R

572e6d1

Co-authored-by: Jakob Richter <code@jakob-r.de>

Update R/TuningInstanceSingleCrit.R

4bacf4d

Co-authored-by: Jakob Richter <code@jakob-r.de>

New design

b03d605

Make search space in AutoTuner optional and docu

d71ff8b

Update examples

f9e2d60

Fix style

97494b6

jakob-r approved these changes Dec 1, 2020

View reviewed changes

Use ParamSet$search_space()

97f867e

mb706 requested changes Dec 2, 2020

View reviewed changes

be-marc added 2 commits December 3, 2020 16:07

Remove TuneToken in ObjectiveTuning$.eval_many

ab1e047

Simplify

75f697a

mb706 mentioned this pull request Dec 3, 2020

Avoid storing learner with TuneToken and search space in ObjectiveTuning #286

Closed

be-marc added 2 commits December 3, 2020 16:30

Fix AutoTuner and add unit tests

07cd087

AutoTuner example

7706c86

be-marc merged commit 9866b45 into master Dec 3, 2020

be-marc deleted the tune_token branch December 3, 2020 15:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create search_space from TuneTokens #285

Create search_space from TuneTokens #285

be-marc commented Nov 25, 2020

jakob-r Nov 26, 2020

jakob-r Nov 26, 2020

be-marc Nov 26, 2020

jakob-r commented Dec 1, 2020

be-marc commented Dec 2, 2020 •

edited

Loading

mb706 left a comment

mb706 Dec 2, 2020

mb706 Dec 2, 2020

mb706 Dec 2, 2020

mb706 Dec 2, 2020

mb706 Dec 2, 2020

mb706 Dec 2, 2020

jakob-r Dec 3, 2020

be-marc Dec 3, 2020

mb706 Dec 3, 2020

be-marc commented Dec 3, 2020

Create search_space from TuneTokens #285

Create search_space from TuneTokens #285

Conversation

be-marc commented Nov 25, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jakob-r commented Dec 1, 2020

be-marc commented Dec 2, 2020 • edited Loading

mb706 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

be-marc commented Dec 3, 2020

be-marc commented Dec 2, 2020 •

edited

Loading