#42 break up tuning into multiple sections WIP

[masongallo]

• Tuning moved to basic section in tutorial
• Basic tuning described in more detail
• Incorporated GSoC work on hyperparameter effects
• Moved advanced tuning like irace to advanced section
• Full GSoC tutorial page
• Decide on how to structure advanced tuning section
• Address Add examples about train performance #44

This is currently a WIP. I have rewritten @schiffner 's excellent work on the tuning section so that the "basic" stuff is in the basic section with more detail and hopefully understandable even by a beginner to mlr. You will notice several FIXMEs as I am awaiting my GSoC PR on mlr as well as deciding how we want to build out the "advanced" tuning tutorials as they are quite complicated and will need a lot of additional detail.

[masongallo]

The Travis failure doesn't look related to my edits - any ideas?

[berndbischl]

there was a renaming problem in the tutorial.

can you rebase pls and check wether it works now?

[berndbischl]

now i am getting this weird unload error with digest and pander. i tried to fix this.
pls let this not block you. simply render out the tutorial locally for now.
the move on to other stuff.

we will fix this later

[masongallo]

@berndbischl done

[kerschke]

This looks actually very nice, I really like your explanations and the colorful plots :-) Nevertheless, I have some (minor) suggestions for improvement.

Let's start with the tune.Rmd (the one for the basic section):

• In the beginning you show an example of a paramset, a ctrl object and the resampling description and always start with "an example of ...". While it is true that these are always examples for those objects, it just sounds strange when reading it. I would suggest to either merge the 3 examples into one or eventually ignore them completely (at that point). I mean, there is not much of a benefit by showing how to create a param set and so on, without doing anything with it and you actually show them at a later stage anyways.

• In the second paragraph of "specifying the search space", you write "This is done via function makeParamSet." Here, I would actually add the article "the" in front of "function". Also, I would add the output of the parameter set example, i.e. just print discrete_ps.

• In the same paragraph you write

  We could also define a search space from 10^-10 to 10^10 for both parameters through the use of trafo arg. Trafos work like this:

I suggest to rewrite it to "We could also define a continuous search space (using makeNumericParam instead of makeDiscreteParam) from 10^-10 to 10^10 for both parameters through the use of the trafo argument. Transformations work like this:"
Mentioning the makeNumericParam at the beginning of that sentence shows more clearly what it is actually used for and also allows you to remove "Notice this time we use makeNumericParam:"

• Also, in the same sentence, you write "original scale (from -12 to 12) in this case". Doesn't the part "in this case" actually belong into the brackets?
• Instead of writing abbreviations such as "arg" and "trafo" (occurs several times within your tutorial), please use the full names "argument" and "transformation".
• I wouldn't call the tuning methods "optimization algorithms". I suggest to mention the equivalency between the two names once somewhere at the begining of the tuning method intro. Further mentioning of optimization algorithms within the tuning context might be misleading / confusing..

Now, let's have a look at learning_curve.Rmd and hyperpar_tuning_effects.Rmd:

• The indent within your R-code, e.g. within generateLearningCurveData, sometimes strongly differs from the indent that we usually use (2 whitespaces). Could you please correct that?
• In the beginning of your hyperpar_tuning, you repeat the 3 tuning ingredients, including "the optimization algorithm". As mentioned in the base tuning, I would at least also mention that optimization algorithm and tuning method are equivalent.
• Always indicate arguments, data sets and so on by using apostrophes, e.g. trafo instead of trafo.
• As mentioned before, please use "argument" instead of "arg".
• You have two sentences that start identically: "In the example below, we perform grid search on the C parameter for SVM on the famous Pima Indians dataset". Copy-paste just doesn't look very appealing ;-)

[masongallo]

Thanks @kerschke! I'll work on the above.

[masongallo]

Feedback is integrated with 2 slight exceptions: I still think we should say optimization algorithm (especially since the docs mention algo) so I added a note in parentheses for both the tutorials. I also didn't find a copy-pasta error, but I tweaked some of the language in any case.

The Travis issue looks like a naming error in plotThreshVsPerf?

[masongallo]

Looks like @zmjones fixed the Travis error with plotThreshVsPerf mlr-org/mlr@2e52b97 so hopefully we're all good 👍

[schiffner]

Feedback is integrated with 2 slight exceptions: I still think we should say optimization algorithm (especially since the docs mention algo) so I added a note in parentheses for both the tutorials. I also didn't find a copy-pasta error, but I tweaked some of the language in any case.

I'm also ok with optimization algorithm.

Thank you very much for all the work you put into this and sorry that I haven't gotten around to look at your PR sooner. Will do this tomorrow at the latest.

[schiffner]

Ok, just read it now. I like it very much.
Also many thanks for correcting things (bad English, code indentation) that were basically my fault. ;)