Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show number of configuration evaluations in log #218

Merged
merged 2 commits into from Jan 22, 2020
Merged

Show number of configuration evaluations in log #218

merged 2 commits into from Jan 22, 2020

Conversation

@be-marc
Copy link
Contributor

be-marc commented Jan 13, 2020

Closes #200

@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Jan 13, 2020

Codecov Report

Merging #218 into master will decrease coverage by 0.29%.
The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff            @@
##           master     #218     +/-   ##
=========================================
- Coverage   91.83%   91.53%   -0.3%     
=========================================
  Files          18       18             
  Lines         306      307      +1     
=========================================
  Hits          281      281             
- Misses         25       26      +1
Impacted Files Coverage Δ
R/TuningInstance.R 90% <0%> (-1.02%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 706f1dc...aa81bab. Read the comment docs.

Copy link
Member

pat-s left a comment

Could you add a reprex to see the change in action? Thanks.

@@ -274,6 +274,7 @@ TuningInstance = R6Class("TuningInstance",
a = bmr$aggregate(measures = self$measures, ids = FALSE)[, mids, with = FALSE]
lg$info("Result:")
lg$info(capture.output(print(cbind(dt, a), class = FALSE, row.names = FALSE, print.keys = FALSE)))
lg$info(paste("Number of configuration evaluations:", self$n_evals))

This comment has been minimized.

Copy link
@pat-s

pat-s Jan 15, 2020

Member

Is the wording here clear enough? Maybe

Suggested change
lg$info(paste("Number of configuration evaluations:", self$n_evals))
lg$info(paste("Iteration #:", self$n_evals))

sufficient?

@be-marc

This comment has been minimized.

Copy link
Contributor Author

be-marc commented Jan 21, 2020

The log looks like this now:

INFO  [21:28:52.784] Starting to tune 2 parameters with '<TunerRandomSearch>' and '<TerminatorEvals>' 
INFO  [21:28:52.793] Terminator settings: n_evals=30 
INFO  [21:28:52.823] Evaluating 10 configurations 
INFO  [21:28:52.830]           cp minsplit 
INFO  [21:28:52.830]  0.007549564        3 
INFO  [21:28:52.830]  0.005017597        8 
INFO  [21:28:52.830]  0.002033883        7 
INFO  [21:28:52.830]  0.077119955        2 
INFO  [21:28:52.830]  0.047365454        8 
INFO  [21:28:52.830]  0.087831422        6 
INFO  [21:28:52.830]  0.034151872        2 
INFO  [21:28:52.830]  0.062732327        1 
INFO  [21:28:52.830]  0.016396179        1 
INFO  [21:28:52.830]  0.014664453       10 
INFO  [21:28:52.978] Benchmark with 10 resampling iterations 
INFO  [21:28:52.981] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.008] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.055] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.091] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.116] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.157] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.195] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.226] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.246] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.275] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.333] Finished benchmark 
INFO  [21:28:53.368] Result of batch 1: 
INFO  [21:28:53.372]           cp minsplit classif.ce 
INFO  [21:28:53.372]  0.007549564        3  0.2734375 
INFO  [21:28:53.372]  0.005017597        8  0.2734375 
INFO  [21:28:53.372]  0.002033883        7  0.2890625 
INFO  [21:28:53.372]  0.077119955        2  0.3007812 
INFO  [21:28:53.372]  0.047365454        8  0.3007812 
INFO  [21:28:53.372]  0.087831422        6  0.3007812 
INFO  [21:28:53.372]  0.034151872        2  0.3007812 
INFO  [21:28:53.372]  0.062732327        1  0.3007812 
INFO  [21:28:53.372]  0.016396179        1  0.2968750 
INFO  [21:28:53.372]  0.014664453       10  0.2968750 
INFO  [21:28:53.375] 10 configurations evaluated 
INFO  [21:28:53.436] Evaluating 10 configurations 
INFO  [21:28:53.439]          cp minsplit 
INFO  [21:28:53.439]  0.06768769        1 
INFO  [21:28:53.439]  0.07203581        2 
INFO  [21:28:53.439]  0.03825856        1 
INFO  [21:28:53.439]  0.02308660        6 
INFO  [21:28:53.439]  0.02737322        5 
INFO  [21:28:53.439]  0.08553937        8 
INFO  [21:28:53.439]  0.07446871        2 
INFO  [21:28:53.439]  0.04804343        9 
INFO  [21:28:53.439]  0.05213364        8 
INFO  [21:28:53.439]  0.08897468        8 
INFO  [21:28:53.568] Benchmark with 10 resampling iterations 
INFO  [21:28:53.570] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.596] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.618] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.642] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.670] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.695] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.727] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.750] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.794] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.810] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:53.863] Finished benchmark 
INFO  [21:28:53.893] Result of batch 2: 
INFO  [21:28:53.898]          cp minsplit classif.ce 
INFO  [21:28:53.898]  0.06768769        1  0.3007812 
INFO  [21:28:53.898]  0.07203581        2  0.3007812 
INFO  [21:28:53.898]  0.03825856        1  0.3007812 
INFO  [21:28:53.898]  0.02308660        6  0.3007812 
INFO  [21:28:53.898]  0.02737322        5  0.3007812 
INFO  [21:28:53.898]  0.08553937        8  0.3007812 
INFO  [21:28:53.898]  0.07446871        2  0.3007812 
INFO  [21:28:53.898]  0.04804343        9  0.3007812 
INFO  [21:28:53.898]  0.05213364        8  0.3007812 
INFO  [21:28:53.898]  0.08897468        8  0.3007812 
INFO  [21:28:53.901] 20 configurations evaluated 
INFO  [21:28:53.965] Evaluating 10 configurations 
INFO  [21:28:53.967]           cp minsplit 
INFO  [21:28:53.967]  0.080738008        6 
INFO  [21:28:53.967]  0.067280352        6 
INFO  [21:28:53.967]  0.012417658        1 
INFO  [21:28:53.967]  0.035016992        6 
INFO  [21:28:53.967]  0.011345561        5 
INFO  [21:28:53.967]  0.001720336        7 
INFO  [21:28:53.967]  0.002078460       10 
INFO  [21:28:53.967]  0.041924113        3 
INFO  [21:28:53.967]  0.022244465        4 
INFO  [21:28:53.967]  0.077070634        5 
INFO  [21:28:54.092] Benchmark with 10 resampling iterations 
INFO  [21:28:54.093] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.109] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.142] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.173] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.200] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.228] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.258] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.289] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.311] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.344] Applying learner 'classif.rpart' on task 'pima' (iter 1/1) 
INFO  [21:28:54.394] Finished benchmark 
INFO  [21:28:54.423] Result of batch 3: 
INFO  [21:28:54.426]           cp minsplit classif.ce 
INFO  [21:28:54.426]  0.080738008        6  0.3007812 
INFO  [21:28:54.426]  0.067280352        6  0.3007812 
INFO  [21:28:54.426]  0.012417658        1  0.2968750 
INFO  [21:28:54.426]  0.035016992        6  0.3007812 
INFO  [21:28:54.426]  0.011345561        5  0.2929688 
INFO  [21:28:54.426]  0.001720336        7  0.2968750 
INFO  [21:28:54.426]  0.002078460       10  0.2812500 
INFO  [21:28:54.426]  0.041924113        3  0.3007812 
INFO  [21:28:54.426]  0.022244465        4  0.3007812 
INFO  [21:28:54.426]  0.077070634        5  0.3007812 
INFO  [21:28:54.432] 30 configurations evaluated 
INFO  [21:28:54.435] Finished tuning after 30 evals 

We show the batch number and the number evaluated configurations in total.

@pat-s
pat-s approved these changes Jan 22, 2020
@pat-s

This comment has been minimized.

Copy link
Member

pat-s commented Jan 22, 2020

Thanks!

@pat-s pat-s merged commit 611c6ca into master Jan 22, 2020
2 of 4 checks passed
2 of 4 checks passed
codecov/patch 0% of diff hit (target 91.83%)
Details
codecov/project 91.53% (-0.3%) compared to 706f1dc
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@pat-s pat-s deleted the log_iterations branch Jan 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants
You can’t perform that action at this time.