Streamline the results file for cross-validation task #372
Conversation
- Include folds file path as one of the return values so that it can be included in the results file.
- Include the path to the folds file when specified. - Do not include stratified folds information when folds file is specified. - Modify number of CV folds to be `K via folds file` if a folds file is specified. - Modify grid search folds to be `K via folds file` if they also end up using the folds file. - Show the value of `use_folds_file_for_grid_search` when appropriate.
- Modify tests.
| if lrd['task'] == 'cross_validate': | ||
| print('Number of Folds: {}'.format(lrd['cv_folds']), | ||
| file=output_file) | ||
| print('Stratified Folds: {}'.format(lrd['stratified_folds']), | ||
| file=output_file) | ||
| if not lrd['cv_folds'].endswith('folds file'): |
There was a problem hiding this comment.
Is "folds file" supposed to be "folds_file" here? Can we just make it check if the string is equal to "folds file"/"folds_file" rather than using endswith or is there some reason that might not work?
There was a problem hiding this comment.
Nope, it's trying to match the string on the right hand side of the colon in the results file which should say 3 via folds file so it will always ends with "folds file" not "folds_file".
There was a problem hiding this comment.
No worries. If you want to see some example output for various conditions, you can play around with cross_validate.cfg in the titanic example and look at the results files.
| file=output_file) | ||
| if (lrd['task'] == 'cross_validate' and | ||
| lrd['grid_search'] and | ||
| lrd['cv_folds'].endswith('folds file')): |
desilinguist
dismissed
mulhod’s stale review
Matt didn't actually mean to request changes :)
tests/test_output.py
Outdated
| @@ -170,7 +171,7 @@ def check_summary_score(use_feature_hashing=False): | |||
| # the learner results dictionaries should have 29 rows, | |||
There was a problem hiding this comment.
I think the comment needs to be updated with the new total number of rows.
The
.resultsfile is produced for theevaluateand thecross_validatetasks. Currently, for thecross_validatetask, the.resultsfile (a) prints out the entire folds dictionary iffolds_fileis specified (b) includes details that aren't relevant to the experiment.In this branch, I streamline the results file in the following way:
_parse_config_fileto include the folds file path.<n> via folds fileif a folds file is specified.<n> via folds fileif the grid search ends up using the folds file.use_folds_file_for_grid_searchwhen appropriate.grid_search_foldsandgrid_objective) only when we are actually doing grid search.To test this, I also included an entirely new test in
test_output.py. Additional changes in the branch include:.resultsand, hence,.results.jsonfiles.