Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'search_results' and 'search_order' #260

Merged
merged 9 commits into from
Dec 10, 2019
Merged

Add 'search_results' and 'search_order' #260

merged 9 commits into from
Dec 10, 2019

Conversation

jeremyliweishih
Copy link
Collaborator

Made self.results in AutoBase to handle more information we would want to hold. This PR addes self.results['search_results] and self.results['search_order']). Relates to #160 and #241. If more information is needed in the future we can extend self.results`.

@codecov
Copy link

codecov bot commented Dec 9, 2019

Codecov Report

Merging #260 into master will increase coverage by 0.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #260      +/-   ##
==========================================
+ Coverage   97.07%   97.08%   +0.01%     
==========================================
  Files          95       95              
  Lines        2731     2742      +11     
==========================================
+ Hits         2651     2662      +11     
  Misses         80       80
Impacted Files Coverage Δ
evalml/models/auto_base.py 93.72% <100%> (+0.02%) ⬆️
evalml/tests/automl_tests/test_autoclassifier.py 100% <100%> (ø) ⬆️
evalml/models/pipeline_search_plots.py 100% <100%> (ø) ⬆️
...l/tests/automl_tests/test_pipeline_search_plots.py 96.69% <100%> (+0.11%) ⬆️
evalml/tests/automl_tests/test_autobase.py 100% <100%> (ø) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68ee2d0...82a1d9e. Read the comment docs.

Copy link
Contributor

@kmax12 kmax12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. proposed an api change to naming

@@ -59,6 +59,8 @@ def __init__(self, problem_type, tuner, cv, objective, max_pipelines, max_time,
else:
raise TypeError("max_time must be a float, int, or string. Received a {}.".format(type(max_time)))
self.results = {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe just init all at once?

self.results = {
    'pipelines': {},
    'search_order` []
}

also what if we called the key pipelines instead of search_results?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kmax12 I originally was going to call it pipelines but thought it would be misleading since it doesn't actually store the pipeline itself. Would pipeline_results work?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure. i like that

Copy link
Contributor

@kmax12 kmax12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. just one comment about noting the breaking change

@@ -6,6 +6,7 @@ Changelog
* Enhancements
* Added ability to create a plot of feature importances :pr:`133`
* Added ROC and confusion matrix metrics and plot for classification problems and introduce PipelineSearchPlots class :pr:`242`
* Enhanced AutoML results with search order :pr:`260`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think we should add a "breaking change" warning to the changelog that the results dictionary changed

kmax12
kmax12 previously approved these changes Dec 10, 2019
Copy link
Contributor

@kmax12 kmax12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one comment about changelog, otherwise LGTM

@@ -21,6 +22,12 @@ Changelog
* Testing Changes
* Added support for testing on Windows with CircleCI :pr:`226`
* Added support for doctests :pr:`233`

**Breaking Changes**
* `autoclassifier.results` and `autoregressor.results` now is a dictionary
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be AutoClassifier and AutoRegressor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants