-
Notifications
You must be signed in to change notification settings - Fork 87
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
API for refitting pipelines on entire training data #876
Conversation
Codecov Report
@@ Coverage Diff @@
## master #876 +/- ##
========================================
Coverage 99.78% 99.79%
========================================
Files 195 195
Lines 8966 9085 +119
========================================
+ Hits 8947 9066 +119
Misses 19 19
Continue to review full report at Codecov.
|
What's missing right now: in order to fix codecov I have to add more test coverage of |
evalml/automl/auto_search_base.py
Outdated
@@ -388,17 +389,18 @@ def _evaluate(self, pipeline, X, y, raise_errors=True, pbar=None): | |||
|
|||
if self.optimize_thresholds and self.objective.problem_type == ProblemTypes.BINARY and self.objective.can_optimize_threshold: | |||
X_train, X_threshold_tuning, y_train, y_threshold_tuning = train_test_split(X_train, y_train, test_size=0.2, random_state=self.random_state) | |||
pipeline.fit(X_train, y_train) | |||
cv_pipeline = pipeline.clone() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@eccabay here's clone
in action! 🎊😁
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did this because there's no need to modify the inputted pipeline here. Once this PR is merged we should update _evaluate
to make this clear
evalml/automl/auto_search_base.py
Outdated
|
||
evaluation_entry = {"all_objective_scores": ordered_scores, "score": score} | ||
if isinstance(cv_pipeline, BinaryClassificationPipeline) and cv_pipeline.threshold is not None: | ||
evaluation_entry['binary_classification_threshold'] = cv_pipeline.threshold |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought it would be helpful to save the binary classification thresholds learned in each CV fold, for reference and debugging.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Previously we were able to access the threshold value from the final CV fold via get_pipeline
; but this was not correct behavior. The threshold should be (optionally) relearned when the model is retrained.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This creates an evaluation_entry
that may or may not have evaluation_entry['binary_classification_threshold']
? Is this favorable over always having a evaluation_entry['binary_classification_threshold'] entry, but that value could either be None or a numerical value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@angela97lin yep your idea is simpler, will do that 😂 thanks
evalml/automl/auto_search_base.py
Outdated
parameters = pipeline_results.get('parameters') | ||
if pipeline_class is None or parameters is None: | ||
raise PipelineNotFoundError("Pipeline class or parameters not found in automl results") | ||
return pipeline_class(parameters) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, I forgot to pass in random_state
here. Will do so.
@@ -53,6 +53,9 @@ def test_convert_to_seconds(): | |||
assert convert_to_seconds("10 hour") == 36000 | |||
assert convert_to_seconds("10 hours") == 36000 | |||
|
|||
with pytest.raises(AssertionError, match="Invalid unit."): | |||
convert_to_seconds("10 years") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Moved this coverage over from the automl tests
52315aa
to
82e69a1
Compare
docs/source/changelog.rst
Outdated
* Update `AutoSearchBase.get_pipelines` to return an untrained pipeline instance :pr:`876` | ||
* Save learned binary classification thresholds in automl results cv data dict :pr:`876` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah let's make these past tense RE our convention (might be worth revisiting this convention / considering if something else is more natural since I've seen this across the team? 🤔)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will do. Yeah, oops!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should add this to our contribution guidelines because I did not know of this convention!
evalml/automl/auto_search_base.py
Outdated
|
||
evaluation_entry = {"all_objective_scores": ordered_scores, "score": score} | ||
if isinstance(cv_pipeline, BinaryClassificationPipeline) and cv_pipeline.threshold is not None: | ||
evaluation_entry['binary_classification_threshold'] = cv_pipeline.threshold |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This creates an evaluation_entry
that may or may not have evaluation_entry['binary_classification_threshold']
? Is this favorable over always having a evaluation_entry['binary_classification_threshold'] entry, but that value could either be None or a numerical value?
evalml/automl/auto_search_base.py
Outdated
@@ -569,6 +575,10 @@ def full_rankings(self): | |||
|
|||
@property | |||
def best_pipeline(self): | |||
"""Returns the best model found""" | |||
"""Returns an untrained instance of the best pipeline and parameters found during automl search. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
assert not automl.optimize_thresholds | ||
|
||
|
||
def test_init_default_binary(): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do these tests here as opposed to the test_automl_search_*
classes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, I'll move them back
assert not automl.optimize_thresholds | ||
|
||
|
||
def test_init_custom_binary(dummy_binary_pipeline_class): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These init tests repeat a lot of code. Might be worth refactoring using parametrize?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dsherry The implementation looks good to me - I just left some minor comments that don't need to be addressed in this PR.
…ipelines to return an untrained pipeline copy instead of a CV-trained copy. Refactor and improve AutoSearchBase init methods Train a cloned copy of the pipeline, not the pipeline itself. Save bin class threshold in cv data
82e69a1
to
d6e9100
Compare
Fix #719
Changes
AutoSearchBase.get_pipeline
to return an untrained pipeline, not the pipeline trained on the final CV foldtest_autobase.py
, parametrized many to run on all problem types, deleted some duplicates.AutoSearchBase._evaluate
to leave the inputted pipeline unmodified, and use the newclone
method to create copies to train with.Future
After this PR, we should circle back and clean up the way
AutoSearchBase
passes pipelines and parameters to and from theAutoMLAlgorithm
class. Currently that code returns a pipeline instance holding parameters, which is passed to_evaluate
. This may no longer be necessary;_evaluate
and_add_result
no longer require a trained pipeline, just the pipeline class and the parameters.