Conversation
Codecov Report
@@ Coverage Diff @@
## master #877 +/- ##
==========================================
+ Coverage 99.77% 99.78% +0.01%
==========================================
Files 193 195 +2
Lines 8881 8950 +69
==========================================
+ Hits 8861 8931 +70
+ Misses 20 19 -1
Continue to review full report at Codecov.
|
|
@eccabay cool, I'm excited to look at this! I see there's one test failing right now, I skimmed the PR and there's no obvious reason why this would be failing given your changes... my hunch is that this is happening because you currently have |
angela97lin
left a comment
angela97lin
left a comment
There was a problem hiding this comment.
Left minor comments but excited to see this work get in as a great start to supporting larger data sets :D
dsherry
left a comment
dsherry
left a comment
There was a problem hiding this comment.
@eccabay great! I liked your error handling and the unit tests. I left comments and tips for updating the limit check and the evaluation impl logic, as well as a suggestion on mocking in the tests. From the looks of it, the next round will be good to go.
dsherry
left a comment
dsherry
left a comment
There was a problem hiding this comment.
Looks great!
I left a few more comments. The only ones blocking merge from my perspective are one final simplification to the data split logic in AutoMLSearch.__init__, and backing out the changes in _compute_cv_scores. The rest could be addressed in a separate PR once this one's merged.
| **Breaking Changes** | ||
| * Pipelines' static ``component_graph`` field must contain either ``ComponentBase`` subclasses or ``str``, instead of ``ComponentBase`` subclass instances :pr:`850` | ||
| * Rename ``handle_component`` to ``handle_component_class``. Now standardizes to ``ComponentBase`` subclasses instead of ``ComponentBase`` subclass instances :pr:`850` | ||
| * Renamed `automl` `cv` argument to `data_split` :pr:`877` |
| if self.data_split is not None and not issubclass(self.data_split.__class__, BaseCrossValidator): | ||
| raise ValueError("Not a valid data splitter") | ||
| if self.problem_type != self.objective.problem_type: | ||
| raise ValueError("Given objective {} is not compatible with a {} problem.".format(self.objective.name, self.problem_type.value)) |
There was a problem hiding this comment.
Nice. This is a part of the validation tracked by #865. I'm in favor of adding this, even if it's not required for this PR.
| list: training and testing split of both X and y inputs | ||
| """ | ||
| train, test = train_test_split(X.index, test_size=self.test_size, train_size=self.train_size, shuffle=self.shuffle, stratify=self.stratify, random_state=self.random_state) | ||
| return [(train, test)] |
| AutoMLSearch(problem_type='binary', data_split=data_splitter) | ||
|
|
||
|
|
||
| @patch('evalml.pipelines.BinaryClassificationPipeline.score') |
There was a problem hiding this comment.
@eccabay : can you please also add this line to all the tests where you mock score
@patch('evalml.pipelines.BinaryClassificationPipeline.fit')This means pipelines' fit methods won't run, and can speed up tests by 5-10x
There was a problem hiding this comment.
Interestingly enough, when I add this, none of the tests pass - the expected scores I set with mock_score don't match and/or an error is thrown complaining about the pipelines not being fit, and there's no change in the amount of time the tests take.
| with pytest.raises(ValueError, match="is not compatible with a"): | ||
| auto = AutoMLSearch(problem_type='binary', objective='R2') | ||
| auto.search(X, y, data_checks=EmptyDataChecks()) | ||
| AutoMLSearch(problem_type='binary', objective='R2') |
3 participants
Closes #840 by moving the cross validation argument in automl to a more generic data splitting argument that can handle a single training/validation split, to which it defaults if there are more than 100k rows in the training dataset.