Update dfs transformer features arg to be empty list for fast mode by tamargrey · Pull Request #3875 · alteryx/evalml

tamargrey · 2022-12-05T22:48:51Z

Avoids passing the DFS Transformer's full list of features into the cloned pipeline for partial dependence fast mode. To achieve this, updates the DFS Transformer to allow an empty list as input for the features arg.

Note: This PR is currently set to merge into #3873 because it's dependent on changes made in that PR. We should merge that first and then merge this into main.

codecov · 2022-12-05T22:57:00Z

Codecov Report

Merging #3875 (6c577a7) into main (9fedf3b) will increase coverage by 0.1%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##            main   #3875     +/-   ##
=======================================
+ Coverage   99.7%   99.7%   +0.1%     
=======================================
  Files        344     344             
  Lines      36209   36262     +53     
=======================================
+ Hits       36072   36125     +53     
  Misses       137     137

Impacted Files	Coverage Δ
evalml/automl/automl_search.py	`99.5% <ø> (ø)`
...ponents/transformers/preprocessing/featuretools.py	`100.0% <100.0%> (ø)`
evalml/tests/automl_tests/test_automl.py	`99.5% <100.0%> (+0.1%)`	⬆️
evalml/tests/component_tests/test_featuretools.py	`100.0% <100.0%> (ø)`
...del_understanding_tests/test_partial_dependence.py	`100.0% <100.0%> (ø)`

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

tamargrey · 2022-12-05T23:03:48Z

        self.index = index
        self.features = features
-        self._passed_in_features = True if features else None
+        self._passed_in_features = True if features is not None else None


To make this fix, I needed the ability to set the parameters of the DFS Transformer so that a transform call would be no op. One option I considered was only passing in the IdentityFeature objects in at _handle_partial_dependence_fast_mode, but I felt that left some ambiguity as to what the behavior would be.

Therefore, I updated the behavior here to no longer treat an empty list passed in for features the same as features being unspecified. This change made sense to me, but I wanted to highlight it in case anyone disagrees or thinks this could be problematic.

I am a little curious about this. It opens up the opportunity for a user to pass in features=[] during AutoMLSearch, which would turn the whole component into a no-op. My intuition says we might want to add some sort of warning to make users aware of what they're doing - though it may be a weird warning in the pd fast mode case since users aren't controlling it there.
Does anyone else have an opinion about this?

Maybe the warning could come from AutoMLSearch itself. If it sees that a user is trying to have a DFS Transformer in automl search with no features, it could raise that warning. If it's passed directly into a pipeline, maybe we're okay not warning users since it might be more clear to users at that point what the impact would be.

Then again, my goal here was specifically to have no-op functionality in the DFS Transformer, but I definitely get the discomfort. At the very least, I need to update the dosctring making this explicit.

Ah, the warning within AutoMLSearch is a good idea. Having it in the docstring is also a super good callout, sounds like a good path forward to me

Turns out AutomlSearch doesn’t add the DFS Transformer to pipelines if features is an empty list.

We definitely add it to the internal search parameters here because we explicitly check if it’s None or not, but when we make_pipeline we exclude it if features is falsy: https://github.com/alteryx/evalml/blob/main/evalml/pipelines/utils.py#L412. We do the same for split pipelines. So currently in automl, an empty features list already is no-op.

My preferance here would be to not touch the automl code (though, maybe it’d be worth adding a note to the docstring there about the behavior when features is an empty list). And it makes me more confident that we probably want this behavior for the component itself as well.

Agreed, I do think the distinction between an empty list and None can be murky to new users. I think it's mostly mitigated by a docstring callout though.

Yep, updated the automl docstring to make this clear there as well

tamargrey · 2022-12-05T23:08:11Z

+
+    assert not mock_calculate_feature_matrix.called
+    # --> Going through automl, because we have to instantiate the DFSTransformer to make a pipeline directly,
+    # which means we don't get the updated parameters applied, causing this test to fail


This was an odd behavior I noticed while testing: Calling pipeline.new with an updated features parameter for the DFS Transformer wasn't resetting the features on the new pipeline. After digging into what was happening, it was because I was defining my pipeline with an instantiated DFSTransformer object DFSTransformer(features=features) so that I could specify features.

It seems at handle_component_class, we’re returning the instantiated component_class with features defined, which is why the features param is ignored when we try to instantiate the component with the new parameters.

Going to automl was my workaround for this, but I feel like there must be a better way. Is this a known behavior?

If I do need to keep this, I'll plan to use the AutoMLTestEnv

eccabay

Looks pretty good! Just left a few clarification questions.

eccabay · 2022-12-07T13:51:11Z

        self.index = index
        self.features = features
-        self._passed_in_features = True if features else None
+        self._passed_in_features = True if features is not None else None


I am a little curious about this. It opens up the opportunity for a user to pass in features=[] during AutoMLSearch, which would turn the whole component into a no-op. My intuition says we might want to add some sort of warning to make users aware of what they're doing - though it may be a weird warning in the pd fast mode case since users aren't controlling it there.
Does anyone else have an opinion about this?

eccabay · 2022-12-07T14:06:55Z

+            # Pass in empty list of features so we don't run calculate feature matrix
+            # which would happen with the full set of features for a single column at refit
+            dfs_transformer["features"] = []
+            pipeline_parameters["DFS Transformer"] = dfs_transformer


A clarification for my own understanding: we only use this function with the single column pipeline, correct? This code would never run when we're generating the transformations from the full dataset?

Correct - the parameters returned here will only be used to train a pipeline on a single column, and the check above this that confirms all features have corresponding columns in X confirms that calculate feature matrix is never called in the original pipeline; this parameter change makes that the case for the single column as well.

Is it not possible to just do:

pipeline_parameters["DFS Transformer"]["features"] = []

yep totally doable

eccabay · 2022-12-07T14:08:52Z

+    """Confirms that the features arg being an empty list is not treated the same as
+    it being unspecified."""


Can we be explicit about testing this? i.e. generate two DFS objects, one with features=[] and one with features=None and test the differences?

Added a check with features=None to this test and confirmed that dfs and calculate feature matrix get called and that the component results in features being made

eccabay · 2022-12-07T14:13:16Z

+
+    assert part_dep.feature_values.notnull().all()
+    assert part_dep.partial_dependence.notnull().all()
+    pd.testing.assert_frame_equal(part_dep, fast_part_dep)


Another question just for my own understanding: does this test fail with our current main branch?

Yes, it does fail, as calculate_feature_matrix gets called in fast mode causing the mock checks above to fail

Ah gotcha. I guess I'm more concerned about the pd.testing.assert_frame_equal(part_dep, fast_part_dep) line specifically, since that's what this PR is aimed at combatting

…nd docs

jeremyliweishih

LGTM - just something to remove

jeremyliweishih · 2022-12-07T17:34:32Z

 from collections import OrderedDict, defaultdict
 from itertools import product
 from math import ceil
+from pdb import set_trace


Need to remove this!

chukarsten · 2022-12-07T17:28:18Z

        self.index = index
        self.features = features
-        self._passed_in_features = True if features else None
+        self._passed_in_features = True if features is not None else None


Agreed, I do think the distinction between an empty list and None can be murky to new users. I think it's mostly mitigated by a docstring callout though.

chukarsten · 2022-12-07T17:32:46Z

+            # Pass in empty list of features so we don't run calculate feature matrix
+            # which would happen with the full set of features for a single column at refit
+            dfs_transformer["features"] = []
+            pipeline_parameters["DFS Transformer"] = dfs_transformer


Is it not possible to just do:

pipeline_parameters["DFS Transformer"]["features"] = []

chukarsten · 2022-12-07T17:37:07Z

 from collections import OrderedDict, defaultdict
 from itertools import product
 from math import ceil
+from pdb import set_trace


removed 🙈

Tamar Grey added 5 commits December 5, 2022 13:47

Handle features with dataframe names that aren't 'X'

1880295

Explain how this change relates to TODO

ed72ea6

Add release note

7f845a5

Explain assumption around dataframe name

4c37408

Fix punctuation

13b17b2

auto-assign bot assigned tamargrey Dec 5, 2022

tamargrey force-pushed the fix-fast-mode-features-arg branch from 623ad8d to b497fb8 Compare December 5, 2022 22:50

tamargrey commented Dec 5, 2022

View reviewed changes

tamargrey requested review from chukarsten, eccabay and jeremyliweishih December 5, 2022 23:09

chukarsten and others added 4 commits December 6, 2022 09:07

Merge branch 'main' into fix-dfs-feature-df-names

81d6e77

cleanup

f45370d

Update dfs transformer features arg to be empty list for fast mode

0229d99

Add release note

a412921

tamargrey force-pushed the fix-fast-mode-features-arg branch from 136edbb to a412921 Compare December 6, 2022 16:08

remove comment

c4aaeaa

Base automatically changed from fix-dfs-feature-df-names to main December 6, 2022 16:49

Tamar Grey added 2 commits December 6, 2022 11:53

Merge branch 'main' into fix-fast-mode-features-arg

4a6e511

Merge branch 'main' into fix-fast-mode-features-arg

54fce56

eccabay approved these changes Dec 7, 2022

View reviewed changes

make behavior of empty list of features more explicit through tests a…

6baa45a

…nd docs

jeremyliweishih approved these changes Dec 7, 2022

View reviewed changes

chukarsten approved these changes Dec 7, 2022

View reviewed changes

Tamar Grey added 2 commits December 7, 2022 12:43

PR comments

ef6282a

Merge branch 'main' into fix-fast-mode-features-arg

6c577a7

chukarsten merged commit eb02c44 into main Dec 7, 2022

chukarsten deleted the fix-fast-mode-features-arg branch December 7, 2022 19:08

christopherbunn mentioned this pull request Dec 8, 2022

Release v0.64.0 #3879

Merged

		"""Confirms that the features arg being an empty list is not treated the same as
		it being unspecified."""

Conversation

tamargrey commented Dec 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Dec 5, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eccabay left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeremyliweishih left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tamargrey commented Dec 5, 2022 •

edited

Loading

codecov bot commented Dec 5, 2022 •

edited

Loading