Integrate TextFeaturizer to automl #1062

eccabay · 2020-08-14T16:45:16Z

Closes #908 and fixes #978 by integrating the TextFeaturizer component into automl and adding nlp_primitives to core-requirements.txt. The component lives between the Imputer and the OneHotEncoder in dynamically generated pipeline graphs.

Performance tests are not at a point where graphs can be generated, but I will post those here once available.

codecov · 2020-08-14T18:05:19Z

Codecov Report

Merging #1062 into main will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1062      +/-   ##
==========================================
+ Coverage   99.95%   99.95%   +0.01%     
==========================================
  Files         213      213              
  Lines       13835    13857      +22     
==========================================
+ Hits        13828    13850      +22     
  Misses          7        7

Impacted Files	Coverage Δ
evalml/__init__.py	`100.00% <100.00%> (ø)`
...lml/automl/automl_algorithm/iterative_algorithm.py	`100.00% <100.00%> (ø)`
evalml/automl/automl_search.py	`99.62% <100.00%> (+0.01%)`	⬆️
...ents/transformers/preprocessing/text_featurizer.py	`100.00% <100.00%> (ø)`
evalml/pipelines/utils.py	`100.00% <100.00%> (ø)`
evalml/tests/automl_tests/test_automl.py	`100.00% <100.00%> (ø)`
...lml/tests/automl_tests/test_iterative_algorithm.py	`100.00% <100.00%> (ø)`
...alml/tests/component_tests/test_text_featurizer.py	`100.00% <100.00%> (ø)`
evalml/tests/pipeline_tests/test_pipelines.py	`100.00% <100.00%> (ø)`
evalml/tests/utils_tests/test_cli_utils.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b40aae1...ba1be2f. Read the comment docs.

evalml/automl/automl_search.py

eccabay · 2020-10-27T17:40:09Z

evalml/automl/automl_search.py

@@ -389,9 +386,14 @@ def search(self, X, y, data_checks="auto", feature_types=None, show_iteration_pl
                X = pd.DataFrame(X)
            X = ww.DataTable(X)

+        text_column_vals = X.select('natural_language')


This produces a warning if there are no text columns in X, open to any suggestions on how to suppress that!

Good catch! I'll file that in woodwork. I don't think there should be a warning in this case; we can discuss on that ticket.

For this PR, you can add:

warnings.filterwarnings('ignore', 'The following selectors were not present in your DataTable')

I think this would this work?

from woodwork.logical_types import NaturalLanguage [col_name for col_name, ltype in X.logical_types.items() if ltype == NaturalLanguage]

@gsheni what do you think?

Filed alteryx/woodwork#322 for this warning

Yes, that should work. We may fix this due to the issue Dylan raised.

@freddyaboulton oops I had written back but didn't submit my comments until after you commented 😆

I think we can just suppress the warning, yeah?

freddyaboulton

@eccabay Looks good to me!

freddyaboulton · 2020-10-27T17:44:50Z

evalml/automl/automl_search.py

@@ -438,6 +440,7 @@ def search(self, X, y, data_checks="auto", feature_types=None, show_iteration_pl
            max_iterations=self.max_iterations,
            allowed_pipelines=self.allowed_pipelines,
            tuner_class=self.tuner_class,
+            text_columns=text_columns,


Do we need to make this change? Can't the logic for identifying the natural language columns be delegated to the fit method of the text transformers (if none are passed into init)?

@freddyaboulton you're right, we could delegate this to the fit method of the text transformers. However, I think this is better. This PR as it stands will use the woodwork datatable to detect and select text features. If users don't like the default selections, they can override and pass a new datatable into search. But if we had the text featurizer do the detection, users wouldn't be able to override as easily. This is one of the big value adds of the datatables for us--it provides users with a clear way to control how automl search treats each feature, and to change the behavior!

Long-term, we'd like all pipelines, components and utils to be able to accept woodwork datatables (#1288). At that point, we can delete this text_columns plumbing in favor of the datatables

freddyaboulton · 2020-10-27T18:03:18Z

evalml/tests/automl_tests/test_automl.py

+         })
+    y = [0, 1, 1, 0, 1, 0]
+    automl = AutoMLSearch(problem_type='binary')
+    automl.search(X, y, data_checks='disabled')


Can you please add a comment explaining why the data checks are disabled? Is it because there isn't enough data for 3-fold cv?

freddyaboulton · 2020-10-27T18:07:48Z

evalml/tests/automl_tests/test_iterative_algorithm.py

@@ -30,6 +31,16 @@ def test_iterative_algorithm_allowed_pipelines(logistic_regression_binary_pipeli
    assert algo.allowed_pipelines == allowed_pipelines


+class MockEstimator(Estimator):


We have a fixture for a mock classifier called dummy_classifier_estimator_class would that work in this test?

dsherry

@eccabay this is🔥 ! Really great test coverage.

All that's blocking IMO is:

I left a suggestion for how to suppress the woodwork warning in automl search text feature selection.
I left one question about test_iterative_algorithm_instantiates_text
Left comment about adding feature_types to breaking changes
Perf test results

dsherry · 2020-10-27T17:50:57Z

core-requirements.txt

+woodwork>=0.0.3
+featuretools>=0.20.0
+nlp-primitives>=1.1.0


Yep, nice!!

For others reading, featuretools appears to have pretty good backwards compatibility for dfs, so if we get a feature request to lower this, we can certainly do that. I bet the same is true with nlp-primitives, but I think its the right call to fix at 1.1.0, because anything at/after that version will have our new treatment for including the nltk corpuses during installation, rather than downloading at first run.

dsherry · 2020-10-27T17:51:32Z

docs/source/release_notes.rst

@@ -14,6 +14,7 @@ Release Notes
        * Updated ``AutoMLSearch`` to support ``Woodwork`` data structures :pr:`1299`
        * Added cv_folds to ``ClassImbalanceDataCheck`` and added this check to ``DefaultDataChecks`` :pr:`1333`
        * Make ``max_batches`` argument to ``AutoMLSearch.search`` public :pr:`1320`
+        * Included TextFeaturizer in options for automl search :pr:`1062`


Yep! To make this easier for users to understand, could you say "Added text support to automl search"?

dsherry · 2020-10-27T17:53:51Z

evalml/automl/automl_algorithm/iterative_algorithm.py

@@ -111,6 +113,10 @@ def _transform_parameters(self, pipeline_class, proposed_parameters):
            component_parameters = proposed_parameters.get(component_class.name, {})
            init_params = inspect.signature(component_class.__init__).parameters

+            # Add the text columns parameter if the component is a TextFeaturizer
+            if component_class.name == "Text Featurization Component":
+                component_parameters['text_columns'] = self._text_columns


Looks good.

I'm hoping that we can remove this pathway in #1229 , when we add support for woodwork to pipelines and components, because at that point the text featurizer can access the woodwork datatable directly!

dsherry · 2020-10-27T17:58:46Z

evalml/automl/automl_search.py

@@ -353,17 +353,14 @@ def _set_data_split(self, X):

        self.data_split = self.data_split or default_data_split

-    def search(self, X, y, data_checks="auto", feature_types=None, show_iteration_plot=True):


Great call! This is totally dead. I genuinely don't know why we didn't delete it sooner 😆

Because its changing the method signature of search, just for completeness, let's add an entry to the breaking changes, like

Removed unused argument feature_types from AutoMLSearch.search

dsherry · 2020-10-27T18:18:43Z

evalml/automl/automl_search.py

@@ -389,9 +386,14 @@ def search(self, X, y, data_checks="auto", feature_types=None, show_iteration_pl
                X = pd.DataFrame(X)
            X = ww.DataTable(X)

+        text_column_vals = X.select('natural_language')


Good catch! I'll file that in woodwork. I don't think there should be a warning in this case; we can discuss on that ticket.

For this PR, you can add:

warnings.filterwarnings('ignore', 'The following selectors were not present in your DataTable')

evalml/tests/automl_tests/test_automl.py

evalml/tests/automl_tests/test_iterative_algorithm.py

dsherry · 2020-10-27T18:40:16Z

evalml/tests/automl_tests/test_iterative_algorithm.py

+    pipeline = algo.next_batch()[0]
+    expected_params = {'text_columns': ['text_col_1', 'text_col_2']}
+    assert pipeline.parameters['Text Featurization Component'] == expected_params
+    assert isinstance(pipeline.component_graph[0], TextFeaturizer)


Hmm, wouldn't pipeline.component_graph[0] be the imputer in this case, and you want pipeline.component_graph[1]? What am I missing?

The pipeline in this test isn't dynamically generated, it's defined a few lines above as only having the TextFeaturizer and an estimator

Ah right, thanks!

evalml/tests/component_tests/test_text_featurizer.py

evalml/tests/pipeline_tests/test_pipelines.py

eccabay · 2020-10-28T15:00:09Z

Performance testing doc is here

dsherry · 2020-10-28T15:09:21Z

@eccabay thanks, I just reviewed the perf tests. Looks great!

dsherry · 2020-10-28T18:59:34Z

@eccabay to fix the check_dependencies_updated_linux failure, I think you gotta update evalml/tests/latest_dependency_versions.txt on your branch--it looks like moving featuretools/nlp_primitives to core added distributed, and that's currently on our watch-list in the circleci config for the job. We could also take it off the watch list, but for now let's just add it to the deps versions, yeah?

eccabay and others added 7 commits August 11, 2020 17:13

Add TextFeaturizer to generated components

2d07c48

Switch Imputer to after TextFeaturizer to get perf tests to run

0aca953

move text_columns in make_pipeline to be optional argument

168ef89

Add make_pipeline and iterative_algorithm tests

671d8e3

Fix output NaN bug and revert component order

92bd44a

Update release notes and lint fixes

04cd7d2

Merge branch 'main' into 908_integrate_text

444153a

eccabay and others added 22 commits August 18, 2020 16:13

Merge branch 'main' into 908_integrate_text

faada11

Remove Part of Speech count to test speed improvement

96d5b7e

Merge branch 'main' into 908_integrate_text

9c0a9de

Re-remove part of speech count primitive

8f9078e

Update requirements to use experimental nlp_primitives branch

39174ea

Add part of speech count back in (with faster primitives)

8fa5582

Add extra primitives for performance testing

8e0670b

Switch nlp_primitives branch to include_data (no downloads)

c794339

Switch back to previous nlp branch to see if it works

2a8f07f

now back again

dc66060

Merge branch 'main' into 908_integrate_text

72ff7fb

fix failing unit tests

697eab9

Add nlp git branch to core-requirements

2e7c7ff

Trivial change to test pushing

4a836ac

Revert back to release of nlp_primitives

281b47a

Comment out nlp_primitives from test-requirements

b832016

Merge branch 'main' into 908_integrate_text

2e55838

update Makefile to install nlp_primitives master

b21cf69

Merge branch 'main' into 908_integrate_text

0ca45b6

lint fix and add nlp_primitives code to all make targets

0b1f2a0

Move ft and nlp_primitives (v.1.1.0) to core-requirements

686c3f0

Lint and small test fixes

be6e1ef

eccabay and others added 2 commits October 27, 2020 08:47

Merge branch 'main' into 908_integrate_text

9843e48

Revert some weird testing errors

50d70c9

dsherry reviewed Oct 27, 2020

View reviewed changes

evalml/automl/automl_search.py Show resolved Hide resolved

dsherry mentioned this pull request Oct 27, 2020

Update components from selecting using pandas dtypes to selecting using DataTable semantic or logical types #1290

Closed

eccabay added 2 commits October 27, 2020 13:16

Update search to automatically select text_columns from woodwork

03004c3

Remove extraneous print statement

0a9bf04

eccabay marked this pull request as ready for review October 27, 2020 17:36

eccabay requested review from dsherry, freddyaboulton, bchen1116 and jeremyliweishih October 27, 2020 17:36

eccabay commented Oct 27, 2020

View reviewed changes

freddyaboulton approved these changes Oct 27, 2020

View reviewed changes

dsherry approved these changes Oct 27, 2020

View reviewed changes

eccabay added 2 commits October 27, 2020 15:50

Address PR comments

c6f1aea

Merge branch 'main' into 908_integrate_text

3d62541

eccabay and others added 3 commits October 28, 2020 15:15

Add distributed to latest_dependency_versions

b531005

Merge branch 'main' into 908_integrate_text

b4ae93a

Merge branch 'main' into 908_integrate_text

ba1be2f

eccabay merged commit 83343aa into main Oct 28, 2020

dsherry mentioned this pull request Oct 29, 2020

Release v0.15.0 #1370

Merged

eccabay deleted the 908_integrate_text branch November 2, 2020 16:33

eccabay mentioned this pull request Nov 4, 2020

Update pipelines and make_pipelines to accept Woodwork DataTables #1393

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate TextFeaturizer to automl #1062

Integrate TextFeaturizer to automl #1062

eccabay commented Aug 14, 2020 •

edited

Loading

codecov bot commented Aug 14, 2020 •

edited

Loading

eccabay Oct 27, 2020

dsherry Oct 27, 2020

freddyaboulton Oct 27, 2020 •

edited

Loading

dsherry Oct 27, 2020

gsheni Oct 27, 2020

dsherry Oct 27, 2020

freddyaboulton left a comment

freddyaboulton Oct 27, 2020 •

edited

Loading

dsherry Oct 27, 2020

dsherry Oct 27, 2020

freddyaboulton Oct 27, 2020

freddyaboulton Oct 27, 2020 •

edited

Loading

dsherry left a comment

dsherry Oct 27, 2020

dsherry Oct 27, 2020

dsherry Oct 27, 2020

dsherry Oct 27, 2020

dsherry Oct 27, 2020

dsherry Oct 27, 2020

eccabay Oct 27, 2020

dsherry Oct 27, 2020

eccabay commented Oct 28, 2020

dsherry commented Oct 28, 2020

dsherry commented Oct 28, 2020

		@@ -30,6 +31,16 @@ def test_iterative_algorithm_allowed_pipelines(logistic_regression_binary_pipeli
		assert algo.allowed_pipelines == allowed_pipelines


		class MockEstimator(Estimator):

		@@ -353,17 +353,14 @@ def _set_data_split(self, X):

		self.data_split = self.data_split or default_data_split

		def search(self, X, y, data_checks="auto", feature_types=None, show_iteration_plot=True):

Integrate TextFeaturizer to automl #1062

Integrate TextFeaturizer to automl #1062

Conversation

eccabay commented Aug 14, 2020 • edited Loading

codecov bot commented Aug 14, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Oct 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton left a comment

Choose a reason for hiding this comment

freddyaboulton Oct 27, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

freddyaboulton Oct 27, 2020 • edited Loading

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eccabay commented Oct 28, 2020

dsherry commented Oct 28, 2020

dsherry commented Oct 28, 2020

eccabay commented Aug 14, 2020 •

edited

Loading

codecov bot commented Aug 14, 2020 •

edited

Loading

freddyaboulton Oct 27, 2020 •

edited

Loading

freddyaboulton Oct 27, 2020 •

edited

Loading

freddyaboulton Oct 27, 2020 •

edited

Loading