Add text processing component #913

eccabay · 2020-07-07T19:24:34Z

Closes #907 by adding a brand new text featurization component that wraps the featuretools nlp-primitives code. Tests at the moment are fairly minimal, so any suggestions on what to add are much appreciated!

codecov · 2020-07-07T19:34:02Z

Codecov Report

Merging #913 into main will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff            @@
##             main     #913    +/-   ##
========================================
  Coverage   99.83%   99.83%            
========================================
  Files         168      170     +2     
  Lines        8366     8578   +212     
========================================
+ Hits         8352     8564   +212     
  Misses         14       14

Impacted Files	Coverage Δ
evalml/pipelines/components/__init__.py	`100.00% <ø> (ø)`
...alml/pipelines/components/transformers/__init__.py	`100.00% <100.00%> (ø)`
.../components/transformers/preprocessing/__init__.py	`100.00% <100.00%> (ø)`
...s/transformers/preprocessing/text_featurization.py	`100.00% <100.00%> (ø)`
...l/tests/component_tests/test_text_featurization.py	`100.00% <100.00%> (ø)`
evalml/tests/component_tests/test_utils.py	`100.00% <100.00%> (ø)`
evalml/tests/pipeline_tests/test_graph_utils.py	`100.00% <100.00%> (ø)`
evalml/tests/utils_tests/test_cli_utils.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3077671...0298829. Read the comment docs.

freddyaboulton

@eccabay Left a comment about making sure the column names passed in __init__ are actually in the dataframe and some questions. Overall, looks good to me!

freddyaboulton · 2020-07-07T19:35:29Z

evalml/tests/component_tests/test_text_featurization.py

+def text_df():
+    df = pd.DataFrame(
+        {'col_1': ['I\'m singing in the rain!$%^ do do do do do da do', 'just singing in the rain.................. \n', '\t\n\n\n\nWhat a glorious feelinggggggggggg, I\'m happy again!!! lalalalalalalalalalala'],
+         'col_2': ['do you hear the people sing?////////////////////////////////////', 'singing the songs of angry men\n', '\tIt is the music of a people who will NOT be slaves again!!!!!!!!!!!']})


evalml/pipelines/components/transformers/preprocessing/text_featurization.py

evalml/tests/component_tests/test_text_featurization.py

evalml/pipelines/components/transformers/preprocessing/text_featurization.py

dsherry

Nice!! Looking good. Left a few comments.

The nlp-primitives repo has unit tests which check the output from each primitive is correct. We don't have that level of coverage yet here but I think we should. My suggestion would be to port the relevant nlp-primitives tests to our repo. Happy to talk more about that, there's definitely a few options.

I see we'll doing the addition to automl in a separate PR in #908. Sounds good to me!

After we merge this PR, featuretools will be a core dependency on evalml. I think before the July milestone goes out, we should make it optional and move it to requirements.txt!

dsherry · 2020-07-09T15:28:01Z

core-requirements.txt

@@ -3,6 +3,8 @@ pandas>=0.25.0
 scipy>=1.2.1
 scikit-learn>=0.21.3,!=0.22,<0.23.0
 scikit-optimize
+featuretools[nlp_primitives]
+nlp-primitives


Is this necessary? I see the instructions for nlp-primitives on pypi say to install featuretools[nlp_primitives]

That was what I tried at first, but CircleCI couldn't import the nlp-primitives without the second line - take a look at some of my earlier commits and you can see that none of the tests were able to run.

this might be a bug in featuretools cc @rwedge

there was a bug when installing the nlp-primitives 0.3.0 via pip, may have been fixed by 0.3.1

Thanks @rwedge

Ah, I see 0.3.1 went out 1 hour ago 😆

@eccabay since it's late on Friday, I suggest you merge this as-is, and we can deal with it later. But, hopefully with the update its now fine to delete this line

Even with nlp 0.3.1

pip install featuretools[nlp_primitives]
doesn't install properly

pip install nlp_primitives works though

evalml/pipelines/components/transformers/preprocessing/text_featurization.py

dsherry · 2020-07-09T15:45:09Z

evalml/tests/pipeline_tests/test_graph_utils.py

@@ -171,6 +173,7 @@ def test_normalize_confusion_matrix(data_type):

 def test_normalize_confusion_matrix_error():
    conf_mat = np.array([[0, 0, 0], [0, 0, 0], [0, 0, 0]])
+    warnings.simplefilter('default', category=RuntimeWarning)


👍 nice. So to recap the discussion we had offline with @gsheni , this is necessary because featuretools runs warnings.simplefilter("ignore", category=RuntimeWarning) on import. But normalize_confusion_matrix listens for a particular warning and turns it into a ValueError. And this was recently fixed in featuretools and will be in the next release.

dsherry · 2020-07-09T19:20:00Z

evalml/pipelines/components/transformers/preprocessing/text_featurization.py

+                         random_state=random_state)
+
+    @property
+    def features(self):


Why expose this? I think we should delete this getter.

I use the features in the tests to verify the component fit/transformed properly, I added this so the tests didn't have to access the private attribute.

Ah, got it. In that case, let's delete this getter and instead check that transform creates the columns we expect it to, with the column names we expect.

evalml/pipelines/components/transformers/preprocessing/text_featurization.py

dsherry · 2020-07-10T14:12:50Z

@eccabay before I forget, could you please update this PR to point at main?

dsherry

@eccabay looks good to me! Nice going :) it's exciting we're getting so close to having our first active integration with featuretools!

After our discussion of LSA on the epic, I was poking around in the nlp-primitives code. Two points from that:

As you pointed out in the epic, we're currently training the TF-IDF on a predetermined corpus from nltk. We should take a look at training TF-IDF on the training data instead. This may require implementing this in the evalml code instead of using a primitive.
If we stick with the predetermined corpus: I see we currently call nltk.download to grab the corpus. We should change that before we include this in automl. We should be able to run all components with or without network access.

dsherry · 2020-07-10T18:53:47Z

evalml/tests/component_tests/test_text_featurization.py

+    X_t = tf.transform(X)
+    cols = [col for col in X_t.columns if 'LSA' in col]
+    features = X_t[cols]
+    np.testing.assert_almost_equal(features, expected_features, decimal=3)


Awesome, thanks for adding these!

dsherry · 2020-07-10T19:44:46Z

evalml/tests/component_tests/test_text_featurization.py

+        expected_col_names.add(f'PART_OF_SPEECH_COUNT(col_1)[{i}]')
+        expected_col_names.add(f'PART_OF_SPEECH_COUNT(col_2)[{i}]')
+    X_t = tf.transform(X)
+    assert set(X_t.columns) == expected_col_names


Nice!

Minor but since the feature order is (or should be?) deterministic, would be nice not to use set and test the ordering too

dsherry · 2020-07-10T20:37:14Z

evalml/tests/latest_dependency_versions.txt

@@ -1,5 +1,6 @@
 catboost==0.23.2
 cloudpickle==1.5.0
+distributed==2.20.0


I pushed a commit to add this. I think 0.3.1 of nlp-primitives which rolled out an hour ago added this package.

featuretools uses distributed

rwedge · 2020-07-10T20:38:20Z

As you pointed out in the epic, we're currently training the TF-IDF on a predetermined corpus from nltk. We should take a look at training TF-IDF on the training data instead. This may require implementing this in the evalml code instead of using a primitive.

Perhaps one approach would be to update the primitive to accept the training data as a corpus to use instead of the predetermined one.

freddyaboulton

@eccabay Looks good to me!

kmax12 · 2020-07-10T21:58:27Z

evalml/pipelines/components/transformers/preprocessing/text_featurization.py

+        X_text['index'] = range(len(X_text))
+
+        es = ft.EntitySet()
+        es = es.entity_from_dataframe(entity_id='X', dataframe=X_text, index='index')


just seeing this now, but i think we should be using the variable_types argument to specify which columns are are text columns when calling this function. you can see any example here: https://docs.featuretools.com/en/stable/loading_data/using_entitysets.html#adding-entities

eccabay added 8 commits July 6, 2020 09:31

Add text featurization component files

2f264ad

Implement text featurization component

5181228

Add preliminary component tests

2570020

Add featuretools to core-requirements

96e2fa1

Fix import statements

0dfde3a

Get nlp_primitives in core_requirements (hopefully)

a5ca9ba

try 2

c29ae1e

workaround for importing featuretools bug

f44d925

eccabay requested review from dsherry, ctduffy and freddyaboulton July 7, 2020 19:24

auto-assign bot assigned eccabay Jul 7, 2020

update changelog

2ef4bda

freddyaboulton suggested changes Jul 7, 2020

View reviewed changes

Add column checking and validation

48da5e6

ctduffy reviewed Jul 8, 2020

View reviewed changes

evalml/tests/component_tests/test_text_featurization.py Show resolved Hide resolved

ctduffy reviewed Jul 8, 2020

View reviewed changes

evalml/pipelines/components/transformers/preprocessing/text_featurization.py Outdated Show resolved Hide resolved

eccabay and others added 3 commits July 8, 2020 14:58

Add column data type validation

e8c422b

Merge branch 'master' into 907_text_component

6dc7990

Temporary fixes for failing tests caused by merge

daae51f

eccabay requested a review from freddyaboulton July 9, 2020 12:27

eccabay added 2 commits July 9, 2020 09:20

Add support for non-string column names

c056f24

Standardize transform output

d06092a

ctduffy reviewed Jul 9, 2020

View reviewed changes

evalml/pipelines/components/transformers/preprocessing/text_featurization.py Outdated Show resolved Hide resolved

eccabay added 2 commits July 9, 2020 13:20

Add small amount of text cleaning

48065d5

Add component to API reference

34bdd4d

dsherry suggested changes Jul 9, 2020

View reviewed changes

eccabay added 2 commits July 10, 2020 08:34

Rename TextFeaturization to TextFeaturizer

f4792a4

Update TextFeaturizer parameters and defaults

7853d8d

Add primitive output testing

82f1484

eccabay changed the base branch from master to main July 10, 2020 14:14

Merge branch 'main' into 907_text_component

c35bc04

eccabay requested a review from dsherry July 10, 2020 14:24

eccabay and others added 2 commits July 10, 2020 15:13

Remove features getter

73fa62a

Merge branch 'main' into 907_text_component

791413b

dsherry approved these changes Jul 10, 2020

View reviewed changes

Update expected deps, from new featuretools dependency

0298829

dsherry reviewed Jul 10, 2020

View reviewed changes

freddyaboulton approved these changes Jul 10, 2020

View reviewed changes

eccabay merged commit 923eb25 into main Jul 10, 2020

kmax12 reviewed Jul 10, 2020

View reviewed changes

This was referenced Jul 10, 2020

Error in pip install and warning on import after adding text/featuretools #925

Closed

Make some text featurizer component dependencies optional, not required #926

Closed

Release v0.11.2 #939

Merged

dsherry deleted the 907_text_component branch October 29, 2020 23:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add text processing component #913

Add text processing component #913

eccabay commented Jul 7, 2020

codecov bot commented Jul 7, 2020 •

edited

Loading

freddyaboulton left a comment

freddyaboulton Jul 7, 2020

dsherry left a comment

dsherry Jul 9, 2020

eccabay Jul 10, 2020

kmax12 Jul 10, 2020 •

edited

Loading

rwedge Jul 10, 2020

dsherry Jul 10, 2020

rwedge Jul 10, 2020

rwedge Jul 10, 2020

dsherry Jul 9, 2020

dsherry Jul 9, 2020

eccabay Jul 10, 2020

dsherry Jul 10, 2020

dsherry commented Jul 10, 2020

dsherry left a comment •

edited

Loading

dsherry Jul 10, 2020

dsherry Jul 10, 2020

dsherry Jul 10, 2020

rwedge Jul 10, 2020

rwedge commented Jul 10, 2020

freddyaboulton left a comment

kmax12 Jul 10, 2020

Add text processing component #913

Add text processing component #913

Conversation

eccabay commented Jul 7, 2020

codecov bot commented Jul 7, 2020 • edited Loading

Codecov Report

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmax12 Jul 10, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsherry commented Jul 10, 2020

dsherry left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rwedge commented Jul 10, 2020

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 7, 2020 •

edited

Loading

kmax12 Jul 10, 2020 •

edited

Loading

dsherry left a comment •

edited

Loading