Primitive refactor #364

kmax12 · 2019-01-07T18:08:28Z

This PR is separates the concept of a Primitive from a Feature. The internals of Featuretools change, but it is has minimal impact on the external API.

Compared to before, a primitive is now only aware of the data it take in. A feature is then defined by input variables and/or features, as well as the primitive that will be applied. Put another way, a feature takes the specific entities and variables of an entityset and the primitive to be applied as its input so the primitive doesn't have to work about it.

This has several advantages

It is easier to unit test primitives. There is no need to have an entity set to test a primitive
Primitive definitions become more reusable since they are not tied to the concept of entity set.
Conceptually the user defining a primitive has to only think about input and output data which is just numpy arrays.

To give an example, here is how a feature is currently defined using the count primitive

from featuretools.primitive import Count
f = Count(es["logs"]["value"], parent_entity=es["customers"])

Now, you define the inputs to the feature and provide the primitive as an input.

import featuretools as ft
from featuretools.primitive import Count
f = ft.Feature(es["logs"]["value"], parent_entity=es["customers"], primitive=Count)

if a primitive has parameters it can be used like this

f = ft.Feature(es["logs"]["comment"], parent_entity=es["customers"], primitive=CountString(string="coffee"))

the API for calling DFS doesn't change with the exception of being able to provide primitive with arguments

ft.dfs(target_entity="customers",
       entityset=es,
       agg_primtives=["count", Sum],
       trans_primtiives=CountString(string="coffee"))

codecov · 2019-01-07T21:02:43Z

Codecov Report

Merging #364 into master will increase coverage by 0.2%.
The diff coverage is 97.48%.

@@            Coverage Diff            @@
##           master     #364     +/-   ##
=========================================
+ Coverage   95.33%   95.53%   +0.2%     
=========================================
  Files          86       89      +3     
  Lines        8032     7555    -477     
=========================================
- Hits         7657     7218    -439     
+ Misses        375      337     -38

Impacted Files	Coverage Δ
featuretools/primitives/api.py	`100% <ø> (ø)`	⬆️
featuretools/wrappers/sklearn.py	`95.65% <ø> (ø)`	⬆️
featuretools/synthesis/dfs.py	`100% <ø> (ø)`	⬆️
featuretools/utils/pickle_utils.py	`100% <ø> (ø)`	⬆️
featuretools/selection/variance_selection.py	`0% <ø> (ø)`	⬆️
featuretools/synthesis/encode_features.py	`98.03% <ø> (ø)`	⬆️
featuretools/selection/selection.py	`100% <ø> (ø)`	⬆️
featuretools/synthesis/deep_feature_synthesis.py	`93.52% <100%> (+0.06%)`	⬆️
featuretools/feature_base/api.py	`100% <100%> (ø)`
...aturetools/tests/entityset_tests/test_timedelta.py	`100% <100%> (ø)`	⬆️
... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ffb8081...40946ba. Read the comment docs.

* remove incorrect commutative attributes * handle rsub override and test reverse overrides * rename weekend primitive is_weekend * updated weekend to is_weekend in docs * test values for scalar_subtract_numeric * rename subtract_numeric and scalar_subtract_numeric to subtract_numeric_feature and scalar_subtract_numeric_feature * revert subtract_numeric_feature to subtract_numeric

rwedge · 2019-01-17T00:36:43Z

featuretools/feature_base/feature_base.py

+        Args:
+            entity (Entity): entity this feature is being calculated for
+            base_featres (list[FeatureBase]): list of base features for primitive
+            primitive (): primitive to calculate. if not initilized when passed, gets initialized with no arguments


missing the type information for primitive

rwedge · 2019-01-17T16:11:49Z

featuretools/feature_base/feature_base.py

+        # if self.use_previous and self.use_previous.is_absolute():
+            # entity = self.entity
+            # time_var = IdentityFeature(entity[entity.time_index])
+            # deps += [time_var]


This has been commented out for a while, I think we should either remove it or make an issue about it.

the time index automatically gets added, so we don't need to put the time index feature as a dependent. removed

rwedge · 2019-01-17T17:05:56Z

featuretools/feature_base/feature_base.py

+
+        base_entity = set([f.entity for f in base_features])
+        assert len(base_entity) == 1, \
+            "More than one entity for base features"


There's potentially two checks in this init for whether the base features share the same entity.

good catch. fixed

featuretools/tests/computational_backend/test_calculate_feature_matrix.py

featuretools/tests/computational_backend/test_pandas_backend.py

featuretools/feature_base/feature_base.py

rwedge · 2019-01-17T22:05:51Z

featuretools/tests/dfs_tests/test_deep_feature_synthesis.py

-    seed_feature_sessions = Count(es['log']["id"], es['sessions']) > 2
-    seed_feature_log = Hour(es['log']['datetime'])
-    session_agg = Last(seed_feature_log, es['sessions'])
+    seed_feature_sessions = ft.Feature(es['log']["id"], parent_entity=es['sessions'], primitive=Count)


I don't think it changes the test but seed_feature_sessions is missing the > 2

ya, i noticed that. i dont think adding the >2 improved the test so i simplified it

kmax12 added 27 commits December 31, 2018 17:52

initial support

06256c6

cfm tests passing

a0cd617

linting

2de087c

progress

d0e786d

progress

d36701d

add lint-fix and autopep

e03bf2d

more passing

352ff58

progress

133e888

progress

aea1236

progress

e592ad3

don't require initialization

f2dab2f

separate override tests

33fe750

lint

c37161c

cleanup

ed11264

more tests passing

4674174

one more test passing

0765d96

all but one test passing

fc68f4b

all test pass

8c8d73d

linting

83f63ad

add to lint-fix

cec3928

all tests pass!

67b0244

cleanup

1567b2c

fix tests

aad7553

fix test

2bd20f7

fix linting

42945aa

docs building

869c8a8

actually fix docs

6e6ead3

kmax12 and others added 2 commits January 7, 2019 18:53

updates

39ad144

kmax12 added 4 commits January 8, 2019 16:32

update rsub

91eda86

Merge branch 'master' into primitive-refactor

709dc66

fix test

8dd67de

Merge branch 'master' into primitive-refactor

a472126

kmax12 changed the title ~~[WIP] Primitive refactor~~ Primitive refactor Jan 16, 2019

cleanup

794b2c8

kmax12 requested a review from rwedge January 16, 2019 21:56

rwedge reviewed Jan 17, 2019

View reviewed changes

remove unecessary primitive initializations

05e3183

rwedge suggested changes Jan 17, 2019

View reviewed changes

addressed review comments

40946ba

rwedge approved these changes Jan 18, 2019

View reviewed changes

rwedge merged commit 36ce3c3 into master Jan 18, 2019

bukosabino mentioned this pull request Jan 21, 2019

update to v0.6.0 Featuretools (using primitive refactor) alteryx/predict-olympic-medals#5

Merged

rwedge mentioned this pull request Jan 30, 2019

v0.6.0 #387

Merged

rwedge deleted the primitive-refactor branch February 19, 2021 21:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Primitive refactor #364

Primitive refactor #364

kmax12 commented Jan 7, 2019

codecov bot commented Jan 7, 2019 •

edited

Loading

rwedge Jan 17, 2019

rwedge Jan 17, 2019

kmax12 Jan 18, 2019

rwedge Jan 17, 2019

kmax12 Jan 18, 2019 •

edited

Loading

rwedge Jan 17, 2019

kmax12 Jan 18, 2019

Primitive refactor #364

Primitive refactor #364

Conversation

kmax12 commented Jan 7, 2019

codecov bot commented Jan 7, 2019 • edited Loading

Codecov Report

rwedge Jan 17, 2019

Choose a reason for hiding this comment

rwedge Jan 17, 2019

Choose a reason for hiding this comment

kmax12 Jan 18, 2019

Choose a reason for hiding this comment

rwedge Jan 17, 2019

Choose a reason for hiding this comment

kmax12 Jan 18, 2019 • edited Loading

Choose a reason for hiding this comment

rwedge Jan 17, 2019

Choose a reason for hiding this comment

kmax12 Jan 18, 2019

Choose a reason for hiding this comment

codecov bot commented Jan 7, 2019 •

edited

Loading

kmax12 Jan 18, 2019 •

edited

Loading