Grouby transform feature #455

rwedge · 2019-03-06T16:21:15Z

This PR creates a new feature class: GroupByTransformFeature

The idea behind this new class of features is to simplify writing primitives like CumulativeSum by handling the grouping necessary for the calculation elsewhere. In order to calculate the cumulative sum of a customer's purchases, you would first need to group the purchases by customer. With this new GroupByTransformFeature class, the primitive function no longer needs to account for this. It can assume the data passed to it has already been grouped.

codecov · 2019-03-06T16:30:32Z

Codecov Report

Merging #455 into master will increase coverage by 0.05%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #455      +/-   ##
==========================================
+ Coverage   96.24%   96.29%   +0.05%     
==========================================
  Files         102      103       +1     
  Lines        8724     8845     +121     
==========================================
+ Hits         8396     8517     +121     
  Misses        328      328

Impacted Files	Coverage Δ
featuretools/feature_base/api.py	`100% <ø> (ø)`	⬆️
...s/tests/primitive_tests/test_transform_features.py	`98.08% <ø> (-0.45%)`	⬇️
...turetools/computational_backends/pandas_backend.py	`98.36% <100%> (+0.16%)`	⬆️
...imitive_tests/test_groupby_transform_primitives.py	`100% <100%> (ø)`
...tools/primitives/standard/cum_transform_feature.py	`100% <100%> (ø)`	⬆️
featuretools/feature_base/feature_base.py	`96.46% <100%> (+0.24%)`	⬆️
...eaturetools/computational_backends/feature_tree.py	`100% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e2e5de...49aa411. Read the comment docs.

kmax12 · 2019-03-06T16:47:22Z

featuretools/tests/primitive_tests/test_groupby_primitives.py

@@ -0,0 +1,292 @@
+import numpy as np


let's name this file test_groupby_transform_primitives.py

featuretools/computational_backends/feature_tree.py

kmax12 · 2019-03-14T18:04:54Z

featuretools/computational_backends/pandas_backend.py

+                set_default_column(frame, f)
+                continue
+
+            groupby = f.groupby.get_name()


if we make the change to feature tree, we can ensure that all features have the same groupby once they get to this step. this means we can do the iterations of for index, group in frame_data.groupby(groupby) only once per group and calculate all the features at once

kmax12 · 2019-03-14T18:07:19Z

featuretools/computational_backends/pandas_backend.py

@@ -277,7 +280,9 @@ def generate_default_df(self, instance_ids, extra_columns=None):
        return default_df

    def _feature_type_handler(self, f):
-        if isinstance(f, TransformFeature):
+        if isinstance(f, GroupByTransformFeature):


might be worth leaving a comment saying the GroupByTransformFeature has to come first since it is a subclass of TransformFeature, unless we change it as suggested above

kmax12 · 2019-03-14T18:09:02Z

featuretools/feature_base/feature_base.py

+        self.groupby = groupby
+        super(GroupByTransformFeature, self).__init__(base_features,
+                                                      primitive=primitive)
+        self.base_features.append(groupby)


why not add this to base_features before calling the super method?

also, not sure it is worth subclassing TransformFeature. Doesn't seem like they actually share much.

kmax12 · 2019-03-14T18:13:39Z

featuretools/primitives/standard/cum_transform_feature.py

@@ -31,67 +29,51 @@ class CumCount(TransformPrimitive):

    def get_function(self):
        def cum_count(values):
-            return values.groupby(values).cumcount() + 1
+            return pd.Series(range(1, len(values) + 1), index=values.index)


another option that is easier to read would be. not sure which is more performant though

values.iloc[:] = range(1, len(values) + 1)

in case anyone's curious, I ran some benchmarks

In [1]: import pandas as pd ...: values = pd.Series(range(100000)) In [2]: %timeit pd.Series(range(1, len(values) + 1), index=values.index) 118 µs ± 1.43 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) In [3]: %timeit values.iloc[:] = range(1, len(values) + 1) 54 ms ± 916 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) In [4]: %timeit np.arange(1, len(values)) 50.1 µs ± 555 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) In [5]: %timeit pd.Series(np.arange(1, len(values) + 1), index=values.index) 107 µs ± 835 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

i think we should just do the 3rd option since pandas backend can handle the conversion to a series for us

kmax12 · 2019-03-22T14:55:16Z

featuretools/primitives/standard/cum_transform_feature.py

@@ -31,67 +29,51 @@ class CumCount(TransformPrimitive):

    def get_function(self):
        def cum_count(values):
-            return values.groupby(values).cumcount() + 1
+            return pd.Series(range(1, len(values) + 1), index=values.index)


in case anyone's curious, I ran some benchmarks

In [1]: import pandas as pd ...: values = pd.Series(range(100000)) In [2]: %timeit pd.Series(range(1, len(values) + 1), index=values.index) 118 µs ± 1.43 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each) In [3]: %timeit values.iloc[:] = range(1, len(values) + 1) 54 ms ± 916 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) In [4]: %timeit np.arange(1, len(values)) 50.1 µs ± 555 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each) In [5]: %timeit pd.Series(np.arange(1, len(values) + 1), index=values.index) 107 µs ± 835 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

i think we should just do the 3rd option since pandas backend can handle the conversion to a series for us

featuretools/computational_backends/pandas_backend.py

kmax12 · 2019-03-22T16:33:44Z

featuretools/computational_backends/pandas_backend.py

+
+                if not isinstance(values, pd.Series):
+                    values = pd.Series(values, index=variable_data[0].index)
+                group_values[f.hash()].append(values)


let's try doing just

frame[f.get_name()].update(values)

then we can remove null handling below

kmax12 · 2019-03-22T16:58:38Z

featuretools/computational_backends/pandas_backend.py

+                else:
+                    values = feature_func(*variable_data)
+
+                if not isinstance(values, pd.Series):


# make sure index is aligned for update call below if isinstance(values, pd.Series): values.index = variable_data[0].index else: values = pd.Series(values, index=variable_data[0].index) group_values[f.hash()].append(values)

kmax12 · 2019-03-22T17:12:07Z

featuretools/feature_base/feature_base.py

+                                                      primitive=primitive)
+
+    def copy(self):
+        return GroupByTransformFeature(self.base_features[:-1],


add comment about excluding groupby

kmax12 · 2019-03-22T17:12:12Z

featuretools/feature_base/feature_base.py

+                                       self.groupby)
+
+    def generate_name(self):
+        base_names = [bf.get_name() for bf in self.base_features[:-1]]


add comment about excluding groupby

…ng all base features

kmax12

Looks good to me

rwedge added 13 commits February 25, 2019 11:02

add GroupByTransformPrimitive

87194f2

update cumulative primitives to inherit from GBTP

862590c

add GroupByTransformFeature

6948485

add GBTF to uses_full_entity logic

9f0a2ff

add groupby handler to pandas backend

55b6b16

move cumulative tests to new test_groupby_primitives file

6398a78

Merge branch 'master' into grouby-transform-feature

3748402

remove GroupByTransformPrimitive

09b84fc

rework groupby logic

9254060

update test cases for cumulative primitives

e3a77d3

handled 'nan' group of groupby

bde94de

redo groupby_feature.get_name

8dc9b62

linting

f89167d

rwedge added 7 commits March 6, 2019 13:07

make GBTF a sublcass of TransformFeature

f17e6e6

change groupby restriction from Id to Discrete

607ccb2

test categorical direct feature as groupby in GBTF

927e62d

test GBTF.copy

67f1fcf

test groupby with empty data

fe5400f

test uses_calc_time with GBTF

d8db338

linting

e55fc51

rwedge requested a review from kmax12 March 7, 2019 18:49

kmax12 suggested changes Mar 14, 2019

View reviewed changes

rwedge added 6 commits March 15, 2019 12:37

Merge branch 'master' into grouby-transform-feature

a866d73

rename test file

9dd526c

have feature tree separate features by groupby

06dac4c

check exact class instead of allowing subclasses in feature handlers

6a9b8d5

add groupby to base_features earlier

d70dc6d

Merge branch 'master' into grouby-transform-feature

84bc534

kmax12 suggested changes Mar 22, 2019

View reviewed changes

rwedge added 6 commits March 22, 2019 15:27

change implementation of cumulative count

95d7f3e

add comments about time where we exclude the groupby feature when usi…

03f1a41

…ng all base features

reassign index when primitive function returns series

a3ee88f

let pandas fill in null values for instances without a group

d9af1bf

Merge branch 'master' into grouby-transform-feature

403ad54

linting

2a41be1

rwedge mentioned this pull request Mar 22, 2019

Groupby dfs #472

Merged

kmax12 added 2 commits March 25, 2019 10:34

Update cum_transform_feature.py

10e0ecc

Update cum_transform_feature.py

49aa411

kmax12 approved these changes Mar 25, 2019

View reviewed changes

rwedge merged commit a45d285 into master Mar 25, 2019

rwedge mentioned this pull request Mar 29, 2019

v0.7.0 #477

Merged

rwedge deleted the grouby-transform-feature branch February 19, 2021 21:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Grouby transform feature #455

Grouby transform feature #455

rwedge commented Mar 6, 2019

codecov bot commented Mar 6, 2019 •

edited

Loading

kmax12 Mar 6, 2019

kmax12 Mar 14, 2019

kmax12 Mar 14, 2019

kmax12 Mar 14, 2019

kmax12 Mar 14, 2019

kmax12 Mar 14, 2019

kmax12 Mar 22, 2019

kmax12 Mar 22, 2019

kmax12 Mar 22, 2019

kmax12 Mar 22, 2019

kmax12 Mar 22, 2019

kmax12 Mar 22, 2019

kmax12 left a comment

Grouby transform feature #455

Grouby transform feature #455

Conversation

rwedge commented Mar 6, 2019

codecov bot commented Mar 6, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kmax12 left a comment

Choose a reason for hiding this comment

codecov bot commented Mar 6, 2019 •

edited

Loading