Add Rolling Transform primitives #1770

tamargrey · 2021-11-04T19:28:32Z

Adds five rolling transform primitives that all takeinteger window_length, gap and min_periods parameters.
closes #1756

tamargrey · 2021-11-04T20:24:21Z

featuretools/primitives/standard/rolling_transform_primitive.py

+    return_type = ColumnSchema(logical_type=Double, semantic_tags={'numeric'})
+    compatibility = [Library.PANDAS, Library.DASK, Library.KOALAS]
+
+    def __init__(self, window_length=3, gap=0, min_periods=1):


Previously the default window-length was 1d, which makes sense and will be a valid window with multiple instances present if the data's sampling frequency is less than 1 day. At this point in the implementation, though, we do not have access to date offsets, and setting the window length to 1 doesn't make much sense, as it would never end up with more than one instance present in the window.

I think I might prefer using window_length=7 or something that has some time-related meaning. But chose 3 for the moment just to show the arbitrary nature of having to set a default for this parameter.

Also, when we implement #1757, we can go back to setting the default to be 1d, but for now, I don't think there's a way to ensure this won't be a breaking change when that's implemented.

tamargrey · 2021-11-04T21:10:39Z

featuretools/primitives/utils.py

+    """
+    # Workaround for pandas' fixed but unreleased bug: https://github.com/pandas-dev/pandas/issues/43016
+    # Can remove when upgraded to pandas 1.4.0
+    if str(series.dtype) == 'Int64':


Created an issue to remove this eventually #1771

codecov · 2021-11-04T22:06:55Z

Codecov Report

Merging #1770 (1389e2f) into main (98988da) will increase coverage by 0.01%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##             main    #1770      +/-   ##
==========================================
+ Coverage   98.70%   98.72%   +0.01%     
==========================================
  Files         139      141       +2     
  Lines       15419    15644     +225     
==========================================
+ Hits        15220    15445     +225     
  Misses        199      199

Impacted Files	Coverage Δ
featuretools/primitives/standard/api.py	`100.00% <100.00%> (ø)`
...primitives/standard/rolling_transform_primitive.py	`100.00% <100.00%> (ø)`
featuretools/primitives/utils.py	`99.24% <100.00%> (+0.04%)`	⬆️
featuretools/tests/conftest.py	`100.00% <100.00%> (ø)`
...ools/tests/primitive_tests/test_primitive_utils.py	`100.00% <100.00%> (ø)`
...ls/tests/primitive_tests/test_rolling_primitive.py	`100.00% <100.00%> (ø)`
...ols/tests/synthesis/test_deep_feature_synthesis.py	`99.33% <100.00%> (+0.01%)`	⬆️
featuretools/utils/plot_utils.py	`92.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 98988da...1389e2f. Read the comment docs.

freddyaboulton

@tamargrey The behavior of these primitives makes sense to me! The only thing that stood out to me was the difference in parameter definitions between EvalML and Featuretools that we chatted about. When we get to datestrings we will need your help in transforming the EvalML definitions to featuretools, but we agreed the "add 1" operation is possible.

Don't think EvalML will ever set min_periods to be different than window_length but that may be useful for other kinds of problems.

If users use RollingSTD with min_periods=1, there will be one extra nan compared to the other primitives because the standard deviation of a single observation is not defined. That makes sense to me.

freddyaboulton · 2021-11-08T16:42:48Z

featuretools/primitives/utils.py

+        window_length (int): The number of rows to be included in each window. For data
+            with a uniform sampling frequency, for example of one day, the window_length will
+            correspond to a period of time, in this case, 7 days for a window_length of 7.
+        gap (int, optional): The number of rows prior to each instance to be skipped before


nit-pick: The use of "skipped" confused me at first. If gap=1, and we're saying that 1 row prior to each instance will be skipped, that makes me think it's doing what gap=2 actually does.

thehomebrewnerd

Overall this is looking decent, but just have a couple things:

For the tests in test_rolling_primitive.py and test_deep_feature_synthesis.py we are only using the pandas version of the test fixture series. Should/could we also test with Dask/Koalas?
Do we have any concerns with changing the init arguments for these primitives from their previous definitions? I think this means any existing applications where these were used as premium primitives will likely break, but I'm not sure that is an issue. Just wanted to mention in case it is important.

featuretools/tests/synthesis/test_deep_feature_synthesis.py

tamargrey · 2021-11-08T21:21:31Z

@thehomebrewnerd

For the tests in test_rolling_primitive.py and test_deep_feature_synthesis.py we are only using the pandas version of the test fixture series. Should/could we also test with Dask/Koalas?

Using all three was originally my plan, which was why I created all three fixtures, but when I started running into various issues (koalas' implementation of rolling sometimes has different behavior than pandas, and dask was having other unexplained failutes), I ended up decided to just allow Pandas, which was how they previously were. I still left the dask and koalas fixtures in place, since _roll_series_with_gap is still tested with them, so in theory we could add the dask and koalas compatibility if it's requested a bit easier.

Happy to remove the extra fixtures if it doesn't make sense to keep them in if the primitives aren't yet compatible. Or, if we're okay with this taking a little longer to get in, I could investigate the dask and koalas issues further to try and get the primitives compatible. @gsheni may have thoughts on whether it'd be worth it.

Do we have any concerns with changing the init arguments for these primitives from their previous definitions? I think this means any existing applications where these were used as premium primitives will likely break, but I'm not sure that is an issue. Just wanted to mention in case it is important.

We'd need to get the date offset implementation in at the same time to have the primitives not break if they are currently in use anywhere. That would be a bigger ask for this PR, that I think we didn't want to have to wait for. Also ran this by @gsheni a while back to confirm this was okay. Can revisit if it's going to be problematic to break them.

gsheni · 2021-11-08T21:23:20Z

@tamargrey is correct. We are punting on the Koalas/Dask implementation of these primitives. Additionally, we are okay with this being a breaking change for the RollingX primitives (compared to premium primitives)

thehomebrewnerd · 2021-11-08T21:43:35Z

I think my vote would be to remove the Dask and Koalas fixtures if we are not supporting those types of inputs now, and we can add them back in when they are needed. Otherwise they could end up hanging around for a long time without really being needed.

Not a big deal to me one way or the other though.

featuretools/tests/primitive_tests/test_primitive_utils.py

rwedge · 2021-11-09T03:21:50Z

featuretools/primitives/standard/rolling_transform_primitive.py

+        window_length (int): The number of rows to be included in each frame. For data
+            with a uniform sampling frequency, for example of one day, the window_length will
+            correspond to a period of time, in this case, 7 days for a window_length of 7.
+        gap (int, optional): The number of rows prior to each instance to be skipped before


How's this sound

The number of rows backward from the target instance to be skipped before the window of usable data begins. Defaults to 0, which will include the target instance in the window

If include_cutoff_time is False does gap=0 function the same as gap=1 would normally?

I like your suggestion. The one thing I might change is just completely getting rid of the word "skipped" based off of @freddyaboulton's comment above. How about

The number of rows backward from the target instance before the window of usable data begins. Defaults to 0, which will include the target instance in the window

I think just talking about "n rows back" better encapsulates the idea that one row back is immediately above and zero rows back is just that instance. Instead of also including the idea of "n rows skipped" which would lead me to think there would be n entire rows that are ignored before starting the window.

To your second point: unless I'm missing something, setting a cutoff time and having include_cutoff_time=False shouldn't have an impact on the values included in any window.

In the example below where we have primitives set with gap=0 and gap=1 and it just impacts the final row.

If I set cutoff_times for each row to be its own time index value (and now we have to have include_cutoff_time=True or else we'll have no data even though windows technically would be before the cutoff time--may need to be a separate issue if we wanted to handle that case specifically), it has no impact at all. (If we also add uses_full_dataframe = True to the primitive because of that separate bug)

Thanks for the explanation, I think I was confusing using the cutoff time to create a window with using the target instance for a window

Ahh fair. Anything I can add to the docstrings to help clarify what happens in these primitives?

I think it's pretty clear

We could put this information in our documentation. Add it as a section or a new page on Time Series. Use it as a chance to explain rolling primitives

Have an issue to do that (#1758 - could also just be adding an entirely new Time series page). Was imagining it could happen once we have a lot of the time series stuff in, but we could always start it now and update it as we go along

@tamargrey your call. Happy with either.

Gonna leave the documentation for later. Should be a bit easier to create a more cohesive narrative when everything is implemented.

rwedge · 2021-11-11T20:03:20Z

@tamargrey can you pull in the latest changes?

featuretools/tests/conftest.py

rwedge

LGTM assuming test pass

featuretools/primitives/utils.py

featuretools/tests/primitive_tests/test_primitive_utils.py

featuretools/tests/synthesis/test_deep_feature_synthesis.py

featuretools/tests/primitive_tests/test_rolling_primitive.py

thehomebrewnerd

LGTM. Seems like the test coverage CI failure is due to some flaky Dask behavior and nothing in this MR.

rwedge · 2021-11-15T17:31:05Z

featuretools/utils/plot_utils.py

+        try:
+            valid_formats = graphviz.backend.FORMATS
+        except AttributeError:
+            from graphviz.parameters import FORMATS
+            valid_formats = FORMATS


try using graphviz.FORMATS

tamargrey requested review from thehomebrewnerd, rwedge and freddyaboulton November 4, 2021 19:28

auto-assign bot assigned tamargrey Nov 4, 2021

tamargrey force-pushed the add-rolling-primitives branch from f7c3afa to 82056ca Compare November 4, 2021 19:31

gsheni requested review from davesque and jeff-hernandez November 4, 2021 19:45

tamargrey commented Nov 4, 2021

View reviewed changes

tamargrey mentioned this pull request Nov 4, 2021

Remove pandas workaround in _roll_series_with_gap #1771

Closed

tamargrey commented Nov 4, 2021

View reviewed changes

freddyaboulton reviewed Nov 8, 2021

View reviewed changes

thehomebrewnerd reviewed Nov 8, 2021

View reviewed changes

featuretools/tests/synthesis/test_deep_feature_synthesis.py Outdated Show resolved Hide resolved

tamargrey force-pushed the add-rolling-primitives branch from efbd45b to 98afebc Compare November 8, 2021 21:11

rwedge reviewed Nov 9, 2021

View reviewed changes

tamargrey requested review from freddyaboulton, rwedge and thehomebrewnerd November 11, 2021 16:29

tamargrey force-pushed the add-rolling-primitives branch from 1c0c5e4 to 85d5ffb Compare November 11, 2021 20:05

rwedge reviewed Nov 11, 2021

View reviewed changes

featuretools/tests/conftest.py Outdated Show resolved Hide resolved

rwedge previously approved these changes Nov 11, 2021

View reviewed changes

gsheni previously approved these changes Nov 11, 2021

View reviewed changes

Tamar Grey added 3 commits November 11, 2021 17:16

first attempts at window with gap

3e45664

Use just shift+rolling helper

2963fee

start adding tests

bcb15cb

Tamar Grey added 10 commits November 11, 2021 17:16

remove dask compatibility with std

e37b772

remove remaining dask dependencies and fix examples

dbcbff1

fix failink dask rolling test

720231d

remove koalas compatibility

ec505f7

fix coverage

44981ae

Fix test dfs imports

4d9c083

remove dask and koalas rolling fixtures

88d4687

Update gap definition in docstring

48a289a

lint fix

d8af0f0

keep as rolling_series_pd

326de65

tamargrey dismissed stale reviews from gsheni and rwedge via 326de65 November 11, 2021 22:16

tamargrey force-pushed the add-rolling-primitives branch from 3b166ad to 326de65 Compare November 11, 2021 22:16

thehomebrewnerd reviewed Nov 12, 2021

View reviewed changes

Tamar Grey added 4 commits November 12, 2021 13:18

PR Comments

f9130d3

Add test with null values

09bd7f2

confirm actual returned values

8096da0

Add fixture to test all rolling primitives with dfs

076dcb3

tamargrey requested a review from thehomebrewnerd November 12, 2021 19:59

thehomebrewnerd previously approved these changes Nov 12, 2021

View reviewed changes

Update bug comment

37542a6

tamargrey dismissed thehomebrewnerd’s stale review via 37542a6 November 15, 2021 14:39

Fix graphviz broken import

b378d54

rwedge reviewed Nov 15, 2021

View reviewed changes

use graphviz.FORMATS

1389e2f

tamargrey force-pushed the add-rolling-primitives branch from 9e7b07d to 1389e2f Compare November 15, 2021 17:40

gsheni approved these changes Nov 15, 2021

View reviewed changes

tamargrey merged commit e368071 into main Nov 15, 2021

tamargrey deleted the add-rolling-primitives branch November 15, 2021 20:03

tamargrey mentioned this pull request Nov 15, 2021

v1.2.0 #1783

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Rolling Transform primitives #1770

Add Rolling Transform primitives #1770

tamargrey commented Nov 4, 2021

tamargrey Nov 4, 2021

tamargrey Nov 4, 2021

codecov bot commented Nov 4, 2021 •

edited

Loading

freddyaboulton left a comment

freddyaboulton Nov 8, 2021

thehomebrewnerd left a comment

tamargrey commented Nov 8, 2021

gsheni commented Nov 8, 2021

thehomebrewnerd commented Nov 8, 2021 •

edited

Loading

rwedge Nov 9, 2021

tamargrey Nov 9, 2021

tamargrey Nov 9, 2021

rwedge Nov 9, 2021

tamargrey Nov 9, 2021

rwedge Nov 9, 2021

gsheni Nov 9, 2021

tamargrey Nov 9, 2021

gsheni Nov 9, 2021 •

edited

Loading

tamargrey Nov 10, 2021 •

edited

Loading

rwedge commented Nov 11, 2021

rwedge left a comment

thehomebrewnerd left a comment

rwedge Nov 15, 2021

Add Rolling Transform primitives #1770

Add Rolling Transform primitives #1770

Conversation

tamargrey commented Nov 4, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 4, 2021 • edited Loading

Codecov Report

freddyaboulton left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thehomebrewnerd left a comment

Choose a reason for hiding this comment

tamargrey commented Nov 8, 2021

gsheni commented Nov 8, 2021

thehomebrewnerd commented Nov 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gsheni Nov 9, 2021 • edited Loading

Choose a reason for hiding this comment

tamargrey Nov 10, 2021 • edited Loading

Choose a reason for hiding this comment

rwedge commented Nov 11, 2021

rwedge left a comment

Choose a reason for hiding this comment

thehomebrewnerd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Nov 4, 2021 •

edited

Loading

thehomebrewnerd commented Nov 8, 2021 •

edited

Loading

gsheni Nov 9, 2021 •

edited

Loading

tamargrey Nov 10, 2021 •

edited

Loading