Add rolling cov

[ivarsfg]

The issue is that since cov output is 2D (#4053 (comment)) chunk overlap is calculated incorrectly. In cov tests the output was only differing in infs and nans. I've marked this as WIP because I'm not completely sure if this is a good approach.

• Tests added / passed
• Passes black dask / flake8 dask

closes issue #4053

[jcrist]

Thanks for the PR. Sinceinf and nan aren't equivalent - instead of changing the tests to ignore this, we should figure out how to make the cov method return the correct results where possible. Do you know why inf is appearing in these cases?

[ivarsfg]

It looks like there is some numerical imprecision when pandas calculates rolling mean (as part of calculating cov) and the exact result depends on the array length. So the difference in infs and nans is from splitting the dataframe. This can be reproduced with no dask involvement:

df = pd.DataFrame(
    {
        "a": np.array([0.01, 0.1, 0.01, 0.1]),
        "b": np.array([1.0, 1.0, 1.0, 1.0])
    }
)
df_half = df.iloc[2:]
print(df.rolling(1).cov())
print(df_half.rolling(1).cov())

       a   b
0 a  NaN NaN
  b  NaN NaN
1 a  NaN NaN
  b  NaN NaN
2 a -inf NaN
  b  NaN NaN
3 a  NaN NaN
  b  NaN NaN
      a   b
2 a NaN NaN
  b NaN NaN
3 a NaN NaN
  b NaN NaN

df = pd.DataFrame({'a': np.array([0.01, 0.1, 0.01, 0.1])})
means = df.rolling(1).mean()['a']
print(means[0], means[2])

0.01 0.009999999999999995

[ivarsfg]

Should we just use a separate dataset for cov() tests?

[jcrist]

I tried this out locally on a few random datasets and everything seems great. Using a different dataset or tweaking the tests appropriately seems fine to me.

[ivarsfg]

Tests were failing on certain pandas versions because of Panel deprecation warning (deprecated in 0.20.0, removed in 0.25.0).

[TomAugspurger]

Does this need to be conditional on the pandas version? I don't see any harm in always applying the warning.

[ivarsfg]

I was thinking that after >= 0.25.0 will be used for all CI test envs this whole thing (filter_panel_warning) could be removed, right now dask-35.yml is failing without this. By "always applying the warning" did you mean - let CI fail?

[TomAugspurger]

Sorry I meant "always apply the warnings filter". Not a big deal though.

[TomAugspurger]

Is this likely to cause new issues with empty partitions? e.g. the one from

In [33]: df = pd.DataFrame({"A": range(12), "B": [True] * 3 + [False] * 3 + [True] * 6})

In [34]: ddf = dd.from_pandas(df, 4)

ddf[df.B].get_partition(1).compute()

We may already have issues with empty partitions, in which case don't worry about it.

[ivarsfg]

I didn't find a test case to trigger this but I'm thinking a check here doesn't add much complexity and could prevent confusion later.

[TomAugspurger]

Restarted the failing CI (http error). Things look good here I think.

[jcrist]

Thanks @ivarsfg!