Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PERF: Poor numerical stability of rolling_kurt and rolling_skew #6929

Closed
jaimefrio opened this issue Apr 22, 2014 · 3 comments · Fixed by #37557
Closed

PERF: Poor numerical stability of rolling_kurt and rolling_skew #6929

jaimefrio opened this issue Apr 22, 2014 · 3 comments · Fixed by #37557
Labels
Numeric Operations Arithmetic, Comparison, and Logical operations Performance Memory or execution speed performance Window rolling, ewma, expanding
Milestone

Comments

@jaimefrio
Copy link
Contributor

jaimefrio commented Apr 22, 2014

The numerical stability of the current implementation of rolling_kurt and rolling_skew is very poor:

In [14]: x = np.random.rand(10)

In [15]: pd.Series(x).rolling(4).skew()
Out[15]:
0         NaN
1         NaN
2         NaN
3   -0.971467
4    0.596391
5   -1.163135
6   -0.611469
7   -1.624713
8   -0.021832
9    0.754653
dtype: float64

In [16]: pd.Series(x + 5000).rolling(4).skew()
Out[16]:
0         NaN
1         NaN
2         NaN
3   -0.968964
4    0.608522
5   -1.171751
6   -0.619354
7   -1.628942
8   -0.033949
9    0.741304
dtype: float64

In [17]: pd.Series(x).rolling(4).kurt()
Out[17]:
0         NaN
1         NaN
2         NaN
3    2.030185
4   -2.647975
5    1.183004
6   -2.178587
7    2.658058
8   -4.258505
9    0.939264
dtype: float64

In [18]: pd.Series(x + 100).rolling(4).kurt()
Out[18]:
0         NaN
1         NaN
2         NaN
3    2.030230
4   -2.648801
5    1.182382
6   -2.178574
7    2.657934
8   -4.258813
9    0.938661
dtype: float64

This can be solved using an updating algorithm, similar to what is done in #6817 for rolling_var.

@dsm054
Copy link
Contributor

dsm054 commented Jun 26, 2014

I think this can be closed. It was fixed in #6817, and #7297 is open to address the secondary issues that introduced.

@jaimefrio
Copy link
Contributor Author

#6817 fixed it for rolling_var (and just added #7572 that should fix #7297), but rolling_kurt and rolling_skew were unchanged by #6817, and have an even worse numerical stability than rolling_var. @jreback specifically asked me to open this issue as a placeholder, I am hoping to find some time during this summer to figure out the math and get those fixed as well.

@seth-p
Copy link
Contributor

seth-p commented Sep 15, 2014

Note also #8086.

@jreback jreback changed the title PERF: Poor numerical stability of rolling_kurt and rolling_skew PERF: Poor numerical stability of rolling_kurt and rolling_skew Sep 30, 2014
@jreback jreback modified the milestones: 0.16.1, 0.16.0 Mar 5, 2015
@jreback jreback modified the milestones: 0.17.0, 0.16.1 Apr 28, 2015
@jreback jreback modified the milestones: Next Major Release, 0.17.0 Aug 15, 2015
@jreback jreback modified the milestones: 0.17.0, Next Major Release Aug 21, 2015
@jreback jreback modified the milestones: Next Major Release, 0.17.0 Sep 2, 2015
@mroeschke mroeschke added Window rolling, ewma, expanding Performance Memory or execution speed performance and removed Algos Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff Enhancement labels Dec 24, 2019
@jreback jreback modified the milestones: Contributions Welcome, 1.2 Nov 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Numeric Operations Arithmetic, Comparison, and Logical operations Performance Memory or execution speed performance Window rolling, ewma, expanding
Projects
None yet
6 participants