Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: rolling.median/quantile does not work with (some) BaseIndexer subclasses #37153

Closed
3 tasks done
mroeschke opened this issue Oct 16, 2020 · 1 comment · Fixed by #37166
Closed
3 tasks done

BUG: rolling.median/quantile does not work with (some) BaseIndexer subclasses #37153

mroeschke opened this issue Oct 16, 2020 · 1 comment · Fixed by #37166
Labels
Bug Window rolling, ewma, expanding

Comments

@mroeschke
Copy link
Member

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Using the example from computation.rst.

In [1]: In [70]: use_expanding = [True, False, True, False, True]
   ...:

In [2]: In [72]: df = pd.DataFrame({'values': range(5)})
   ...:

In [3]: In [2]: from pandas.api.indexers import BaseIndexer
   ...:

In [4]: ...: class CustomIndexer(BaseIndexer):
   ...: ...:
   ...: ...:    def get_window_bounds(self, num_values, min_periods, center, closed):
   ...: ...:        start = np.empty(num_values, dtype=np.int64)
   ...: ...:        end = np.empty(num_values, dtype=np.int64)
   ...: ...:        for i in range(num_values):
   ...: ...:            if self.use_expanding[i]:
   ...: ...:                start[i] = 0
   ...: ...:                end[i] = i + 1
   ...: ...:            else:
   ...: ...:                start[i] = i
   ...: ...:                end[i] = i + self.window_size
   ...: ...:        return start, end
   ...:

In [5]: indexer = CustomIndexer(window_size=1, use_expanding=use_expanding)

# Each window
In [7]: for window in df.rolling(indexer):
   ...:     print(window)
   ...:
   values
0       0
   values
1       1
   values
0       0
1       1
2       2
   values
3       3
   values
0       0
1       1
2       2
3       3
4       4

# Should be [0, 1, 1, 3, 2]
In [8]: df.rolling(indexer).median()
Out[8]:
   values
0     0.0
1     1.0
2     1.5
3     NaN
4     3.0

# Should be [0, 1, 1, 3, 2]
In [10]: df.rolling(indexer).quantile(0.5)
Out[10]:
   values
0     0.0
1     1.0
2     1.5
3     NaN
4     3.0

The cython aggregations probably assume some sort of property of the windows that may not hold with the custom indexers

@mroeschke mroeschke added Bug Needs Triage Issue that has not been reviewed by a pandas team member Window rolling, ewma, expanding and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Oct 16, 2020
@mroeschke mroeschke changed the title BUG: rolling.median/quantile does not work with BaseIndexer subclasses BUG: rolling.median/quantile does not work with (some) BaseIndexer subclasses Oct 16, 2020
@phofl
Copy link
Member

phofl commented Oct 16, 2020

Problem are the non monotonic starting points of the windows. Same problem with non monotonic ending points

use_expanding = [True, False, True, False, True]

df = pd.DataFrame({'values': range(5)})
from pandas.api.indexers import BaseIndexer

class CustomIndexer(BaseIndexer):
   
    def get_window_bounds(self, num_values, min_periods, center, closed):
       start = np.empty(num_values, dtype=np.int64)
       end = np.empty(num_values, dtype=np.int64)
       for i in range(num_values):
           if self.use_expanding[i]:
               start[i] = i
               end[i] = max(i -1, 1)
           else:
               start[i] = i
               end[i] = i + self.window_size
       return start, end

indexer = CustomIndexer(window_size=1, use_expanding=use_expanding)

# Each window
for window in df.rolling(indexer):
    print(window)


# print(df)
rolled = df.rolling(indexer)
print(rolled.median())

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Window rolling, ewma, expanding
Projects
None yet
2 participants