Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: support monotonic_decreasing on rolling #19248

Closed
armona opened this issue Jan 15, 2018 · 1 comment · Fixed by #28297
Closed

ENH: support monotonic_decreasing on rolling #19248

armona opened this issue Jan 15, 2018 · 1 comment · Fixed by #28297
Labels
Enhancement Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timeseries
Milestone

Comments

@armona
Copy link

armona commented Jan 15, 2018

Example:

import pandas as pd

index = [
    pd.Timestamp('20130101 09:00:00'),
    pd.Timestamp('20130101 09:00:02'),
    pd.Timestamp('20130101 09:00:03'),
    pd.Timestamp('20130101 09:00:05'),
    pd.Timestamp('20130101 09:00:06')
]

df = pd.DataFrame({'price': [3, 4, 4, 2, 1]}, index=reversed(index))
df.rolling('2s').min()

Problem description

I'm trying to work with time series data frames in reversed order. It seems pandas is failing due to index must be monotonic, although the index is clearly mononotonic decreasing.

Note: This stackoverflow answer seems like a workaround

The error raised:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-2-77eb98350c02> in <module>()
----> 1 df.rolling('1s').mean()

~/.local/share/virtualenvs/src-l39fhq8N/lib/python3.6/site-packages/pandas/core/generic.py in rolling(self, window, min_periods, freq, center, win_type, on, axis, closed)
   7065                                    min_periods=min_periods, freq=freq,
   7066                                    center=center, win_type=win_type,
-> 7067                                    on=on, axis=axis, closed=closed)
   7068 
   7069         cls.rolling = rolling

~/.local/share/virtualenvs/src-l39fhq8N/lib/python3.6/site-packages/pandas/core/window.py in rolling(obj, win_type, **kwds)
   2067         return Window(obj, win_type=win_type, **kwds)
   2068 
-> 2069     return Rolling(obj, **kwds)
   2070 
   2071 

~/.local/share/virtualenvs/src-l39fhq8N/lib/python3.6/site-packages/pandas/core/window.py in __init__(self, obj, window, min_periods, freq, center, win_type, axis, on, closed, **kwargs)
     84         self.win_freq = None
     85         self.axis = obj._get_axis_number(axis) if axis is not None else None
---> 86         self.validate()
     87 
     88     @property

~/.local/share/virtualenvs/src-l39fhq8N/lib/python3.6/site-packages/pandas/core/window.py in validate(self)
   1102                                          timedelta))):
   1103 
-> 1104             self._validate_monotonic()
   1105             freq = self._validate_freq()
   1106 

~/.local/share/virtualenvs/src-l39fhq8N/lib/python3.6/site-packages/pandas/core/window.py in _validate_monotonic(self)
   1134             formatted = self.on or 'index'
   1135             raise ValueError("{0} must be "
-> 1136                              "monotonic".format(formatted))
   1137 
   1138     def _validate_freq(self):

ValueError: index must be monotonic

Expected Output

The expected output should be that the min should be calculated from the 2s previous samples

                     price
2013-01-01 09:00:00    3.0
2013-01-01 09:00:02    4.0
2013-01-01 09:00:03    2.0
2013-01-01 09:00:05    1.0
2013-01-01 09:00:06    1.0

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.10.0-42-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: 3.2.5
pip: 9.0.1
setuptools: 38.2.4
Cython: None
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: None
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: 1.1.15
pymysql: None
psycopg2: 2.7.3.2 (dt dec pq3 ext lo64)
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
@jreback
Copy link
Contributor

jreback commented Jan 15, 2018

I suppose this could be supported internally, virtually all time based operations required monotonicity (increasing), so this is correct.

If someone wants to implement for monotonic_decreasing then would be ok.

@jreback jreback added Enhancement Timeseries Reshaping Concat, Merge/Join, Stack/Unstack, Explode Difficulty Intermediate labels Jan 15, 2018
@jreback jreback added this to the Next Major Release milestone Jan 15, 2018
@jreback jreback changed the title Decreasing index raises not monotonic on rolling ENH: support monotonic_decreasing on rolling Jan 15, 2018
@jreback jreback modified the milestones: Contributions Welcome, 1.0 Oct 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timeseries
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants