Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: implement __iter__ for window objects #11704

Open
jreback opened this Issue Nov 25, 2015 · 7 comments

Comments

Projects
None yet
4 participants
@jreback
Copy link
Contributor

jreback commented Nov 25, 2015

xref #11603

might be slightly non-trivial as the current impl does NOT use an explict iterator, rather does a sliding window in cython and marginal calcs.

@kawochen

This comment has been minimized.

Copy link
Contributor

kawochen commented Nov 30, 2015

I don't know if this is too ambitious, but for custom rolling aggregation, if you can let users supply a StateVariables object which is initialized using the first window, and supports two methods, evolve() and compute(), then you may be able to retain the efficiency when the evolution of the state variables is governed by a simple set of equations. __iter__ becomes a special case of this where the StateVariables contains the whole window, and evolve moves the window forward/replaces the old window with the new window, and compute just returns the data structure inside.

@jhamman jhamman referenced this issue Dec 2, 2015

Merged

Feature/rolling #668

4 of 4 tasks complete

@jreback jreback modified the milestones: Next Major Release, 0.18.0 Jan 24, 2016

@selik

This comment has been minimized.

Copy link
Contributor

selik commented Oct 31, 2017

Currently, iterating over a pandas.core.window.Rolling object will produce bizarre behavior. For DataFrames as it looks for columns named 0, 1, 2, etc. until it hits a KeyError. For Series it raises AttributeError as it looks for a "columns" attribute.

Until the __iter__ is implemented correctly, perhaps there should be a temporary measure of simply making it non-iterable by raising NotImplementedError.

@jreback

This comment has been minimized.

Copy link
Contributor Author

jreback commented Nov 1, 2017

@selik if you want to submit a PR to raise NotImplementedError would be great!

selik added a commit to selik/pandas that referenced this issue May 9, 2018

ENH: Raise useful error when iterating a Window
Until Issue pandas-dev#11704 is completed, raise a NotImplementedError to provide
a more clear error message when attempting to iterate over a Rolling
or Expanding window.

TomAugspurger added a commit that referenced this issue May 12, 2018

ENH: Raise useful error when iterating a Window (#20996)
* ENH: Raise useful error when iterating a Window

Until Issue #11704 is completed, raise a NotImplementedError to provide
a more clear error message when attempting to iterate over a Rolling
or Expanding window.

topper-123 pushed a commit to topper-123/pandas that referenced this issue May 13, 2018

ENH: Raise useful error when iterating a Window (pandas-dev#20996)
* ENH: Raise useful error when iterating a Window

Until Issue pandas-dev#11704 is completed, raise a NotImplementedError to provide
a more clear error message when attempting to iterate over a Rolling
or Expanding window.

topper-123 pushed a commit to topper-123/pandas that referenced this issue May 13, 2018

ENH: Raise useful error when iterating a Window (pandas-dev#20996)
* ENH: Raise useful error when iterating a Window

Until Issue pandas-dev#11704 is completed, raise a NotImplementedError to provide
a more clear error message when attempting to iterate over a Rolling
or Expanding window.
@antoinecomp

This comment has been minimized.

Copy link

antoinecomp commented Dec 17, 2018

Okay, how to cope with this problem then ?

I have an object, EWM [com=9.5,min_periods=0,adjust=True,ignore_na=False,axis=0] that I want to add to a dataframe: predictions_df_list['ewma'] which raised this issue

@selik

This comment has been minimized.

Copy link
Contributor

selik commented Dec 18, 2018

@antoinecomp Could you share a bit more of your code and the error message? The traceback will help. Also, what kind of object is EWM?

@antoinecomp

This comment has been minimized.

Copy link

antoinecomp commented Jan 3, 2019

@selik Sorry for answering so late. This is the code I used:

predictions_df['ewma'] = pd.DataFrame.ewm(predictions_df["prices"], span=60)#, freq="D")

And this is the error message:

NotImplementedError                       Traceback (most recent call last)
<ipython-input-153-2a1b97ea7cf7> in <module>
----> 1 predictions_df['ewma'] = pd.DataFrame.ewm(predictions_df["prices"], span=60)#, freq="D")

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in __setitem__(self, key, value)
   3117         else:
   3118             # set column
-> 3119             self._set_item(key, value)
   3120 
   3121     def _setitem_slice(self, key, value):

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _set_item(self, key, value)
   3192 
   3193         self._ensure_valid_index(value)
-> 3194         value = self._sanitize_column(key, value)
   3195         NDFrame._set_item(self, key, value)
   3196 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _sanitize_column(self, key, value, broadcast)
   3385             value = _sanitize_index(value, self.index, copy=False)
   3386 
-> 3387         elif isinstance(value, Index) or is_sequence(value):
   3388             from pandas.core.series import _sanitize_index
   3389 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\dtypes\inference.py in is_sequence(obj)
    470 
    471     try:
--> 472         iter(obj)  # Can iterate over it.
    473         len(obj)   # Has a length associated with it.
    474         return not isinstance(obj, string_and_binary_types)

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\window.py in __iter__(self)
    184     def __iter__(self):
    185         url = 'https://github.com/pandas-dev/pandas/issues/11704'
--> 186         raise NotImplementedError('See issue #11704 {url}'.format(url=url))
    187 
    188     def _get_index(self, index=None):

NotImplementedError: See issue #11704 https://github.com/pandas-dev/pandas/issues/11704
@selik

This comment has been minimized.

Copy link
Contributor

selik commented Jan 9, 2019

It looks like is_sequence's test of iter(obj) catching TypeError and AttributeError is inconsistent with __iter__ raising NotImplementedError.

except (TypeError, AttributeError):

Reverting to the old behavior won't fix this, because iter(obj) would cause a KeyError, which is also unexpected.

@topper-123 @evectant I'm not familiar with the ewm method. Is a NotImplementedError reasonable here?

@jreback Should is_sequence catch NotImplementedError in addition to TypeError and AttributeError when trying iter(obj)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.