Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: implement __iter__ for window objects #11704

Closed
jreback opened this issue Nov 25, 2015 · 16 comments
Closed

ENH: implement __iter__ for window objects #11704

jreback opened this issue Nov 25, 2015 · 16 comments

Comments

@jreback
Copy link
Contributor

@jreback jreback commented Nov 25, 2015

xref #11603

might be slightly non-trivial as the current impl does NOT use an explict iterator, rather does a sliding window in cython and marginal calcs.

@kawochen
Copy link
Contributor

@kawochen kawochen commented Nov 30, 2015

I don't know if this is too ambitious, but for custom rolling aggregation, if you can let users supply a StateVariables object which is initialized using the first window, and supports two methods, evolve() and compute(), then you may be able to retain the efficiency when the evolution of the state variables is governed by a simple set of equations. __iter__ becomes a special case of this where the StateVariables contains the whole window, and evolve moves the window forward/replaces the old window with the new window, and compute just returns the data structure inside.

@jreback jreback added this to the Next Major Release milestone Jan 24, 2016
@jreback jreback removed this from the 0.18.0 milestone Jan 24, 2016
@selik
Copy link
Contributor

@selik selik commented Oct 31, 2017

Currently, iterating over a pandas.core.window.Rolling object will produce bizarre behavior. For DataFrames as it looks for columns named 0, 1, 2, etc. until it hits a KeyError. For Series it raises AttributeError as it looks for a "columns" attribute.

Until the __iter__ is implemented correctly, perhaps there should be a temporary measure of simply making it non-iterable by raising NotImplementedError.

@jreback
Copy link
Contributor Author

@jreback jreback commented Nov 1, 2017

@selik if you want to submit a PR to raise NotImplementedError would be great!

selik added a commit to selik/pandas that referenced this issue May 9, 2018
Until Issue pandas-dev#11704 is completed, raise a NotImplementedError to provide
a more clear error message when attempting to iterate over a Rolling
or Expanding window.
TomAugspurger added a commit that referenced this issue May 12, 2018
* ENH: Raise useful error when iterating a Window

Until Issue #11704 is completed, raise a NotImplementedError to provide
a more clear error message when attempting to iterate over a Rolling
or Expanding window.
topper-123 pushed a commit to topper-123/pandas that referenced this issue May 13, 2018
* ENH: Raise useful error when iterating a Window

Until Issue pandas-dev#11704 is completed, raise a NotImplementedError to provide
a more clear error message when attempting to iterate over a Rolling
or Expanding window.
topper-123 pushed a commit to topper-123/pandas that referenced this issue May 13, 2018
* ENH: Raise useful error when iterating a Window

Until Issue pandas-dev#11704 is completed, raise a NotImplementedError to provide
a more clear error message when attempting to iterate over a Rolling
or Expanding window.
@antoinecomp
Copy link

@antoinecomp antoinecomp commented Dec 17, 2018

Okay, how to cope with this problem then ?

I have an object, EWM [com=9.5,min_periods=0,adjust=True,ignore_na=False,axis=0] that I want to add to a dataframe: predictions_df_list['ewma'] which raised this issue

@selik
Copy link
Contributor

@selik selik commented Dec 18, 2018

@antoinecomp Could you share a bit more of your code and the error message? The traceback will help. Also, what kind of object is EWM?

@antoinecomp
Copy link

@antoinecomp antoinecomp commented Jan 3, 2019

@selik Sorry for answering so late. This is the code I used:

predictions_df['ewma'] = pd.DataFrame.ewm(predictions_df["prices"], span=60)#, freq="D")

And this is the error message:

NotImplementedError                       Traceback (most recent call last)
<ipython-input-153-2a1b97ea7cf7> in <module>
----> 1 predictions_df['ewma'] = pd.DataFrame.ewm(predictions_df["prices"], span=60)#, freq="D")

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in __setitem__(self, key, value)
   3117         else:
   3118             # set column
-> 3119             self._set_item(key, value)
   3120 
   3121     def _setitem_slice(self, key, value):

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _set_item(self, key, value)
   3192 
   3193         self._ensure_valid_index(value)
-> 3194         value = self._sanitize_column(key, value)
   3195         NDFrame._set_item(self, key, value)
   3196 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\frame.py in _sanitize_column(self, key, value, broadcast)
   3385             value = _sanitize_index(value, self.index, copy=False)
   3386 
-> 3387         elif isinstance(value, Index) or is_sequence(value):
   3388             from pandas.core.series import _sanitize_index
   3389 

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\dtypes\inference.py in is_sequence(obj)
    470 
    471     try:
--> 472         iter(obj)  # Can iterate over it.
    473         len(obj)   # Has a length associated with it.
    474         return not isinstance(obj, string_and_binary_types)

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\window.py in __iter__(self)
    184     def __iter__(self):
    185         url = 'https://github.com/pandas-dev/pandas/issues/11704'
--> 186         raise NotImplementedError('See issue #11704 {url}'.format(url=url))
    187 
    188     def _get_index(self, index=None):

NotImplementedError: See issue #11704 https://github.com/pandas-dev/pandas/issues/11704

@selik
Copy link
Contributor

@selik selik commented Jan 9, 2019

It looks like is_sequence's test of iter(obj) catching TypeError and AttributeError is inconsistent with __iter__ raising NotImplementedError.

except (TypeError, AttributeError):

Reverting to the old behavior won't fix this, because iter(obj) would cause a KeyError, which is also unexpected.

@topper-123 @evectant I'm not familiar with the ewm method. Is a NotImplementedError reasonable here?

@jreback Should is_sequence catch NotImplementedError in addition to TypeError and AttributeError when trying iter(obj)?

@jreback jreback removed this from the Contributions Welcome milestone Jul 20, 2019
@jreback jreback added this to the 1.0 milestone Jul 20, 2019
@TexFly
Copy link

@TexFly TexFly commented Sep 16, 2019

I'm getting the same error in core\windows.py
raise NotImplementedError("See issue #11704 {url}".format(url=url))

Any solution or idea how to fix or troubleshoot?
Thanks!

@selik
Copy link
Contributor

@selik selik commented Sep 16, 2019

@TexFly Are you trying to iterate over a window object? If so, it's as the error says: not implemented.

@mjwillson
Copy link

@mjwillson mjwillson commented Oct 30, 2019

It's easy to run into this error if you (forgivably, IMO) assume that series.ewm(...) will return another Series, and you try to iterate over it.

In fact it's returning a pandas.core.window.EWM object, and you have to call .mean() on it to get the actual exponentially weighted moving average.

@yasersakkaf
Copy link

@yasersakkaf yasersakkaf commented Dec 27, 2019

@antoinecomp Did you solve the error. I am getting the same.

@TomAugspurger
Copy link
Contributor

@TomAugspurger TomAugspurger commented Dec 30, 2019

There's a stalled branch at #27399 if anyone wants to pick this up.

@TomAugspurger TomAugspurger removed this from the 1.0 milestone Dec 30, 2019
@TomAugspurger TomAugspurger added this to the Contributions Welcome milestone Dec 30, 2019
@fallenreaper
Copy link

@fallenreaper fallenreaper commented Feb 14, 2020

I have a Dataframe i was planning to use with ewm, but it seems that when I say:

dt["myCol"].ewm(span=21, ignore_na=true)

it will tell me it is a NotImplementedError and points to come to this Github error. Is there a new way we should be doing this, in either 0.25.0 or 1.0.1?

@harimohanshrm7
Copy link

@harimohanshrm7 harimohanshrm7 commented Feb 19, 2020

@antoinecomp
this error will go away if you use an aggregator function at the end of statement.
e.g.
predictions_df['ewma'] = pd.DataFrame.ewm(predictions_df["prices"], span=60).mean()

anthonytw added a commit to anthonytw/pandas that referenced this issue May 13, 2020
@jreback jreback removed this from the Contributions Welcome milestone May 17, 2020
@jreback jreback added this to the 1.1 milestone May 17, 2020
@banderlog
Copy link

@banderlog banderlog commented Jun 9, 2020

In which pandas version it will appear?

The pandas-1.04 still shows NotImplementedError

@TomAugspurger
Copy link
Contributor

@TomAugspurger TomAugspurger commented Jun 9, 2020

1.1 (See the "milestone" on this issue).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.