Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.apply() for core.window.Window (feature request) #19286

Open
geekoverdose opened this issue Jan 17, 2018 · 4 comments
Open

.apply() for core.window.Window (feature request) #19286

geekoverdose opened this issue Jan 17, 2018 · 4 comments
Labels
Apply Apply, Aggregate, Transform Enhancement Window rolling, ewma, expanding

Comments

@geekoverdose
Copy link

geekoverdose commented Jan 17, 2018

Code Sample, a copy-pastable example if possible

import pandas as pd
import numpy as np

# create ordinary example data frame
myDf = pd.DataFrame(data={'myCol': pd.Series(np.random.randn(1000)).cumsum()})

# rolling + apply example 1: this works as expected because win_type=None returns a core.window.Rolling which provides apply functionality
myDf2 = myDf.rolling(window=50, center=True, win_type=None).apply(func=lambda x2: np.percentile(a=x2, q=50, interpolation='nearest'))
# rolling + apply example 2: this does not work because as soon as users change win_type to be anything else than None a core.window.Window is returned instead - which does *not* provide apply functionality yet
myDf3 = myDf.rolling(window=50, center=True, win_type='hamming').apply(func=lambda x2: np.percentile(a=x2, q=50, interpolation='nearest'))

Problem description

core.window.Window does not yet provide apply functionality like core.window.Rolling does, hence functions other than the readily provided ones (like sum(), mean(), etc) cannot be applied to it.

(I'm not to deep into the details, but IMHO returning different object types for just changing the window type from rectangular to some other window form seems rather unintuitive altogether. What was the reason for it being designed this way? I couldn't quickly come up with any. It leads to e.g. example 1 from above working fine as long as the win_type parameter is untouched - but breaks as soon as win_type is changed to a different-than-rectangular window, which does not semantically seem to make sense.)

Expected Output

core.window.Window should provide apply functionality, probably via .apply() like core.window.Rolling, so that example 2 from above works fine too.

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

INSTALLED VERSIONS

commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.13.0-26-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: 3.2.1
pip: 9.0.1
setuptools: 36.5.0.post20170921
Cython: 0.26.1
numpy: 1.13.3
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.1.0
sphinx: 1.6.3
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.1.0
openpyxl: 2.4.8
xlrd: 1.1.0
xlwt: 1.3.0
xlsxwriter: 1.0.2
lxml: 4.1.0
bs4: 4.6.0
html5lib: 0.999999999
sqlalchemy: 1.1.13
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@chris-b1
Copy link
Contributor

The direct answer to your question is here #11603 (comment) - as you may or may not be aware, there used to be set a of top level pd.rolling_* functions that were refactored into the current OO interface. pd.rolling_window got stuffed into .rolling, not really out of design, but just as a place to put it.

As you note, it's sort of half-implemented - only supports sum and mean. I wouldn't be opposed to having it more fully implemented under a .window accessor with .apply, etc, but not sure how much it actually gets used (never something I've used myself).

@geekoverdose
Copy link
Author

Thanks, acknowledged. .apply() would be used e.g. for signal processing where different window forms are required and where people want to avoid transforming pandas data into the underlying scipy APIs just for doing that.
Btw: core.window.rolling() distinguishing win_type=None and win_type='boxcar' and returning different object types based on this seems to be caused by same underlying inconsistency. Both have the same semantic meaning and AFAIK do the same thing in the background (see #17893 (comment)).

@jreback
Copy link
Contributor

jreback commented Jan 18, 2018

@pirius this is very simple. The implementation of a weighted window only accepts mean or sum. It should prob share code with the remainder of rolling but it doesn't. a pull request for enhancements would be accepted.

@jreback
Copy link
Contributor

jreback commented Jan 18, 2018

FYI, only for boxcar does this work the same. That is become the windowing pattern is unweighted.

@jreback jreback added Enhancement Reshaping Concat, Merge/Join, Stack/Unstack, Explode Difficulty Intermediate labels Jan 18, 2018
@jreback jreback added this to the Next Major Release milestone Jan 18, 2018
@geekoverdose geekoverdose changed the title .appy() for core.window.Window (feature request) .apply() for core.window.Window (feature request) Feb 5, 2018
@WillAyd WillAyd added the Window rolling, ewma, expanding label Oct 5, 2018
@mroeschke mroeschke added Apply Apply, Aggregate, Transform and removed Reshaping Concat, Merge/Join, Stack/Unstack, Explode labels May 8, 2020
@mroeschke mroeschke removed this from the Contributions Welcome milestone Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Apply Apply, Aggregate, Transform Enhancement Window rolling, ewma, expanding
Projects
None yet
Development

No branches or pull requests

6 participants