Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Allow adjust=False when times is provided #59142

Merged
merged 9 commits into from
Jul 8, 2024

Conversation

tserrao
Copy link
Contributor

@tserrao tserrao commented Jun 28, 2024

This change enables EWMA to be calculated recursively ("infinite history", i.e. adjust=False) for irregular-interval time series. Previously new_wt was held constant in the ewm run loop, but now new_wt is updated each step through the series under the narrow condition that adjust=False and com=1.

There may be a more elegant expression that will generalize to com values different than one, but using 1 - old_wt works nicely if halflife is the only provided decay parameter since we are guaranteed com=1. The additional parameter restrictions to force this condition seem acceptable since there is already some restriction on decay parameters.

Exactly one of com, span, halflife, or alpha must be provided if times is not provided. If times is provided, halflife and one of com, span or alpha may be provided.

FWIW I confirmed that results match polars and @azmyrajab's nice test function:

ema_no_adjust = ema_test(vals, dt_seconds, half_life=half_life_seconds)[-1]
ema_no_adjust_pl = ema_polars(df["val"].rename_axis(index="ts").rename("val"), half_life,
                              by="ts").select("val")[-1].item()

print(ema_no_adjust)
print(ema_no_adjust_pl)
print(df.ewm(halflife=half_life, times=id, adjust=False).mean().iloc[-1]['val'])

0.2062994740159002
0.2062994740159002
0.2062994740159002

@tserrao tserrao requested a review from WillAyd as a code owner June 28, 2024 19:51
@tserrao tserrao changed the title Allow adjust=False when times is provided ENH: Allow adjust=False when times is provided Jul 2, 2024
@tserrao
Copy link
Contributor Author

tserrao commented Jul 2, 2024

pre-commit is finally green but a lot of the actions are failing with the same message:

Error: Failed to CreateArtifact: Received non-retryable error: Failed request: (409) Conflict: an artifact with this name already exists on the workflow run

Seeing lots of PRs get these errors. Maybe a recent GHA setup change is breaking some tests?

@tserrao
Copy link
Contributor Author

tserrao commented Jul 8, 2024

CI all green :)

@mroeschke mroeschke added the Window rolling, ewma, expanding label Jul 8, 2024
@mroeschke mroeschke added this to the 3.0 milestone Jul 8, 2024
@mroeschke mroeschke merged commit bd405e8 into pandas-dev:main Jul 8, 2024
49 checks passed
@mroeschke
Copy link
Member

Thanks @tserrao

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Window rolling, ewma, expanding
Projects
None yet
Development

Successfully merging this pull request may close these issues.

EWMA weighted by time with adjust=True is flawed, and adjust=False is not supported
2 participants