Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable range-partitioning implementation for groupby().rolling() by default #6942

Closed
dchigarev opened this issue Feb 19, 2024 · 0 comments · Fixed by #6943
Closed

Enable range-partitioning implementation for groupby().rolling() by default #6942

dchigarev opened this issue Feb 19, 2024 · 0 comments · Fixed by #6943
Assignees
Labels
new feature/request 💬 Requests and pull requests for new features P1 Important tasks that we should complete soon

Comments

@dchigarev
Copy link
Collaborator

Doing measurements, I found out that using range-partitioning implementation for groupby().rolling() is always faster than full-axis implementation, so it's worth enabling range-partitioning impl by default for this case (the same as we did for groupby().apply() #6804)

16 cores

image

44 cores

image

script to measure
import modin.pandas as pd
import pandas

import numpy as np

import modin.config as cfg
from timeit import default_timer as timer
from modin.utils import execute

NROWS = [10_000, 30_000, 50_000, 100_000, 500_000, 1_000_000, 5_000_000]
NCOLS = [10, 40]
NGROUPS = [10, 10_000]
USE_RANGE_PART = [True, False]
NITERS = 3
WINDOW = 5
method = "mean"

total_iters = len(NROWS) * len(NCOLS) * len(NGROUPS) * len(USE_RANGE_PART) * NITERS
its = 0

res_s = pandas.DataFrame(index=pandas.Index(NROWS, name="num_rows"), columns=pandas.MultiIndex.from_product([NCOLS, NGROUPS, USE_RANGE_PART], names=["num_cols", "num_groups", "use range-part"]))

for nrows in NROWS:
    for ncols in NCOLS:
        for ngroups in NGROUPS:
            data = {
                "key": np.tile(np.arange(ngroups), nrows // ngroups),
                **{f"data_col{i}": np.random.randint(0, 1_000_000, size=nrows) for i in range(ncols - 1)}
            }

            for use_rpart in USE_RANGE_PART:
                cfg.RangePartitioningGroupby.put(use_rpart)
                md_df = pd.DataFrame(data)
                execute(md_df)
                times = []
                for i in range(NITERS):
                    print(f"{round((its / total_iters) * 100, 2)}%")
                    t1 = timer()
                    res = getattr(md_df.groupby("key").rolling(WINDOW), method)()
                    execute(res)
                    md_time = timer() - t1
                    times.append(md_time)
                    its += 1
                tm = np.median(times)
                res_s.loc[nrows, (ncols, ngroups, use_rpart)] = tm

                res_s.to_excel("rolling.xlsx")
@dchigarev dchigarev added new feature/request 💬 Requests and pull requests for new features P1 Important tasks that we should complete soon labels Feb 19, 2024
@dchigarev dchigarev self-assigned this Feb 19, 2024
dchigarev added a commit to dchigarev/modin that referenced this issue Feb 19, 2024
…).rolling()' by default

Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
YarShev pushed a commit that referenced this issue Feb 19, 2024
…by default (#6943)

Signed-off-by: Dmitry Chigarev <dmitry.chigarev@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature/request 💬 Requests and pull requests for new features P1 Important tasks that we should complete soon
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant