Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FIX-#6899: Avoid sending lazy categorical proxies to workers #6900

Merged
merged 3 commits into from
Jan 31, 2024
Merged

FIX-#6899: Avoid sending lazy categorical proxies to workers #6900

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6b7fc09
FIX-#6899: Avoid sending lazy categorical proxies to workers
dchigarev Jan 31, 2024
9ac84ef
modify groupby tests
dchigarev Jan 31, 2024
05a9ea4
Merge remote-tracking branch 'origin/master' into issue_6899
dchigarev Jan 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
17 changes: 16 additions & 1 deletion modin/core/dataframe/pandas/dataframe/dataframe.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3852,7 +3852,22 @@
# 2. The second one works slower, but only gathers light pandas.Index objects,
# so there should be less stress on the network.
if add_missing_cats or not IsRayCluster.get():
original_dtypes = self.dtypes if self.has_materialized_dtypes else None
if self.has_materialized_dtypes:
original_dtypes = pandas.Series(
{
# lazy proxies hold a reference to another modin's DataFrame which can be
# a problem during serialization, in this scenario we don't need actual
# categorical values, so a "category" string will be enough
name: (
"category"
if isinstance(dtype, LazyProxyCategoricalDtype)
else dtype
)
for name, dtype in self.dtypes.items()
}
)
else:
original_dtypes = None

Check warning on line 3870 in modin/core/dataframe/pandas/dataframe/dataframe.py

View check run for this annotation

Codecov / codecov/patch

modin/core/dataframe/pandas/dataframe/dataframe.py#L3870 

Added line #L3870 was not covered by tests

def compute_aligned_columns(*dfs, initial_columns=None):
"""Take row partitions, filter empty ones, and return joined columns for them."""
Expand Down
19 changes: 10 additions & 9 deletions modin/pandas/test/test_groupby.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3164,14 +3164,6 @@ def test_groupby_agg_provided_callable_warning():
pandas_groupby.agg(func)


def _apply_transform(df):
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dask hangs if I try to submit a function from the global testing scope, so rewritten this function as a lambda

if len(df) == 0:
df = df.copy()
df.loc[0] = 10
return df.squeeze()
return df.sum()


@pytest.mark.parametrize(
"modify_config", [{RangePartitioningGroupby: True}], indirect=True
)
Expand All @@ -3183,7 +3175,16 @@ def _apply_transform(df):
pytest.param(lambda grp: grp.sum(), id="sum"),
pytest.param(lambda grp: grp.size(), id="size"),
pytest.param(lambda grp: grp.apply(lambda df: df.sum()), id="apply_sum"),
pytest.param(lambda grp: grp.apply(_apply_transform), id="apply_transform"),
pytest.param(
lambda grp: grp.apply(
lambda df: (
df.sum()
if len(df) > 0
else pandas.Series([10] * len(df.columns), index=df.columns)
)
),
id="apply_transform",
),
],
)
@pytest.mark.parametrize(
Expand Down