fix: groupby-sum fast path reduces multi-dim DataArray grouper over all its dims#824
Conversation
…sum fast path Detect a genuine N-D grouper (all dims are data dims) and stack the data over those dims into a single key, so the fast path reduces over every grouper dim at once instead of routing a 2-D grouper through to_pandas() into the multikey path (which leaked one dim into the result).
Merging this PR will not alter performance
Comparing Footnotes
|
|
@FabianHofmann Great that you found that bug right away. Im really sorry about the regression! EDIT: Nevermind, seems like it wasnt my PR... |
|
@FabianHofmann I dived a bit deeper in and found some other issues (positional alignment when using pd.Series or xr.DataArray). This is arguably a bug, but could also be part of v1. Ill create issues for both. We should decide if we treat this as a bugfix or a semantics thing! |
|
See #827 |
|
@FabianHofmann This should be merged no matter what happens in #827 |
Closes #823.
This aligns the grouping mechanism across fallback and non-fallback groupby operations.
Note
The following content was generated by AI.
Changes proposed in this Pull Request
LinearExpression.groupby(...).sum()returned wrong dimensions when grouping by a multi-dimensionalDataArraygrouper on its default (fast) path: the reduction happened over only one of the grouper's dimensions and the others survived in the result. Onlyuse_fallback=Truegave the correct result.Root cause: the fast path did
group.to_pandas(), which turns a 2-D grouper into aDataFramethat is then routed through the multikey encoding path — reducing over only the row axis. A genuinely N-D grouper (whose values define the groups jointly over all its dims) was never handled. This is a long-standing bug present since v0.8.0, not a regression from #802.The fix:
DataArraywithndim > 1whose dims are all data dims — and stack the data over those dims into a single key, so the fast path reduces over every grouper dim at once. This preserves the fast path's memory benefit.DataArrays whose extra axis is not a data dim (i.e. multikey frames built from aDataFrame) keep flowing through the existingDataFrame/multikey handling._grouped_sumnow accepts the (stacked) dataset to operate on.Tests
test_linear_expression_groupby_ndimnow exercises the fast path too (use_fallback=[True, False]) and asserts it equals the fallback; the stale TODO was removed.test_linear_expression_groupby_multidim_preserves_extra_dimcovers the issue's reproducer (2-D grouper over(i, j), preservedk).Checklist
AGENTS.md).doc.doc/release_notes.rstof the upcoming release is included.