-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[C++] Group_By from Substrait Shuffles Column Names #33616
Comments
I'm pretty sure the reason this happens is that Acero's aggregate node outputs in We could change the aggregate node but this would be a breaking change for things like If we want to avoid the breaking change then we could patch this in the emit step. This is very similar to a problem faced by the asof join node which is being addressed in #14799 or the solution that we used for project where we provided a specialization of ProcessEmitInfo. |
This is a duplicate of #32897 but this issue has a reproducer and more information so I am going to close the other one. |
@jorisvandenbossche @amol- @AlenkaF any particular opinion on this change? I agree that keys followed by measures is more intuitive (this matches SQL). However, it would be a breaking change so I didn't want to make the change in pyarrow if not needed (I can fix Substrait without changing pyarrow behavior pretty easily) |
I am personally fine with changing this in the user-facing behaviour of pyarrow's |
… before aggregates (apache#34551) * Closes: apache#33616 Lead-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Weston Pace <weston.pace@gmail.com>
… before aggregates (apache#34551) * Closes: apache#33616 Lead-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Weston Pace <weston.pace@gmail.com>
Describe the bug, including details regarding any error messages, version, and platform.
When using a Substrait plan that includes a group_by at the end, it appears that the resulting table will have correct results, but the columns have their names shuffled. This occurs on nightly Arrow 11.0.0.dev406, and a reproducer is included below. The plan in question is included in blob form in the reproducer, but comes from the following Ibis code:
The particular error is as follows: instead of the continent column storing continents in the output table, it stores total population. The total population column stores average area, and the average area column stores continents. This is interpretable, but likely problematic if this table is meant for ingestion elsewhere.
Reproducer:
Component(s)
C++
The text was updated successfully, but these errors were encountered: