-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-8972][SQL]Incorrect result for rollup #7343
Conversation
Test build #37024 has finished for PR 7343 at commit
|
Can you explain what is the problem in the JIRA? |
Thanks you @yhuai , I've updated the description of the bug, and also add more unit tests for the fixing. |
Test build #37112 has finished for PR 7343 at commit
|
@chenghao-intel Thanks for the update. I find |
That's correct, I am not just simply change the group by keys, but with an alias. e.g.
|
nonAttributeGroupByExpression += alias | ||
(other, alias.toAttribute) // (Aliased complex expression, the associated attribute) | ||
} | ||
}) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe add toMap
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably we want to keep the order group by expressions in the Aggregate operator, let's just keep it as it used to be.
oh, i see. Yeah, you are right. Can we use another term for |
@yhuai thanks for the reviewing, any better idea for the term |
Test build #1050 has finished for PR 7343 at commit
|
Test build #37300 has finished for PR 7343 at commit
|
// find all of the non-attribute expressions in the GROUP BY keys | ||
val nonAttributeGroupByExpressions = new ArrayBuffer[Alias]() | ||
|
||
// The pair of (the non-attributes expression, associated attribute (alias)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is not very accurate since you also have named expressions as keys.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yes, I will update it.
The fix looks good to me. |
Test build #37313 has finished for PR 7343 at commit
|
seems not related. |
retest this please |
Test build #21 has finished for PR 7343 at commit
|
Test build #37340 has finished for PR 7343 at commit
|
@yhuai any more comments? |
LGTM. I am merging it to master. |
We don't support the complex expression keys in the rollup/cube, and we even will not report it if we have the complex group by keys, that will cause very confusing/incorrect result.
e.g.
SELECT key%100 FROM src GROUP BY key %100 with ROLLUP
This PR adds an additional project during the analyzing for the complex GROUP BY keys, and that projection will be the child of
Expand
, so toExpand
, the GROUP BY KEY are always the simple key(attribute names).