[SPARK-43851][SQL] Support LCA in grouping expressions #41804
Conversation
...yst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveReferencesInAggregate.scala
Outdated
Show resolved
Hide resolved
| g.transformWithPruning(_.containsAnyPattern(UNRESOLVED_ATTRIBUTE, | ||
| LATERAL_COLUMN_ALIAS_REFERENCE)) { |
There was a problem hiding this comment.
The patterns can guarantee that we have only UnresolvedAttribute and LateralColumnAliasReference, so you can write more generic code:
case u: NamedExpression =>
selectList.find(ne => conf.resolver(ne.name, u.name)).getOrElse(u)There was a problem hiding this comment.
I'm not sure if this way will match cases that shouldn't be matched, because _.containsAnyPattern(UNRESOLVED_ATTRIBUTE, LATERAL_COLUMN_ALIAS_REFERENCE also return true when one plan children node contains UnresolvedAttribute or LateralColumnAliasReference. Then may match other NamedExpression
| selectList.find(ne => conf.resolver(ne.name, u.name)).getOrElse(u) | ||
| g.transformWithPruning(_.containsAnyPattern(UNRESOLVED_ATTRIBUTE, | ||
| LATERAL_COLUMN_ALIAS_REFERENCE)) { | ||
| case u @ (_: UnresolvedAttribute | _: LateralColumnAliasReference) => |
There was a problem hiding this comment.
shall we update the lca resolution rule to not resolve columns in grouping expressions to LateralColumnAliasReference? The produced plan is very weird. also cc @anchovYu
There was a problem hiding this comment.
Thanks @cloud-fan , I'll try to implement it in followup PR, please help to see if it meets the requirements when it ready.
What changes were proposed in this pull request?
This PR bring support lateral column alias reference in grouping expressions.
Why are the changes needed?
add new feature for LCA
Does this PR introduce any user-facing change?
No
How was this patch tested?
exist test