[SPARK-10960][SQL] SQL with windowing function should be able to refer column in inner select #9011
Conversation
|
Test build #43331 has finished for PR 9011 at commit
|
|
cc @cloud-fan can you take a look at this? |
|
Hi @viirya , can you add more description about the reason of this bug? e.g. why does this bug exist and how do you handle it, so that other people can understand your fix easily. Thanks! |
|
@cloud-fan Thanks for suggesting. I've updated it. |
| case e: NamedExpression => extractExpr(e) | ||
| }) | ||
| ne | ||
|
|
There was a problem hiding this comment.
A one-line change can fix this and pass your test: case attr: Attribute => extractExpr(attr)
I think the missing case is Attribute, not Alias.
There was a problem hiding this comment.
Thanks. I've updated it according to your suggestion.
|
Test build #43507 has finished for PR 9011 at commit
|
|
ping @cloud-fan Any other comments? |
|
cc @yhuai to take another look. |
|
LGTM. @viirya Thank you for fixing this bug. |
|
Merging to master and 1.5 branch. |
| WindowData(5, "c", 9), | ||
| WindowData(6, "c", 10) | ||
| ) | ||
| sparkContext.parallelize(data).toDF().registerTempTable("windowData") |
There was a problem hiding this comment.
btw, you can use withTempTable in future to let our test infra automatically drop the temp table created at here.
There was a problem hiding this comment.
Ah, thanks for suggesting it.
…er column in inner select JIRA: https://issues.apache.org/jira/browse/SPARK-10960 When accessing a column in inner select from a select with window function, `AnalysisException` will be thrown. For example, an query like this: select area, rank() over (partition by area order by tmp.month) + tmp.tmp1 as c1 from (select month, area, product, 1 as tmp1 from windowData) tmp Currently, the rule `ExtractWindowExpressions` in `Analyzer` only extracts regular expressions from `WindowFunction`, `WindowSpecDefinition` and `AggregateExpression`. We need to also extract other attributes as the one in `Alias` as shown in the above query. Author: Liang-Chi Hsieh <viirya@appier.com> Closes #9011 from viirya/fix-window-inner-column. (cherry picked from commit fcb37a0) Signed-off-by: Yin Huai <yhuai@databricks.com>
…er column in inner select JIRA: https://issues.apache.org/jira/browse/SPARK-10960 When accessing a column in inner select from a select with window function, `AnalysisException` will be thrown. For example, an query like this: select area, rank() over (partition by area order by tmp.month) + tmp.tmp1 as c1 from (select month, area, product, 1 as tmp1 from windowData) tmp Currently, the rule `ExtractWindowExpressions` in `Analyzer` only extracts regular expressions from `WindowFunction`, `WindowSpecDefinition` and `AggregateExpression`. We need to also extract other attributes as the one in `Alias` as shown in the above query. Author: Liang-Chi Hsieh <viirya@appier.com> Closes #9011 from viirya/fix-window-inner-column. (cherry picked from commit fcb37a0) Signed-off-by: Yin Huai <yhuai@databricks.com>
JIRA: https://issues.apache.org/jira/browse/SPARK-10960
When accessing a column in inner select from a select with window function,
AnalysisExceptionwill be thrown. For example, an query like this:Currently, the rule
ExtractWindowExpressionsinAnalyzeronly extracts regular expressions fromWindowFunction,WindowSpecDefinitionandAggregateExpression. We need to also extract other attributes as the one inAliasas shown in the above query.