-
Notifications
You must be signed in to change notification settings - Fork 28.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unresolved state #20914
Conversation
ok to test |
Test build #88630 has finished for PR 20914 at commit
|
|
retest this please |
Test build #88642 has finished for PR 20914 at commit
|
retest this please |
Test build #88646 has finished for PR 20914 at commit
|
be950e4
to
d9a2ee8
Compare
Test build #88705 has finished for PR 20914 at commit
|
@@ -107,7 +112,7 @@ class PropagateEmptyRelationSuite extends PlanTest { | |||
val query = testRelation1 | |||
.where(left) | |||
.join(testRelation2.where(right), joinType = jt, condition = Some('a.attr == 'b.attr)) | |||
val optimized = Optimize.execute(query.analyze) | |||
val optimized = Optimize.execute(query.analyze).analyze |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the subsequent analyze here is to fill in timezones on casts inserted by Optimize
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should fill the time zone in the optimizer rule
Test build #88717 has finished for PR 20914 at commit
|
@@ -43,7 +43,7 @@ object PropagateEmptyRelation extends Rule[LogicalPlan] with PredicateHelper { | |||
|
|||
// Construct a project list from plan's output, while the value is always NULL. | |||
private def nullValueProjectList(plan: LogicalPlan): Seq[NamedExpression] = | |||
plan.output.map{ a => Alias(Literal(null), a.name)(a.exprId) } | |||
plan.output.map{ a => Alias(Cast(Literal(null), a.dataType), a.name)(a.exprId) } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please use CastSupport.cast
Test build #88759 has finished for PR 20914 at commit
|
@gatorsmile how does it look now? |
LGTM |
…esolved state ## What changes were proposed in this pull request? Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules ## How was this patch tested? Added unit test Author: Robert Kruszewski <robertk@palantir.com> Closes #20914 from robert3005/rk/propagate-empty-fix. (cherry picked from commit 5cfd5fa) Signed-off-by: gatorsmile <gatorsmile@gmail.com>
…esolved state ## What changes were proposed in this pull request? Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules ## How was this patch tested? Added unit test Author: Robert Kruszewski <robertk@palantir.com> Closes apache#20914 from robert3005/rk/propagate-empty-fix.
…esolved state ## What changes were proposed in this pull request? Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules ## How was this patch tested? Added unit test Author: Robert Kruszewski <robertk@palantir.com> Closes apache#20914 from robert3005/rk/propagate-empty-fix.
…esolved state Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules Added unit test Author: Robert Kruszewski <robertk@palantir.com> Closes apache#20914 from robert3005/rk/propagate-empty-fix. (cherry picked from commit 5cfd5fa) Signed-off-by: gatorsmile <gatorsmile@gmail.com> Change-Id: I9c673e572454a9fa43e296b7fe66e7f2fc569854
What changes were proposed in this pull request?
Add cast to nulls introduced by PropagateEmptyRelation so in cases they're part of coalesce they will not break its type checking rules
How was this patch tested?
Added unit test