[SPARK-18146] [SQL] Avoid using Union to chain together create table and repair partition commands #15665
Conversation
|
Test build #67677 has finished for PR 15665 at commit
|
| val createCmd = CreateTable(tableDesc, mode, Some(df.logicalPlan)) | ||
| val cmd = if (tableDesc.partitionColumnNames.nonEmpty && | ||
| val result = df.sparkSession.sessionState.executePlan( | ||
| CreateTable(tableDesc, mode, Some(df.logicalPlan))) |
There was a problem hiding this comment.
Is the execution of this command triggered at here?
There was a problem hiding this comment.
Looks like you want to call result.toRdd at here to trigger the execution of a command? Then inside the loop to use toRdd again.
There was a problem hiding this comment.
yea, we don't need the result variable.
|
Test build #67689 has finished for PR 15665 at commit
|
| @@ -254,7 +254,8 @@ final class DataFrameWriter[T] private[sql](ds: Dataset[T]) { | |||
| ) | |||
| } | |||
|
|
|||
| df.sparkSession.sessionState.executePlan( | |||
| df.sparkSession.sessionState. | |||
There was a problem hiding this comment.
what's going on here? a dot ?
There was a problem hiding this comment.
Some github text editing gone wrong... fixed it
|
Test build #67712 has finished for PR 15665 at commit
|
|
Test build #67714 has finished for PR 15665 at commit
|
|
Test build #67716 has finished for PR 15665 at commit
|
|
cc @hvanhovell @cloud-fan did you have anything to do with this in the past? |
|
This was a discussed follow-up the pr on datasource partition management: #15515 |
|
LGTM, merging to master! |
…nd repair partition commands ## What changes were proposed in this pull request? The behavior of union is not well defined here. It is safer to explicitly execute these commands in order. The other use of `Union` in this way will be removed by apache#15633 ## How was this patch tested? Existing tests. cc yhuai cloud-fan Author: Eric Liang <ekhliang@gmail.com> Author: Eric Liang <ekl@databricks.com> Closes apache#15665 from ericl/spark-18146.
…nd repair partition commands ## What changes were proposed in this pull request? The behavior of union is not well defined here. It is safer to explicitly execute these commands in order. The other use of `Union` in this way will be removed by apache#15633 ## How was this patch tested? Existing tests. cc yhuai cloud-fan Author: Eric Liang <ekhliang@gmail.com> Author: Eric Liang <ekl@databricks.com> Closes apache#15665 from ericl/spark-18146.
What changes were proposed in this pull request?
The behavior of union is not well defined here. It is safer to explicitly execute these commands in order. The other use of
Unionin this way will be removed by #15633How was this patch tested?
Existing tests.
cc @yhuai @cloud-fan