[SPARK-41726][SQL] Remove OptimizedCreateHiveTableAsSelectCommand
#39263
Conversation
| assert(commands.head.nodeName == "Execute CreateHiveTableAsSelectCommand") | ||
|
|
||
| val v1WriteCommand = commands(1) | ||
| if (isConverted && isConvertedCtas) { |
There was a problem hiding this comment.
the removed test in HiveExplainSuite.scala has been covered in this test.
|
cc @cloud-fan |
ulysses-you
force-pushed
the
SPARK-41726
branch
from
384a937
to
b91971b
Compare
| // `InsertIntoHiveTable` is derived from `CreateHiveTableAsSelectCommand`. | ||
| // This pattern would not cause conflicts because this rule is always applied before | ||
| // `HiveAnalysis` and both of these rules are running once. | ||
| case InsertIntoHiveTable(tableDesc, _, query, overwrite, ifPartitionNotExists, _) |
There was a problem hiding this comment.
wait, how do we optimize hive table insertion before? Given there is no case for InsertIntoHiveTable before.
There was a problem hiding this comment.
Before, we optimize hive table insertion inside OptimizedCreateHiveTableAsSelectCommand. It returns InsertIntoHadoopFsRelationCommand in getWritingCommand. Now, CreateHiveTableAsSelectCommand always use InsertIntoHiveTable for data writing and we match this pattern to do optimize.
There was a problem hiding this comment.
I mean normal table insertion, not table insertion inside CTAS.
There was a problem hiding this comment.
Before we match InsertIntoStatement and covert hive relation to HadoopFsRelation then datasource can tune it to InsertIntoHadoopFsRelationCommand. see line 223
That said, we can make CreateHiveTableAsSelectCommand use InsertIntoStatement instead of InsertIntoHiveTable but one issue is we can not aware of if the InsertIntoStatement is from CTAS. So here I use InsertIntoHiveTable.
There was a problem hiding this comment.
I see, can we make the code comments clearer? This only matches table insertion inside Hive CTAS.
There was a problem hiding this comment.
updated comment
ulysses-you
force-pushed
the
SPARK-41726
branch
2 times, most recently
from
00cf68f
to
280c255
Compare
ulysses-you
force-pushed
the
SPARK-41726
branch
from
280c255
to
8db2dbe
Compare
dongjoon-hyun
changed the title
OptimizedCreateHiveTableAsSelectCommand
There was a problem hiding this comment.
+1, LGTM. Merged to master for Apache Spark 3.4.
Thank you, @ulysses-you and @cloud-fan .
|
thank you @dongjoon-hyun and @cloud-fan ! |
What changes were proposed in this pull request?
This pr removes
OptimizedCreateHiveTableAsSelectCommandand move the code that tuneInsertIntoHiveTabletoInsertIntoHadoopFsRelationCommandintoRelationConversions.Why are the changes needed?
CTAS use a nested execution to do data writing, so it is unnecessary to have
OptimizedCreateHiveTableAsSelectCommand. The insideInsertIntoHiveTablewould be converted toInsertIntoHadoopFsRelationCommandif possible.Does this PR introduce any user-facing change?
no
How was this patch tested?
fix test