[SPARK-31682][SQL] Turn on spark.sql.legacy.createHiveTableByDefault.enabled by default

[Ngone51]

What changes were proposed in this pull request?

Set spark.sql.legacy.createHiveTableByDefault.enabled to true by default.

Why are the changes needed?

According to the latest status of [DISCUSS] Resolve ambiguous parser rule between two "create table"s, we decide to turn this conf on by default to unblock Spark 3.0 release.

Does this PR introduce any user-facing change?

No(comparing to Spark 2.4)

How was this patch tested?

Pass jenkins.

[Ngone51]

@cloud-fan @gatorsmile @HeartSaVioR @rdblue

[gatorsmile]

Also check the migration guide. We need to remove one line/

[SparkQA]

Test build #122507 has finished for PR 28500 at commit ffccb4a.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #122509 has finished for PR 28500 at commit 0fc974f.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Hi, @Ngone51 .
Could you fix the UT? Those are relevant failures.

[rdblue]

+1 for this in master as an intermediate step.

-1 for this in 3.0 because it is not safe to release 3.0 with SPARK-30098, even with the property turned off. That PR introduces confusing behavior for CREATE TABLE that is not well understood. I don't think it is a good idea to include SPARK-30098 because we would have to support the flag with that behavior.

[dongjoon-hyun]

Hi, @Ngone51 . Since you are already an experienced contributor, I want to add some comments.

First of all, this is not SPARK-30098.

[SPARK-30098][SQL] Use default datasource as provider for CREATE TABLE syntax

Also, technically, I tried this as SPARK-31136 officially. And, at that time, @cloud-fan and @HyukjinKwon were there. Since SPARK-31136 was not accepted and the JIRA issue is closed. I only made some test cases neutral to the conf.

• #27894

Given the updated circumstance (the discussions and email), I agree with this. However, we had better file a new JIRA as a subtask of SPARK-31085 and update this PR's title. This decision is worth of it. Thanks.

[HeartSaVioR]

The discussion thread is still active. Let's make a consensus there before moving on.

[Ngone51]

However, we had better file a new JIRA as a subtask of SPARK-31085 and update this PR's title.

Thanks for your advice. Updated.

[dongjoon-hyun]

Thank you so much, @Ngone51 .

[dongjoon-hyun]

Since #28517 is merged, I'll close this PR. Thank you all.