[SPARK-16825] [SQL] Replace hive.default.fileformat by spark.sql.default.fileformat

[gatorsmile]

What changes were proposed in this pull request?

Currently, we are using hive.default.fileformat for specifying the default file format in CREATE TABLE statement. Multiple issues exist:

• This parameter value is not from hive-site.xml. Thus, even if users change the hive.default.fileformat in hive-site.xml, Spark will ignore it. To change the parameter values, users have to use Spark interface, (e.g., by a SET command or API).
• This parameter is not documented.
• This parameter value will not be sent to Hive metastore. It is being used by Spark internals when processing CREATE TABLE statement.
• This parameter is case sensitive.

Since this is being used by Spark only, it does not make sense to use a parameter starting from hive. we might follow the other Hive-related parameters and introduce a new Spark parameter here. It should be public. Thus, this PR is to replace hive.default.fileformat by spark.sql.default.fileformat. It also makes it case insensitive.

How was this patch tested?

Added test cases

[gatorsmile]

These statements are dead code. If we really need, we have to use the runSqlHive API of HiveClient. Since it does not cause errors, I remove them.

[SparkQA]

Test build #63056 has finished for PR 14430 at commit 0dc7cda.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #63057 has finished for PR 14430 at commit c0817fe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[rxin]

We already have default sources. It is very confusing to have default format on top of that.

[gatorsmile]

@rxin That is definitely true! I had the same concern when changing the code.

Now, we are having two types of tables: one is Data source tables and the other is Hive tables. The default value of the default format for data source tables is parquet. The default format of hive tables is textfile, if users do not specify the format.

Maybe we can use the same configuration parameter but different default values? spark.sql.sources.default. Let me try it and you can judge whether it is better

[SparkQA]

Test build #63067 has finished for PR 14430 at commit 7b64edc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[xmubeta]

Please bear me if my question is silly.

I have the same request to set the default file format to parquet for spark-sql. I didn't see this PR was merged into any version. Is that true? Does it mean I need to patch the code by myself? Any other way to set the default file format other than using set command in session.

Thank you.