[SPARK-15025][SQL] fix duplicate of PATH key in datasource table options

[xwu0226]

What changes were proposed in this pull request?

The issue is that when the user provides the path option with uppercase "PATH" key, options contains PATH key and will get into the non-external case in the following code in createDataSourceTables.scala, where a new key "path" is created with a default path.

val optionsWithPath =
      if (!options.contains("path")) {
        isExternal = false
        options + ("path" -> sessionState.catalog.defaultTablePath(tableIdent))
      } else {
        options
      }

So before creating hive table, serdeInfo.parameters will contain both "PATH" and "path" keys and different directories. and Hive table's dataLocation contains the value of "path".

The fix in this PR is to convert options in the code above to CaseInsensitiveMap before checking for containing "path" key.

How was this patch tested?

A testcase is added

[xwu0226]

@yhuai @liancheng @liancheng @hvanhovell Can any of you help take a quick look at this change? Thank you very much!

[yhuai]

ok to test.

[SparkQA]

Test build #57819 has finished for PR 12804 at commit 633c850.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

Can we avoid of calling metadataHive.runSqlHive? For this case, we want to check that there is a single entry in table properties with the key as PATH and the location is the one we want, right?

[xwu0226]

@yhuai Thanks for your input!. Yes, you are right. I can get the CatalogTable.storage.serdeProperties to check the 'PATH' key. I will modify the test case.

[xwu0226]

@yhuai I updated the testcase to check the key path from CatalogTable.storage.serdeProperties. In this case, we are supposed to see only PATH key-value. Let me know if you have any question. Thanks!

[SparkQA]

Test build #58167 has finished for PR 12804 at commit 8c3fbe6.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #58166 has finished for PR 12804 at commit 1f6264e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

test this please

[SparkQA]

Test build #58174 has finished for PR 12804 at commit 8c3fbe6.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[yhuai]

LGTM. Merging to master and branch 2.0.