[SPARK-4293][SQL] Make Cast be able to handle complex types.

[ueshin]

Inserting data of type including ArrayType.containsNull == false or MapType.valueContainsNull == false or StructType.fields.exists(_.nullable == false) into Hive table will fail because Cast inserted by HiveMetastoreCatalog.PreInsertionCasts rule of Analyzer can't handle these types correctly.

Complex type cast rule proposal:

• Cast for non-complex types should be able to cast the same as before.
• Cast for ArrayType can evaluate if
  • Element type can cast
  • Nullability rule doesn't break
• Cast for MapType can evaluate if
  • Key type can cast
  • Nullability for casted key type is false
  • Value type can cast
  • Nullability rule for value type doesn't break
• Cast for StructType can evaluate if
  • The field size is the same
  • Each field can cast
  • Nullability rule for each field doesn't break
• The nested structure should be the same.

Nullability rule:

• If the casted type is nullable == true, the target nullability should be true

[SparkQA]

Test build #23041 has started for PR 3150 at commit 287f410.

• This patch merges cleanly.

[SparkQA]

Test build #23041 has finished for PR 3150 at commit 287f410.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23041/
Test PASSed.

[chenghao-intel]

I am wondering if throwing exception will be more informative, than plain UnresolvedException thrown in logical plan analyzing.

[ueshin]

Hmm, I think the resolve check should be in logical plan analyzing.

[ueshin]

Some expressions are checking the resolved in the dataType method, though.

[chenghao-intel]

It looks good to me in general, and I like the idea of summarizing the convertible data type checking, but in the meantime, I am a little afraid it might be error-prone for future maintenance or new data type added.
Or can we remove the resolve method?

[ueshin]

@chenghao-intel, Thank you for your comments.
If resolve method is removed, the nullability check (e.g. cast from ArrayType(IntegerType, containsNull = true) to ArrayType(IntegerType, containsNull = false) is apparently invalid) is also removed and it will cause unexpected errors. If there is a better way to ensure the nullability check, we can remove the method.

[SparkQA]

Test build #23081 has started for PR 3150 at commit f677c30.

• This patch merges cleanly.

[SparkQA]

Test build #23081 has finished for PR 3150 at commit f677c30.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23081/
Test PASSed.

[SparkQA]

Test build #23096 has started for PR 3150 at commit 8999868.

• This patch merges cleanly.

[SparkQA]

Test build #23096 has finished for PR 3150 at commit 8999868.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23096/
Test PASSed.

[SparkQA]

Test build #23408 has started for PR 3150 at commit ba14003.

• This patch merges cleanly.

[SparkQA]

Test build #23408 has finished for PR 3150 at commit ba14003.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23408/
Test PASSed.

[SparkQA]

Test build #23899 has started for PR 3150 at commit e935939.

• This patch merges cleanly.

[SparkQA]

Test build #23899 has finished for PR 3150 at commit e935939.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23899/
Test PASSed.

[marmbrus]

Sorry for the delay and thanks for working on this! Merging to master.

[marmbrus]

Actually, apache is down :( LGTM, will merge later.