[SPARK-33529][SQL] Handle '__HIVE_DEFAULT_PARTITION__' while resolving V2 partition specs

[MaxGekk]

What changes were proposed in this pull request?

1. Extract the code for partition values casting from DSv1 to the common place sql.util.PartitioningUtils - the method castPartitionValues().
2. Re-use castPartitionValues() from DSv2 resolver of partition specs - ResolvePartitionSpec.

Why are the changes needed?

To have the same behavior as DSv1 which interprets __HIVE_DEFAULT_PARTITION__ as NULL:

spark-sql> CREATE TABLE tbl11 (id int, part0 string) USING parquet PARTITIONED BY (part0);
spark-sql> ALTER TABLE tbl11 ADD PARTITION (part0 = '__HIVE_DEFAULT_PARTITION__');
spark-sql> INSERT INTO tbl11 PARTITION (part0='__HIVE_DEFAULT_PARTITION__') SELECT 1;
spark-sql> SELECT * FROM tbl11;
1	NULL

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

Add new test to AlterTablePartitionV2SQLSuite.

[MaxGekk]

Addressed @cloud-fan 's comment #30454 (comment)

[MaxGekk]

'__HIVE_DEFAULT_PARTITION__' should be handled as null

[SparkQA]

Test build #131641 has finished for PR 30482 at commit 4ad95c5.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

I'm not sure about it. It's more like a hive specific thing and we should let v2 implementation to decide how to handle null partition values. This should be internal details and shouldn't be exposed to end users.

[MaxGekk]

ok. How can users specify null partition value?

[cloud-fan]

does part_col = null work?

[MaxGekk]

For example, if we have a string partitioned column - how could we distinguish null from "null"?

[cloud-fan]

The parser should recognize different literals, e.g. part_col = null and part_col = "null"

[MaxGekk]

does part_col = null work?

I have checked that. null is recognized as a string "null".

[MaxGekk]

It's more like a hive specific thing and we should let v2 implementation to decide ...

It is already Spark specific thing too. Implementations don't see '__HIVE_DEFAULT_PARTITION__' at all because it is replaced by null at the analyzing phase.

[SparkQA]

Test build #131647 has finished for PR 30482 at commit 26b83a1.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MaxGekk]

jenkins, retest this, please

[SparkQA]

Test build #131694 has started for PR 30482 at commit 26b83a1.

[MaxGekk]

jenkins, retest this, please

[SparkQA]

Test build #131833 has finished for PR 30482 at commit 26b83a1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MaxGekk]

@cloud-fan Should I close this?

[cloud-fan]

Yea let's close it. __HIVE_DEFAULT_PARTITION__ should just be a normal string. Only hive catalog should handle it specially.