[SPARK-47681][SQL] Add schema_of_variant expression.

[chenhao-db]

What changes were proposed in this pull request?

This PR adds a new SchemaOfVariant expression. It returns schema in the SQL format of a variant.

Usage examples:

> SELECT schema_of_variant(parse_json('null'));
 VOID
> SELECT schema_of_variant(parse_json('[{"b":true,"a":0}]'));
 ARRAY<STRUCT<a: BIGINT, b: BOOLEAN>>

Why are the changes needed?

This expression can help the user explore the content of variant values.

Does this PR introduce any user-facing change?

Yes. A new SQL expression is added.

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[HyukjinKwon]

Can we reuse TypeCoercion rule here?

[chenhao-db]

I personally like to keep the current code. I don't think TypeCoercion contains a suitable rule that can be directly used here. If we use findTightestCommonType, we still need most of the code that handles decimal/struct/array and can hardly simplify the code. If we use any function that calls findWiderTypeForDecimal (like findWiderTypeForTwo), its semantics will be undesired because If the wider decimal type exceeds system limitation, this rule will truncate the decimal type (and we still need custom code for struct/array). Using these rules may fruitlessly visit the whole type object, and we need to do a second pass of visit. Since this function is used in the expression evaluation, I think we do care about its efficiency.

Essentially, mergeSchema only need to handle the result of mergeSchema and schemaOf, and we can have a better control over them if we have all the type resolution logic inside and avoid calling any libaray functions.

[cloud-fan]

Interesting, so the variant format spec does not require the array elements to be the same data type.

[chenhao-db]

This is true. It is necessary because JSON allows such flexibility.

[cloud-fan]

The problem of not reusing existing code is we need to reason about the behavior difference. Why do you think it's better to return double than rounding the decimal? Double is less accurate than a rounded decimal.

[cloud-fan]

Does AnsiTypeCoercion.findWiderTypeForTwo(...).getOrElse(VariantType) work?

[chenhao-db]

I think it will be more intuitive if the variant can be successfully cast into the inferred schema (even at the cost of a precision loss). If we return a truncated decimal here, the cast will deterministically fail.

Actually, this mergedSchema function, including this fallback-to-double logic, is largely adapted from the existing JSON schema inference code: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala#L364. I find it is not too difficult to reuse this code, so I changed the implementation to depend on it instead.

[cloud-fan]

shall we call it defaultDataType?

[cloud-fan]

thanks, merging to master!