[Variant] Support more Arrow Datatypes from Variant primitive types

[klion26]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We have implemented the variant primitive to arrow types for all variant primitive types now(1-1 mapping), but maybe the user will pass a different type which is not a valid arrow primitive type, this issue wants to discuss which arrow types we will support transformed from variant primitive with cast, or we don't need to support these types.

Describe the solution you'd like

Here is data types I collected for now

Arrow DataType	Variant primitive Type	Cast logic
Date64	Date32	Date32 * (24 * 60 * 60 * 1000)`
Timestamp_[ntz](Second)	Timestamp_[ntz](Micros/Nanos)	- use micros as an example - let the timestamp be t - there are two options for unhappy path 1. return None if t % 1000000 != 0 2 return floor[t / 1000000]
TImestamp_[ntz](Millisecond)	Timestamp_[ntz](Micros/Nanos)	Same as above
Time32(Second/Millisecond)	Time	same as above
Time64(Nanosecond)	Time(Macors)	same as above

Describe alternatives you've considered

If there is any types we want to support, we can ref to the logic of variant_to_arrow.rs#make_variant_to_arrow_row_builder for more info

Additional context

[klion26]

cc @alamb @scovich @liamzwbao @friendlymatthew

[scovich]

Seems reasonable. Do we want to introduce more granular cast options perhaps?

• Strict (or not): values that cannot be converted produce an error (or null)
• Lossy (or not): casts are allowed to lose information (or fail)
• Converting (or not): conversions are allowed to convert to/from string (or fail)

If we allow string conversions tho, then we need to get into locales, timestamp formats, etc. Very messy.

[klion26]

@scovich Sorry for the late reply(missed this notification...).

Adding strict/lossy/converting would be a better option and could be applied to the conversion process described above.

Regarding string conversion, I think it's okay to not support it here currently. My understanding of the conversion is that converting string to something like a timestamp would be quite messy, and if necessary, the caller could perform another string conversion. If we find that supporting this would be more beneficial, we can continue to support it in the future.

[klion26]

I found that there are some related issues[1][2] when trying to implement this one. [1] wants to define the behavior of casting error in variant_get, [2] wants to unify the castOptions in parquet-variant-compute.

Currently, the plan is to implement the current issue without introducing any more casting logic, so that we'll have all arrow primitive types supported for variant to arrow(without Interval and Duration -- which I think we can't support for now)

Then we can implement more cast logic when [1][2] reaches a consensus, after that, we'll revisit all the places that used the cast option, respecting the new cast logics we added(if any)

[1] #8086
[2] #8873

[klion26]

take