variant: Support Arrow UInt8/UInt16/UInt32 typed values

[HuaHuaY]

Which issue does this PR close?

Closes #10416.

Rationale for this change

Arrow allows mapping Uint8/16/32 to Parquet types Int16/32/64. However, the current code does not support this.

Arrow's documentation: https://arrow.apache.org/docs/format/CanonicalExtensions.html#primitive-type-mappings

What changes are included in this PR?

• Support UInt8, UInt16, and UInt32 Arrow Variant typed_value fields.
• Continue rejecting unsigned types in Parquet Variant schemas.
• Add lossless UInt32 Arrow array to Parquet INT64 writer coercion

Are these changes tested?

Yes.

Are there any user-facing changes?

VariantArray and shred_variant now support canonical UInt8/UInt16/UInt32 typed_value fields.

ArrowWriter can also write UInt32 arrays to INT64 columns when using a compatible custom Parquet schema. This is because it's not convenient to distinguish whether the column is in a variant within the context of arrow_writer/mod.rs. I think this is a side effect of this PR; please point out any better suggestions.

[HuaHuaY]

Hi @alamb and @sdf-jkl. When implementing variant code for the arrow cpp repository, I noticed inconsistencies between arrow-rs and the arrow specification. However, I am not familiar with the developers of the arrow-rs repository. Could you please review the code or tag the relevant developers?

[sdf-jkl]

Hi @HuaHuaY!

Arrow Variant closely follows the Parquet Variant spec - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md

The arrow primitive type mapping you've linked is following the table from the spec here - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types - It's about unshredded Variant.

The shredding spec here - https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types - lists which Parquet types should be mapped to each Variant type for shredding.

Variant Type	Parquet Physical Type	Parquet Logical Type
...
int8	INT32	INT(8, true)
int16	INT32	INT(16, true)
int32	INT32
int64	INT64
...

Here Int* Variant types have to be represented by signed integers in Parquet -> we use signed int arrow arrays to store them.

cc @klion26

[HuaHuaY]

Hi @HuaHuaY!

Arrow Variant closely follows the Parquet Variant spec - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md

The arrow primitive type mapping you've linked is following the table from the spec here - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types - It's about unshredded Variant.

The shredding spec here - https://github.com/apache/parquet-format/blob/master/VariantShredding.md#shredded-value-types - lists which Parquet types should be mapped to each Variant type for shredding.

Variant Type Parquet Physical Type Parquet Logical Type
...
int8 INT32 INT(8, true)
int16 INT32 INT(16, true)
int32 INT32
int64 INT64
...
Here Int* Variant types have to be represented by signed integers in Parquet -> we use signed int arrow arrays to store them.

cc @klion26

Hi @sdf-jkl!

As I understand it, the documentation you provided regarding "unshredded" and "shredded" variants refers to the mapping between variant primitive types and Parquet types. It does not address how Arrow's variant type maps to the Parquet variant. Notably, the Arrow documentation mentions unsigned integers, whereas the Parquet documentation does not.

This PR is about unsigned integer types. This PR doesn't aim to directly store unsigned integers; rather, it's precisely to allow arrow-rs to store them as signed integers.

[sdf-jkl]

@HuaHuaY you are correct! I was reading more into it and saw where I'm wrong.

The arrow primitive type mapping you've linked is following the table from the spec here - https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types - It's about unshredded Variant.

This point I made was incorrect and the arrow spec indeed says: "A field named typed_value which can be a Primitive Type Mappings or a List, LargeList, ListView or Struct"

So the PR and the issue make sense given the spec, but I wonder if the spec itself should support Uint for typed value.
Arrow Uint -> Parquet INT -> Arrow Int
Given any UInt will be written as a signed INT Parquet type and never read back as UInt, it never round trips. The only places a UInt typed_value can come from are in-memory construction or IPC.

I'll raise a question on the Arrow mailing list.

[sdf-jkl]

@HuaHuaY I submitted an issue here apache/arrow#50622