[C++] [Parquet] Writing uint32 does not preserve parquet's LogicalType

[asfimport]

When writing a uint32 column, (parquet's) logical type is not written, limiting interoperability with other engines.

Minimal Python

import pyarrow as pa

data = {"uint32", [1, None, 0]}
schema = pa.schema([pa.field('uint32', pa.uint32())])

t = pa.table(data, schema=schema)
pa.parquet.write_table(t, "bla.parquet")

 
Inspecting it with spark:

from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

df = spark.read.parquet("bla.parquet")
print(df.select("uint32").schema)

shows StructType(List(StructField(uint32,LongType,true))). "LongType" indicates that the field is interpreted as a 64 bit integer. Further inspection of the metadata shows that both convertedType and logicalType are not being set. Note that this is independent of the arrow-specific schema written in the metadata.

Reporter: Jorge Leitão / @jorgecarleitao

Related issues:

• [C++][Python] Switch default Parquet version to 2.4 (is related to)

_{Note: This issue was originally created as ARROW-12201. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Hmm, that's bummer. Hopefully we can fix this for 4.0.0.

[asfimport]

Antoine Pitrou / @pitrou:
Oh, actually, you simply need to add version='2.0' to the write_table call, and the annotation will be written out.

[asfimport]

Antoine Pitrou / @pitrou:
Will close as this works as intended (by default maximum compatibility with old readers is ensured). However, we may want to bump the default version as some point.

[asfimport]

Jorge Leitão / @jorgecarleitao:
Got it.

In this case, doesn't compatibility require setting ConvertedType = UINT_32?

[asfimport]

Antoine Pitrou / @pitrou:
I have no idea why it wasn't done. This seems to date back to a 2016 PR:
https://github.com/apache/parquet-cpp/pull/158/files#r607025397

[asfimport]

Jorge Leitão / @jorgecarleitao:
holy cow. Ok, that explains it. Thanks a lot, and sorry for the noise.

[asfimport]

Micah Kornfield / @emkornfield:
There was another bug that discussed this at length. int64 for int32 was used so the type could properly round trip. But yes this is WAI.