[Python] Expose new FLOAT16 logical type in the pyarrow.parquet bindings

[jorisvandenbossche]

Reading and writing data with a float16 field works just fine (because its implemented on the C++ side):

>>> table = pa.table({"a": np.array([0.1, 0.2], "float16"), "b": np.array([1, 2], "int8")})
>>> pq.write_table(table, "/tmp/test_float16.parquet")
>>> meta = pq.read_metadata("/tmp/test_float16.parquet")
>>> meta.schema
<pyarrow._parquet.ParquetSchema object at 0x7f488ec55000>
required group field_id=-1 schema {
  optional fixed_len_byte_array(2) field_id=-1 a (Float16);
  optional int32 field_id=-1 b (Int(bitWidth=8, isSigned=true));
}

But in a few parts of the API you can see we didn't add it to the python bindings:

>>> meta.schema.column(0).logical_type
<pyarrow._parquet.ParquetLogicalType object at 0x7f488ef60210>
  Float16
>>> meta.schema.column(1).logical_type
<pyarrow._parquet.ParquetLogicalType object at 0x7f4894c9fcf0>
  Int(bitWidth=8, isSigned=true)

>>> meta.schema.column(0).logical_type.type
'UNKNOWN'                                          # <--- UNKNOWN instead of FLOAT16 here
>>> meta.schema.column(1).logical_type.type
'INT'

That comes from

arrow/python/pyarrow/_parquet.pyx

Lines 1290 to 1306 in 374b8f6

	cdef logical_type_name_from_enum(ParquetLogicalTypeId type_):
	return {
	ParquetLogicalType_UNDEFINED: 'UNDEFINED',
	ParquetLogicalType_STRING: 'STRING',
	ParquetLogicalType_MAP: 'MAP',
	ParquetLogicalType_LIST: 'LIST',
	ParquetLogicalType_ENUM: 'ENUM',
	ParquetLogicalType_DECIMAL: 'DECIMAL',
	ParquetLogicalType_DATE: 'DATE',
	ParquetLogicalType_TIME: 'TIME',
	ParquetLogicalType_TIMESTAMP: 'TIMESTAMP',
	ParquetLogicalType_INT: 'INT',
	ParquetLogicalType_JSON: 'JSON',
	ParquetLogicalType_BSON: 'BSON',
	ParquetLogicalType_UUID: 'UUID',
	ParquetLogicalType_NONE: 'NONE',
	}.get(type_, 'UNKNOWN')

(it might actually be the only place to add it)

[tlm365]

take

[jorisvandenbossche]

Issue resolved by pull request 42103
#42103