[GO]: pqarrow (github.com/apache/arrow/go/v9/parquet/pqarrow) cannot handle arrow's DICTIONARY field

[asfimport]

Hey, Arrow Go Dev:
 
I was trying to save some arrow tables out to parquet files, with the help of the "github.com/apache/arrow/go/v9/parquet/pqarrow" package. btw, it's generally a great design (of Arrow) and a great Go implementation. 

 
However, one issue sticks out: in my original arrow Table I have some DICTIONARY fields, which pqarrow does NOT currently support.
 
I would assume supporting them will be quite straightward: just "denormalize" the DICTIONARY value into corresponding values (string, Timestamp, etc), and it's up to the Parquet to do the right trick (using DICTIONARY encoding, etc). 
 
I would have done this conversion on-the-fly by myself, by converting each DICTIONARY field into underlying values. However, the arrow table schema is dynamic and outside my control, and I need to iterate through fields (maybe structs) to locate those) -> it would be much better if pqarrow can support this natively. 
 
Can anyone help? thanks!

Reporter: Kevin Yang
Assignee: Matthew Topol / @zeroshade

_{Note: This issue was originally created as ARROW-18288. Please see the migration documentation for further details.}

[asfimport]

Matthew Topol / @zeroshade:
This isn't quite as straightforward as denormalizing the values in order to ensure proper statistics handling and efficient propagation. (Sure, you could just naively denormalize and then write, but that could cause unnecessary copies and other inefficient handling). I've started working on this but ran into a couple snags where I will need to utilize and enhance the compute package. I'll have a few things up for this soon as I work it out as this is going to require:

• Enabling proper casting from Dictionary types to values (unpacking dictionaries)
• One of the following:
  • Implementing hash kernels for the Compute module to efficiently perform a unique operation on the dictionary indexes to find the min/max for stats
  • Implementing aggregation kernels to implement using MinMax to find the min-max on the dictionary array directly (more efficient than hash for uniqueness but will take longer / harder to do).