Truncate stats from Parquet files

[rdblue]

Lower and upper bound values from Parquet files are not currently truncated, which takes more space than necessary in manifests. Truncating strings and binary values will probably improve performance for large tables.

[rdblue]

@aokolnychyi, #78 reminded me about this item as well.

[feng-tao]

@rdblue is it an issue good for n00b? if yes, I am interested to take this one :)

[rdblue]

@feng-tao, I think this could be a good first issue. Let us know if you need any help or context.

[feng-tao]

@rdblue , I am still reading the code base. It would be great if you could guide me a little bit on the context or the related code path. Thanks a lot :)

[rdblue]

DataFile and Metrics are the classes that contain metrics and are good candidates for where truncation could happen. I think we would want truncation to be configurable using settings in TableProperties. Metrics are scraped from Parquet metadata in ParquetMetrics, which is called by ParquetWriter.

You might want to explore passing a truncate length option to ParquetWriter. The writer would pass it to ParquetMetrics to truncate values right away. The setting would come from the table when creating a writer. For that, I think you'd update the write builder in Parquet.

[feng-tao]

thanks @rdblue , will take a look