[C++][Parquet] Allow Truncate min-max Statistics

[mapleFU]

Describe the enhancement requested

Currently, in parquet c++, if the min-max is string / binary:

1. During writing values, min-max will be collect
2. When building statistics, there is apply truncate, which will discard min-max if they're longer than expected

Can we use a "truncate" for this:

1. For BINARY
2. If it's minimum, just truncate to length is ok
3. if it's maximum:
   1. if the truncated binary would be 0xFF 0xFF ... 0xFF, we cannot truncate it
   2. Otherwise, get the "next" valid truncated binary
4. For String
   1. If it's minimum, just truncate and to a valid utf8 is ok
   2. If it's maximum, first truncate to a valid utf8, then try to advance it.

References:

1. PARQUET-1214: Column indexes: Truncate min/max values parquet-mr#481
2. Truncate Min/Max values in the Column Index arrow-rs#4389

Component(s)

C++, Parquet

[westonpace]

Does truncate discard? Or does it write the truncated value and mark that it is truncated?

For example, if I know the minimum value is "Hello W..." then I can reject the row group if there is a filter "value == 'blue'" even if I don't know the complete minimum value.

[mapleFU]

@westonpace Truncate is just Prefix truncate. Parquet-2.10 has released this:

1. Truncate means min might less than the exact mean, max might greater than exact maximum
2. Truncated value must be a valid value for this type