Rework Thrift Encoding / Decoding of Parquet Metadata

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Currently we make use of the Apache thrift crate for encoding and decoding Parquet metadata. This has served us well, but there are a few problems:

• The performance is sub-optimal - How to improve Parquet reading performances? datafusion#7737 (reply in thread)
• It does not appear to be maintained anymore - https://issues.apache.org/jira/browse/THRIFT-5640
  • https://docs.rs/databend-thrift/latest/databend_thrift/
  • https://docs.rs/tent-thrift/latest/tent_thrift/
• It can panic - Panic in parquet::file::footer::decode_metadata #3279

Describe the solution you'd like

Not sure yet. Parquet requires a very limited portion of the thrift specification, only making use of the compact encoding, and none of the protocol functionality, and so one option might be to just support this subset with a custom implementation.

Describe alternatives you've considered

Additional context

[mapleFU]

Is that means transfering to a new rust thrift lib? Or using other way to workaround? Since parquet 2.10 might released later, I think the thrift idl might change in the future ( though last release is public severo years ago...)

[tustvold]

I think my hope would be that we can get by with the existing thrift-compiler, perhaps with some sed, but then swap out the protocol implementation. I'm playing with this now

[mapleFU]

I think this can benefit some partial thrift decoding (e.g. user only want schema, but the whole FileMetadata thrift need to be deserialized)

However I don't know the performance and maintainance of hand-written parsing🤔 Looking forward the great work here

[tustvold]

label_issue.py automatically added labels {'parquet'} from #4892