Investigate parquet performance improvements

[hermanschaaf]

Recent benchmarks on v2 show that performance of the Parquet writer has degraded when compared to v1 (which used a different Parquet library). We should investigate and see how performance can be improved--this may need to happen upstream in Arrow.

NB: this requires a lot of investigation work, so needs to be approached if there's nothing with higher priority.

[candiduslynx]

Closing for now, as the following is currently true:

1. The lib (xitongsys/parquet-go) we used previously is no longer actively supported, and it doesn't support all required Apache Arrow types
2. Another lib (segmentio/parquet-go) that's out there for Parquet in Go requires concrete structs, so creating schema from arrow schema & reconstructing data into structs might be a rabbit hole on its own

[hermanschaaf]

@candiduslynx I don't agree that's really a reason to close this? Yes, those libraries are different, but this issue is not saying we should migrate away from Arrow Parquet; it's saying we observed a ~10x performance regression in direct benchmarks after migrating to the Arrow parquet library, compared to xitongsys/parquet-go, and we should look into why there is such a large performance gap and find out if there is anything that can be done to improve it in the upstream Arrow Parquet library.

[candiduslynx]

@hermanschaaf I do agree that we might want to track this in more detail, but I propose doing this only after the reports from the users, as the rabbit hole of tracking why the write is slower will require a bit of hacking.

[disq]

This PR has some findings about segmentio's parquet package (it wasn't looking great at the time)

[hermanschaaf]

@candiduslynx Let's leave it open if you don't mind; we don't have to do it now, but I think it's worth doing a comparative benchmark at some point that we can share with the upstream Arrow maintainers for feedback. We kind-of have one already in https://github.com/cloudquery/filetypes/blob/main/parquet/write_read_test.go#L70-L98, which was previously working with xitongsys/parquet-go; for a 1:1 comparison we can maybe use only very basic types like int and string to keep it simple. If there's still a difference we can report it upstream