Hardwood Reaches Beta: S3, Predicate Push-Down, CLI, and More - Gunnar Morling

[giscus[bot]]

Hardwood Reaches Beta: S3, Predicate Push-Down, CLI, and More - Gunnar Morling

I am pleased to announce the release of Hardwood 1.0.0.Beta1!
Hardwood is a new parser for Apache Parquet, optimized for minimal dependencies and great performance. Since the project’s initial …

https://www.morling.dev/blog/hardwood-reaches-beta-s3-predicate-push-down-cli/

[tcerda95]

Nice love the S3 and Avro support. Great job, Hardwood definitely fills a gap in the Java ecosystem for ergonomic and efficient Parquet parsing. Curious will there be support for:

1. Parsing files directly out of an S3-presigned URL? I believe these work slightly different than the S3 API but things like byte-range fetching should be possible there.
2. Parsing directly into a strongly-typed Avro SpecificRecord?

Currently we have a Flink pipeline processing over 100M rows per second parsed out of Parquet files provided by S3 Pre-signed urls. We use the java-parquet avro parser to parse them directly into Specific Records. We download the presigned URL completely into memory; the files are at most 10MB. Using Specific Record instead of Generic Record yielded great performance boosts to the downstream grouping we make as hashcodes and equals get JIT optimised.

Anyways, thank you for this work!

[gunnarmorling]

Hey @tcerda95, thanks a lot for commenting and the nice feedback! As for your questions, both are good points. I have logged hardwood-hq/hardwood#558 and hardwood-hq/hardwood#559 for tracking them. We should be able to have those for 1.1 (due sometime this summer). Any contributions welcome :)