Is ZSTD compression level 22 used for producing the Parquet files?

[Mark Litwintschik (marklit)]

I'm not sure if there is a public repo of the workflows used to produce the final parquet files. Are you using level 22 compression on the final Parquet files that are published to S3?

Level 3 is the default in the ZSTD C lib most tools use but overriding it to level 22 can turn 88 MB of data into 70 MB.

[Jacob Wasserman (jwass)]

For some historical reasons, that code isn't public but is something we're looking to fix.
The parquet files are written by Spark with code that looks basically like this:

df.write.format("geoparquet")
        .option("geoparquet.version", "1.1.0")
        .option("geoparquet.crs", "")
        .option("geoparquet.covering.geometry", "bbox")
        .option("compression", "zstd")
        .option("parquet.block.size", 16 * 1024 * 1024)
        .save(path)

So it's using Spark's defaults for zstd... from a quick search the zstd compression level used to write out the parquet might not be configurable in Spark.

[Mark Litwintschik (marklit)]

February's release is 474 GB versus January's 471 GB. I can only see another ~5 GB of US addresses added to February's release so I suspect everything is being compressed at level 3 and not 22. Are there plans for level 22 compression in March's release?

[Jacob Wasserman (jwass)]

Thanks for the feedback, Mark Litwintschik (@marklit). My original response is still true. We're just using Spark/Sedona to write the dataset out which (as far as I can tell) does not allow us to adjust the zstd compression level so we're stuck with the default for now. If I'm wrong about that let me know.

I do think we may want to evaluate the additional compute/time cost of using a higher compression level before just going with the highest.

[Mark Litwintschik (marklit)]

Understood. The finished Parquet could be run through DuckDB to re-compress at level 22. I'm not sure what sort of CPUs you're running everything on but on my 14900K I can barely see the wall clock time difference between level 3 and 22.