-
Notifications
You must be signed in to change notification settings - Fork 3k
Description
While doing a hot spot analysis of the benchmark runs we have for vectorized read path, reading and decompressing of the parquet data pages stood out as one of the major hot spots.
We can significantly improve performance if we can take out this read/decompress from the hot code path by prefetching the next data page asynchronously. Preliminary benchmark runs showed up to 60% performance gains when using prefetching.
An initial implementation is here - https://github.com/samarthjain/incubator-iceberg/tree/samarth_prefetch_vectorized
This is going to need some Parquet level changes as the underlying native compressors used are not thread safe. apache/parquet-java#671 adds support for airlift based compressors which are written in pure Java and don't use any JNI resources.