Skip to content

Prefetch parquet data pages #647

@samarthjain

Description

@samarthjain

While doing a hot spot analysis of the benchmark runs we have for vectorized read path, reading and decompressing of the parquet data pages stood out as one of the major hot spots.

We can significantly improve performance if we can take out this read/decompress from the hot code path by prefetching the next data page asynchronously. Preliminary benchmark runs showed up to 60% performance gains when using prefetching.
An initial implementation is here - https://github.com/samarthjain/incubator-iceberg/tree/samarth_prefetch_vectorized

This is going to need some Parquet level changes as the underlying native compressors used are not thread safe. apache/parquet-java#671 adds support for airlift based compressors which are written in pure Java and don't use any JNI resources.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions