Push ChunkReader
into SerializedPageReader
#2463
Labels
enhancement
Any new improvement worthy of a entry in the changelog
parquet
Changes to the parquet crate
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Unlike
SerializedFileReader
, orSerializedRowGroupReader
,SerializedPageReader
is created with aT: Read
instead ofR: ChunkReader
.In order to preserve this when creating the async reader, I created a separate
InMemoryColumnChunkReader
.Similarly when adding page skipping support in #2044 a
VecDeque<T>
was passed into the constructor, to avoid changing the signature.Whilst working on #2460 we find ourselves in the same position where the lack of a ChunkReader complicates matters
Describe the solution you'd like
I think we have reached a point where introducing a breaking change to push the
ChunkReader
intoSerializedPageReader
is justified, as it will allow unifyingInMemoryColumnChunkReader
withSerializedPageReader
and eliminating a load of additional complexityDescribe alternatives you've considered
Additional context
FYI @Ted-Jiang @thinkharderdev
The text was updated successfully, but these errors were encountered: