Skip to content

Core: Coalesce nearby blob reads in PuffinReader.readAll - #17418

Open
vishnuprakaz wants to merge 1 commit into
apache:mainfrom
vishnuprakaz:puffin-readall-coalesce
Open

Core: Coalesce nearby blob reads in PuffinReader.readAll#17418
vishnuprakaz wants to merge 1 commit into
apache:mainfrom
vishnuprakaz:puffin-readall-coalesce

Conversation

@vishnuprakaz

Copy link
Copy Markdown
Contributor

PuffinReader.readAll currently issues one seek + read per blob. This resolves
the long-standing TODO from #4537:

Contiguous (or overlapping) blobs are now read in a single request and handed back
as no-copy ByteBuffer views over that region. Since Puffin stats/index blobs are
written contiguously, a readAll over many blobs collapses from N reads to one
the win matters most on object stores, where each read is a round trip.

Approach

This coalesces contiguous blob reads: after sorting by offset, adjacent/overlapping blobs are merged into a single read, and a run is split once it would exceed a max read size.

Also RangeReadable.readVectored(...) was considered as an alternative, but its default implementation doesn't coalesce on S3/Hadoop (only some FileIOs, e.g. GCS analytics, optimize it) and eagerly buffers every range.

Coalescing contiguous reads captures the benefit for real Puffin layouts without reading bytes the caller didn't request, and reads stay lazy (a region is fetched only when iterated).

The max read size is capped at 8 MiB the same value as MANIFEST_TARGET_SIZE, since Puffin blobs are metadata. The cap bounds the extra memory coalescing holds compared to the original per-blob reads (a single blob larger than the cap is still read on its own).

Are these changes tested?

  • Existing testReadMetricDataUncompressed / testReadMetricDataCompressedZstd cover
    readAll end-to-end for both codecs (contiguous blobs, nonzero-offset slice).
  • New end-to-end readAll test with blobs far apart, asserting both decode across two
    separate reads.
  • New unit tests for the grouping logic: contiguous/overlap merge, gap split, region-size
    split, and an oversized single blob kept in its own read.

Are there any user-facing changes?

No

@github-actions github-actions Bot added the core label Jul 29, 2026
Resolve the TODO from apache#4537 by reading contiguous blobs in a single
  request, handed back as no-copy ByteBuffer views. A contiguous run is
  split into bounded reads (capped at MANIFEST_TARGET_SIZE, 8 MiB) so a
  large blob section never allocates one oversized buffer.
@vishnuprakaz
vishnuprakaz force-pushed the puffin-readall-coalesce branch from 8f46a60 to b663983 Compare July 29, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant