Skip to content

[python] Avoid an extra file-size I/O for Blob reads - #9125

Merged
JingsongLi merged 4 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/blob-reader-file-size
Aug 9, 2026
Merged

[python] Avoid an extra file-size I/O for Blob reads#9125
JingsongLi merged 4 commits into
apache:masterfrom
XiaoHongbo-Hope:codex/blob-reader-file-size

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Purpose

Avoid the extra file-size lookup performed when SplitRead creates a FormatBlobReader. DataFileMeta already contains the file size, so object-store reads do not need an additional metadata request before reading the Blob footer and index.

Changes

  • Add an optional file_size argument to FormatBlobReader.
  • Reuse DataFileMeta.file_size in both normal and Data Evolution raw Blob read paths.
  • Preserve the existing lookup when FormatBlobReader is constructed without a valid file size.

Tests

  • Provided file size avoids FileIO.get_file_size.
  • Missing or zero file size falls back to one lookup.
  • Both SplitRead Blob paths forward DataFileMeta.file_size.
  • Blob tests: 186 passed, 1 unrelated Ray/pandas compatibility test deselected, 44 subtests passed.
  • Deferred Blob tests: 15 passed.
  • Flake8 and git diff --check passed.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Avoid redundant file size lookup for Blob reads [python] Avoid an extra file-size I/O for Blob reads Aug 9, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 9, 2026 03:44
@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi
JingsongLi merged commit 3fdcabb into apache:master Aug 9, 2026
15 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants