chunkers: only use the direct read path for known regular files - #10065
Merged
ThomasWaldmann merged 1 commit intoAug 9, 2026
Merged
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10065 +/- ##
=======================================
Coverage 86.75% 86.76%
=======================================
Files 98 98
Lines 17098 17100 +2
Branches 2590 2589 -1
=======================================
+ Hits 14834 14837 +3
Misses 1570 1570
+ Partials 694 693 -1 ☔ View full report in Codecov by Harness. |
The direct read path (big readv directly into the caller's buffer) was seen blocking forever on NetBSD 10 when reading a FIFO via create --read-special (test_create_read_special_symlink hung NetBSD CI deterministically, at the same spot in two runs), while the buffered block reader path's 1 MiB os.read() calls work fine there. Restrict the direct path to known regular files: FileReader only goes direct for S_ISREG modes. chunkify() accepts the file's os.stat_result if the caller has it anyway (process_file does, so the create hot path adds no stat syscall); otherwise FileReader fstats fh / fd.fileno() itself, staying on the buffered path for objects without an OS-level fd. Special files (--read-special) route onto the buffered path via their stat mode, exactly as before this branch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ThomasWaldmann
force-pushed
the
reader-direct-regular-only
branch
from
August 9, 2026 16:50
130232d to
800a4b8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the deterministic NetBSD CI hang introduced with PR #10060's direct read path.
The hang:
test_create_read_special_symlink(FIFO viacreate --read-special) hung NetBSD 10.1 forever, at the same spot in both attempts of the PR #10060 CI run — one pytest-xdist worker stuck in the test from minute 2, the suite stalled at 88%. The earlier branch push without the direct-read commit passed NetBSD in 39 min, and FreeBSD/OpenBSD/OmniOS/Linux pass with it, so it bisects cleanly to the direct path reading a FIFO. Special files are opened blocking, and NetBSD was seen never returning from the direct path's bigreadv()on the FIFO, while the buffered block reader's 1 MiBos.read()calls have always worked there.The fix: only known regular files take the direct path.
FileReaderrequiresS_ISREGfordirect.chunkify()(ChunkerBase, ChunkerFixed, ChunkerFailing + all.pyistubs) accepts an optionalst(os.stat_result) for callers that already have one.process_file()passes the stat result it already has — the create hot path adds no stat syscall. Regular files keep the zero-copy direct path;--read-specialFIFOs/devices route onto the buffered path via their stat mode.FileReaderfstatsfh/fd.fileno()itself (one cheap syscall per file). Objects without an OS-level fd (BytesIO, recreate/transfer chunk-iterator wrappers) stay on the buffered path, as before fixed chunker: read directly into per-chunk buffers #10060.stgiven (data + EOF verified); direct path via FileReader's own fstat (fh and file object); buffered for file-likes without an OS fd; direct refused for a FIFO'sst.Local runs: chunkers suite 114 passed, create_cmd + benchmark tests 72 passed, mypy unchanged (same 12 pre-existing unrelated errors).
Note: with PR #10064's
PYTEST_TIMEOUT=300, a hang like this would fail in 5 minutes with a stack dump instead of stalling CI for hours — the two PRs complement each other.🤖 Generated with Claude Code