Changes from 3.3.0 to 3.3.1
This release is about making reads from on-disk frames faster.
Reads from on-disk frames got noticeably faster. The default filesystem I/O
backend now uses positioned reads and writes (pread/pwrite, ReadFile/
WriteFile with an explicit offset on Windows) instead of seek + stdio, and a
frame keeps a single read handle open instead of opening and closing the file
several times per chunk fetch. Scattered small reads out of a cframe are about
3x faster in our microbenchmarks; the gain grows with how many small reads a
workload does. User-registered I/O backends are unaffected: they keep their
one-handle-per-reader contract.
Concurrent readers gain the most, since the per-access open/close this removes
was paid by every process and contended in the kernel. Eight processes each
doing 300 random slice reads over the same 269 MB frame went from 0.52 s to
0.36 s of wall time, and from 3.85 s to 2.58 s of CPU (Apple M4 Pro): about
1.4x, against 13% for a single reader. The advantage of opening such files with
mmap_mode="r" narrows accordingly -- not because memory mapping got slower,
but because the regular path caught up.
Note that an open on-disk schunk now holds one file descriptor for as long as it
stays open, where before descriptors were only taken for the duration of each
read. To keep that from exhausting the process's descriptor budget, the number
of cached handles is capped at min(96, RLIMIT_NOFILE/16) -- schunks past the cap
fall back to the previous open-per-read behaviour. Set the
BLOSC_MAX_CACHED_READERS environment variable to choose a different cap, or
to 0 to switch the cache off entirely. Handle caching is currently POSIX-only:
on Windows the C runtime opens files without FILE_SHARE_DELETE, so a cached
handle would make unlinking or renaming an open frame file fail.
As a side effect of dropping fseek(), reads and writes past 2 GB no longer
fail on 32-bit builds whose off_t is 64-bit wide.
There are no API or format changes in this release.