v0.5.1
GeoPQ Workbench 0.5.1 stops remote reads paying for bytes nobody wanted.
Reading a remote file
Parquet reads a page header through a streaming read and takes the page
body through a bounded one. Every streaming read began with a fixed
256 kB window, so each column chunk cost 256 kB to consume about twenty
bytes of header. A 2 MB file cost 31 MB to read, and no amount of
row-group pruning could hide it.
The first window is now sized to what the caller actually asked for,
with a 4 kB floor, and doubles from there as before, so a consumer that
really does stream still reaches the ceiling in a handful of requests.
The last window fetched is also kept, so the bounded read that follows a
header is served from bytes already paid for instead of asking twice.
Measured on a viewport export of a sorted fixture, as a fraction of the
file read:
| read | |
|---|---|
| 0.5.0 | 1458% |
| adaptive window | 161% |
| plus the window cache | 45% |
This applies to every remote read, not only exports.
Exporting the current viewport
"Viewport only" read the whole file and filtered afterwards. On a remote
source that is the entire download for a handful of features. It now
prunes row groups from the same metadata boxes the viewer prunes with,
before reading a byte.
Panning a remote layer
Refinement resolves each intersecting row group against its bbox column,
one round trip apiece, and those ran one after another: twenty row
groups put several seconds between the camera settling and the first
byte of geometry being requested. They now resolve in parallel. The
budget still walks them in order, so which groups make the cut does not
depend on who answered first.
Install
Prebuilt binaries for macOS (Apple Silicon and Intel), Linux x86_64 and
Windows x86_64 are attached, each with a SHA-256 checksum. macOS builds
are signed and notarized.