Install: pip install oxihipo==0.9.0
Added
-
Compressionis a (codec, layout) pair, not a flat list of six named
combinations:Codec::{None,Lz4,Lz4Hc,Gzip,Zstd}x
Layout::{PerChunk,PerBank,PerColumn}. All 15 pairs have a wire tag and
round-trip. The six historical names survive as associated constants, so
Compression::Lz4PerColumnis still valid source at all 229 call sites and
still means exactly what it meant (Lz4HcxPerColumn). -
Zstandard, levels 1-6 via
Compression::with_zstd_level. The level is a
writer-side knob and never reaches the wire — one tag decodes them all,
unlike LZ4/LZ4-HC which burn two. On a 248 MB real CLAS12 file,
Zstd x PerColumnis 2.22x smaller and scans in 20.9 ms, against
2.03x/28.2 ms forLz4Hc x PerColumnand 2.32x/21.9 ms for
Gzip x PerColumn— but writes in 0.69 s where gzip takes 2.52 s and LZ4-HC
7.66 s.Tags 4 and 5, left poisoned when
Lz4ChunkedandLz4ByBankv1 were
removed in 0.x, are reused for Zstd. That is safe specifically because a
zstd frame begins with the magic0xFD2FB528: a stale file carrying one of
those tags fails the frame check rather than decoding as something
plausible. Tag 15 is the only one left unassigned.Only the six pairs that predate the matrix are readable by
hipo-cppand
hipo-java; the other nine are oxihipo extensions those readers reject as
an unknown tag. The split-record directory stays LZ4 for every layout so
tags 6 and 7 remain byte-compatible. -
Python:
compression=takes the pair too —"<codec>+<layout>", e.g.
"zstd+percolumn"or"zstd6+perbank". A bare codec meansperchunk, and
the six older names still work and still mean the same thing
("lz4percolumn"islz4hc+percolumn, notlz4+percolumn— the split
codecs were always high-compression). An unknown name lists what is valid;
a zstd level outside 1-6 is an error rather than a silent clamp. -
Compressionprints aslz4hc+perbank/zstd3+percolumn— the same
grammar the Python binding accepts, soformat!("{c:?}")round-trips back
through it. The derivedDebugprinted a struct dump containing{,}and
:, which is noise in a log and illegal in a Windows filename.
Fixed
-
read_columnshanded back buffers carrying their growth slack. Assembly
grew each column withextend_from_sliceper record, so the result kept the
last doubling's headroom — measured at 1.664x the payload on a real
CLAS12 DST, retained for as long as the caller held it. That is the lifetime
of the NumPy array for the Python binding, whereinto_pyarraymoves the
Vecacross with the slack included. Now 1.000x.Every chunk is already in hand when the buffers are assembled, so the final
length is known before any appending and the buffers are sized exactly up
front. That is faster than the naive fix as well as smaller: a first version
grew and thenshrink_to_fit, which left best-of-15 unchanged but median ~5%
worse on a 9-column read (the end-of-assembly realloc). Sizing up front
removes both the doubling and the realloc — measured against the pre-session
baseline, best-of 1.283 -> 1.229 s and median 1.606 -> 1.551 s.