Skip to content

v0.9.0

Latest

Choose a tag to compare

@mathieuouillon mathieuouillon released this 31 Jul 18:44

Install: pip install oxihipo==0.9.0

Added

  • Compression is a (codec, layout) pair, not a flat list of six named
    combinations: Codec::{None,Lz4,Lz4Hc,Gzip,Zstd} x
    Layout::{PerChunk,PerBank,PerColumn}. All 15 pairs have a wire tag and
    round-trip. The six historical names survive as associated constants, so
    Compression::Lz4PerColumn is still valid source at all 229 call sites and
    still means exactly what it meant (Lz4Hc x PerColumn).

  • Zstandard, levels 1-6 via Compression::with_zstd_level. The level is a
    writer-side knob and never reaches the wire — one tag decodes them all,
    unlike LZ4/LZ4-HC which burn two. On a 248 MB real CLAS12 file,
    Zstd x PerColumn is 2.22x smaller and scans in 20.9 ms, against
    2.03x/28.2 ms for Lz4Hc x PerColumn and 2.32x/21.9 ms for
    Gzip x PerColumn — but writes in 0.69 s where gzip takes 2.52 s and LZ4-HC
    7.66 s.

    Tags 4 and 5, left poisoned when Lz4Chunked and Lz4ByBank v1 were
    removed in 0.x, are reused for Zstd. That is safe specifically because a
    zstd frame begins with the magic 0xFD2FB528: a stale file carrying one of
    those tags fails the frame check rather than decoding as something
    plausible. Tag 15 is the only one left unassigned.

    Only the six pairs that predate the matrix are readable by hipo-cpp and
    hipo-java; the other nine are oxihipo extensions those readers reject as
    an unknown tag. The split-record directory stays LZ4 for every layout so
    tags 6 and 7 remain byte-compatible.

  • Python: compression= takes the pair too"<codec>+<layout>", e.g.
    "zstd+percolumn" or "zstd6+perbank". A bare codec means perchunk, and
    the six older names still work and still mean the same thing
    ("lz4percolumn" is lz4hc+percolumn, not lz4+percolumn — the split
    codecs were always high-compression). An unknown name lists what is valid;
    a zstd level outside 1-6 is an error rather than a silent clamp.

  • Compression prints as lz4hc+perbank / zstd3+percolumn — the same
    grammar the Python binding accepts, so format!("{c:?}") round-trips back
    through it. The derived Debug printed a struct dump containing {, } and
    :, which is noise in a log and illegal in a Windows filename.

Fixed

  • read_columns handed back buffers carrying their growth slack. Assembly
    grew each column with extend_from_slice per record, so the result kept the
    last doubling's headroom — measured at 1.664x the payload on a real
    CLAS12 DST, retained for as long as the caller held it. That is the lifetime
    of the NumPy array for the Python binding, where into_pyarray moves the
    Vec across with the slack included. Now 1.000x.

    Every chunk is already in hand when the buffers are assembled, so the final
    length is known before any appending and the buffers are sized exactly up
    front. That is faster than the naive fix as well as smaller: a first version
    grew and then shrink_to_fit, which left best-of-15 unchanged but median ~5%
    worse on a 9-column read (the end-of-assembly realloc). Sizing up front
    removes both the doubling and the realloc — measured against the pre-session
    baseline, best-of 1.283 -> 1.229 s and median 1.606 -> 1.551 s.