Releases: mathieuouillon/oxihipo
Release list
v0.9.0
Install: pip install oxihipo==0.9.0
Added
-
Compressionis a (codec, layout) pair, not a flat list of six named
combinations:Codec::{None,Lz4,Lz4Hc,Gzip,Zstd}x
Layout::{PerChunk,PerBank,PerColumn}. All 15 pairs have a wire tag and
round-trip. The six historical names survive as associated constants, so
Compression::Lz4PerColumnis still valid source at all 229 call sites and
still means exactly what it meant (Lz4HcxPerColumn). -
Zstandard, levels 1-6 via
Compression::with_zstd_level. The level is a
writer-side knob and never reaches the wire — one tag decodes them all,
unlike LZ4/LZ4-HC which burn two. On a 248 MB real CLAS12 file,
Zstd x PerColumnis 2.22x smaller and scans in 20.9 ms, against
2.03x/28.2 ms forLz4Hc x PerColumnand 2.32x/21.9 ms for
Gzip x PerColumn— but writes in 0.69 s where gzip takes 2.52 s and LZ4-HC
7.66 s.Tags 4 and 5, left poisoned when
Lz4ChunkedandLz4ByBankv1 were
removed in 0.x, are reused for Zstd. That is safe specifically because a
zstd frame begins with the magic0xFD2FB528: a stale file carrying one of
those tags fails the frame check rather than decoding as something
plausible. Tag 15 is the only one left unassigned.Only the six pairs that predate the matrix are readable by
hipo-cppand
hipo-java; the other nine are oxihipo extensions those readers reject as
an unknown tag. The split-record directory stays LZ4 for every layout so
tags 6 and 7 remain byte-compatible. -
Python:
compression=takes the pair too —"<codec>+<layout>", e.g.
"zstd+percolumn"or"zstd6+perbank". A bare codec meansperchunk, and
the six older names still work and still mean the same thing
("lz4percolumn"islz4hc+percolumn, notlz4+percolumn— the split
codecs were always high-compression). An unknown name lists what is valid;
a zstd level outside 1-6 is an error rather than a silent clamp. -
Compressionprints aslz4hc+perbank/zstd3+percolumn— the same
grammar the Python binding accepts, soformat!("{c:?}")round-trips back
through it. The derivedDebugprinted a struct dump containing{,}and
:, which is noise in a log and illegal in a Windows filename.
Fixed
-
read_columnshanded back buffers carrying their growth slack. Assembly
grew each column withextend_from_sliceper record, so the result kept the
last doubling's headroom — measured at 1.664x the payload on a real
CLAS12 DST, retained for as long as the caller held it. That is the lifetime
of the NumPy array for the Python binding, whereinto_pyarraymoves the
Vecacross with the slack included. Now 1.000x.Every chunk is already in hand when the buffers are assembled, so the final
length is known before any appending and the buffers are sized exactly up
front. That is faster than the naive fix as well as smaller: a first version
grew and thenshrink_to_fit, which left best-of-15 unchanged but median ~5%
worse on a 9-column read (the end-of-assembly realloc). Sizing up front
removes both the doubling and the realloc — measured against the pre-session
baseline, best-of 1.283 -> 1.229 s and median 1.606 -> 1.551 s.
v0.7.1
Install: pip install oxihipo==0.7.1
A single-purpose release: undo 0.7.0's split-codec format-version bump,
which broke the C++ and Java implementations of those codecs. The composite fix
0.7.0 shipped is kept in full.
Fixed
-
Lz4PerBankandLz4PerColumnfiles written by 0.7.0 are unreadable by
hipo-cppandhipo-java. 0.7.0 raised the on-disk
ext_format_versionfrom 2 to 3 (by-bank) and 1 to 2 (by-column) when it
appended the compositeheader_sizetable. Those version numbers turn out to
be a cross-implementation contract: the
hipo-cppand
hipo-java
feature/bybank-bycolumn-compressionbranches document and implement exactly
versions 2 and 1. Measured on a JLab farm node against both:file oxihipo hipo-javahipo-cppby-bank / by-column @ 0.6.0 ✅ ✅ ✅ by-bank / by-column @ 0.7.0 ✅ ❌ failed to decode ByBank record section❌ segfault by-bank / by-column @ 0.7.1 ✅ ✅ ✅ All three now produce byte-identical checksums on the same files, array
(T#N) columns included.The bump was never necessary. The
header_sizetable is appended after
every other directory table, so a reader that predates it never looks that
far — proven by patching only the version byte back on a 0.7.0 file, after
which both other implementations read it perfectly. The library now detects
the table by directory length instead of by the version byte, which is
what lets the version stay fixed while the format grows.0.7.1 still reads the version-3/2 files 0.7.0 wrote, so nothing already
written is lost. Rewrite them with 0.7.1 if you need to share them.
Added
tests/composite_codecs.rsasserts the on-diskext_format_versionis 2
(by-bank) and 1 (by-column), reading the byte back off the file. The contract
is with other codebases, so it needed a test that fails if it drifts again.
Documentation
- The claim that the split codecs are readable only by this library was wrong
and is corrected everywhere it appeared. The released C++hipo4and Java
readers do not know wire tags 6 and 7, but the branches above do.
v0.7.0
Install: pip install oxihipo==0.7.0
Breaking
TagRegistry::insertandTagRegistry::from_namesreturnResult. A tag
name that cannot survive the on-diskname=bittext form is now refused
instead of silently mangled (below).WriterBuilder::tag_nameskeeps its
signature and surfaces the error frombuild.
Changed
-
Split-codec record format bumped:
Lz4PerBank2 → 3,Lz4PerColumn1 → 2.
The directory gained aB × u8compositeheader_sizetable, appended after
the existing tables so every offset before it is byte-for-byte unchanged. One
parser reads both versions, with the tail defaulting to 0 — which is exactly
what the old versions meant. Any other version is refused rather than
misread — measured: a 0.6.0 reader opens a 0.7.0 split-codec file and reports
its event count, then fails the firstevents()item withLz4PerBank: unsupported extension-format version. It never returns wrong data.Compatibility was checked in both directions, all four codecs, by building
v0.6.0 and HEAD side by side and cross-reading:NoneLz4Lz4PerBankLz4PerColumnHEAD reads 0.6.0 ✅ ✅ ✅ ✅ 0.6.0 reads HEAD ✅ ✅ clean error clean error The blob codecs are not merely compatible, they are byte-identical: the
same inputs written by 0.6.0 and by HEAD hash to the same SHA-256. Since those
are the only codecs C++ hipo4 and Java can read, interoperability with them
cannot have changed. Neither split codec was ever readable outside this
library.
Fixed
-
Composite banks lost their format string under
Lz4PerBankand
Lz4PerColumn. A bank's structure length word packs the data size into its
low 24 bits and the compositeheader_size— the format string's length —
into its top byte. Both split codecs take a record apart and store bank
payloads separately, discarding the structure headers, and rebuilt that byte
as zero. A composite bank therefore came back looking like an ordinary one:
header_sizeread as 0 andcomposite()returnedNone. The two blob
codecs,NoneandLz4, were unaffected, so the same file written two ways
disagreed about whether its banks were composite. Fixed by the format change
above, plus carrying the byte through the by-bank structure iterator and both
event synthesisers.Validated on real data as well as fixtures: 2,000 events of an 8.5 GB CLAS12
DST (71 distinct banks) and of a simulation file (106 banks) were rewritten
through all four codecs and compared bank for bank — everyheader_sizeand
payload identical to the source. Neither file carries a composite bank, so
this bug was not corrupting CLAS12 reconstruction output; it was corrupting
composites, which the format allows anywhere. -
OwnedEvent::compositereturnedNoneon every split-codec event. It
delegated toEventCtx::composite, which needs original structure bytes and
documents itself as returningNonefor by-bank backends — advising callers
to up-convert toOwnedEvent, the very thing that did not work. It now goes
through the synthesised event blob, which (since the fix above) carries the
compositeheader_size. -
Lz4PerColumnstored a composite bank column-major when a schema happened
to describe its group/item and its payload divided evenly into rows — a layout
that assumes fixed-width rows the bank does not have. Composites are now kept
opaque regardless. Belt-and-braces: the split is a permutation the synthesiser
inverts, so no round-trip through this library could observe it.
Added
tests/composite_codecs.rs— composite banks across all four codecs, checked
both for the survivingheader_sizeand for decoded field values. Two
composites with different format-string padding, since only the 8-byte-padded
one divides evenly into rows and so reaches theLz4PerColumnguard.Lz4PerBankadded to the read benchmark's format list. It reaches the reader
through the by-bank structure iterator rather than the per-column synthesiser,
so benching onlyLz4PerColumnleft that scan path unmeasured.tests/salvage.rs— four tests for the sequential scan: the trailer is not
indexed as data, salvage resynchronises past a damaged header, an impossible
event_countis rejected, and an intact file still indexes exactly on every
codec.tests/event_tag.rs— tag names that cannot round-trip are refused, by the
registry and by the writer; ordinary names still survive a real write and read.tests/no_alloc.rs— the per-event allocation contract is now checked on
every codec, by comparing two files with the same record count and 4× the
events rather than against a fixed budget.
Performance
-
Lz4PerBankandLz4PerColumnsequential reads are ~25 % faster.
Removing the per-eventArcallocation below: 959 µs → 724 µs and 998 µs →
713 µs on the benchmark fixture. The blob codecs are unchanged — two
interleaved A/B rounds disagreed on their sign (+4 % then −2.6 %), which is
the machine's noise floor, andOwnedEventis the same 56 bytes it was. -
A tag name containing
=, a line break, or edge whitespace was written and
silently read back as something else. The registry is stored asname=bit
lines and the reader splits on the first=, splits lines on\n, and trims
each name. Of five names written, only one survived unchanged:written read back plainplainhas=equalsdropped has\nnewlinenewline␣␣padded␣␣padded`` (empty) dropped Two came back under a different name, so
mask("has\nnewline")returned
Noneand the flag quietly stopped matching. Such names are now refused when
inserted, andWriter::buildfails rather than writing one. -
The scan indexed the trailer as a data record. A trailer is an ordinary
one-event record carrying thefile::indexbank — no header bit sets it
apart — and the fallback scan walked straight into it. The comment said the
normal path never met this; it does, whenever a trailer exists but does not
parse, which is exactly when the fallback runs. A 12-event file with a
corrupted trailer index reported 13 events. -
One damaged record header cost the whole file,
open_salvageincluded.
The scan propagated the parse error instead of resynchronising, so the salvage
path — whose entire purpose is recovering from this — recovered nothing.
Salvage now finds the next record and continues: on a 12-event file with one
destroyed header it returns the 8 events on either side of the damage. The
normal path still reports the corruption, deliberately. -
event_countwas taken on trust. A corrupt header propagated straight
intoChain::event_count(): flipping one record's count to 1,000,000 made a
12-event file report 1,000,009, and the firstevents()item was an error.
The record's index array bounds the real count at four bytes per event, and
that is a header field, so no decompression is needed. Checked against real
data before relying on it: across 1,951 records of an 8.5 GB CLAS12 DST, a
simulation file, and C++ hipo4's own golden file,index_array_lengthis
exactlyevent_count * 4. -
Every split-codec event heap-allocated a cell it usually never filled.
OwnedEventheld the lazy whole-event blob asArc<OnceLock<Vec<u8>>>, so
constructing one allocated even when nothing ever asked for the blob: 852
allocations for 800 events created and dropped untouched. TheArcmoved
inside the cell (cell::OnceCell<Arc<Vec<u8>>>), which allocates only on
first use and keepsOwnedEventat 56 bytes.Chain::eventsdocuments "no
per-event allocation"; that is now true on every codec, and tested.
Documentation
Chain::events' memory contract now says what the split codecs actually cost
(more per record, still nothing per event) and notes that the whole-event
views synthesise a blob on first use.- The split codecs sort banks by
(group, item)— load-bearing, since the
reader binary-searches that table — so they do not preserve the order
banks were added in.structures()yields ascending(group, item)there and
write order on the blob codecs. Nothing addresses a bank by position, so this
is an iteration-order difference rather than a data one; it is now documented
onCompressionand at both sort sites rather than left to be discovered.
Removed
tests/zz_b.rs— a diagnostic that printed recordbit_infowords and
asserted nothing, committed by accident in the record-header bit fix.
v0.6.0
Install: pip install oxihipo==0.6.0
A correctness release. Each entry is something that was silently wrong — the
library returned success and stored the wrong bytes, or aborted the process where
it should have returned an error — rather than something that failed visibly.
Breaking
BankBuilder::finishandEventBuilder::addreturnResult. Both are
public, so this is a semver break, taken deliberately: the alternative is an
API that silently corrupts data (below). Neither is used byhipo-tools, which
builds and passes its 204 tests against this release unchanged.
Fixed
-
A bank at or past 2^24 bytes was written truncated, with no error. The
structure length word carries the data size in its low 24 bits — the top byte
is the compositeheader_sizefield — andBankBuilder::finishwrote a full
u32into it.Writer::finishreturnedOkregardless, so the loss was
invisible until the file was read back. On a one-columnIntbank:rows written data bytes rows read back 4,194,303 16,777,212 4,194,303 4,194,304 16,777,216 0 5,000,000 20,000,000 805,696 At exactly 2^24 the size masks to zero and the overflowed
0x01re-reads as
header_size = 1, so the bank comes back looking composite. Every codec was
affected — this is the structure header, not the compression. The boundary is
nowHipoError::BankTooLarge. -
Bank::readwas documented "Infallible" and panicked in release builds.
Every check in it was adebug_assert, whileColumnHandle::placeholder()is
public and safe — and thebank_row!-generatedresolve_handleshands one out
for any column a runtime schema lacks. Reading through it fell into
schema.entries()[65535]and aborted with "index out of bounds: the len is 1
but the index is 65535", withdebug_assertions = false.The documentation was the defect: it now states the real contract with a
# Panicssection, and the bounds check is unconditional with a message that
names the mistake. Returning an empty column instead was rejected —readis
the bulk accessor and callers zip parallel columns, so a zero-length result
silently truncates the loop to no rows, trading a loud abort for quietly wrong
physics.read_handle_or_defaultremains the per-row path that does accept
placeholders. -
read_columnscould hand back buffers that contradicted their own
offsets.merge_chunksstates the contractColumnBuffersowes its caller —
offsets starting at 0 and non-decreasing, each column holding exactly
total_rows * inner_lenvalues — but only as adebug_assert. A corrupted
Lz4PerColumnrecord whose row counts and column payloads disagree produces
exactly that violation, so a release build returned buffers whose data
length did not match their offsets and slicing a row read the wrong values.
The invariant is now enforced and returnsHipoError::CorruptRecord.Found by the new sweep, and only in a debug build — the release runs it had
been checked against compile the assertion out, which is precisely why it had
survived. CI runs the debug profile and caught it on the first push. -
Iterating onto a zero-event record panicked.
EventIter::next_result
refilled the current record withifrather thanwhile.advance_record
resets the event cursor to 0, so the guard was tested against the record being
left and never against the one arrived at; landing on an empty record indexed
event_offsets[1]on a one-element table. A library may returnErrfor
damaged input, but a panic leaves its caller — a CLI, a Python binding, a batch
job — no way to handle the file at all.The empty record need not come from corruption:
Writer::flush_recordwill not
emit one, but the publicWriter::write_recordtakes a prebuilt record and
checks only that it is at least a header long.
Internal
- New
tests/mutation_sweep.rs. Mutates every byte of a real file, truncates
at every length, and writes hostile values into every header word, then drives
the whole public read path over each result asserting only that nothing panics.
It is what found the iterator bug — six single-byte mutations of a 2 KB file
reached it, each the low byte of a record's event count.corruption.rs
already had an empty-record test that could not have caught it: that test also
blanks the trailer to force the scan path, and the scan drops empty records
before iteration ever sees one.
v0.5.3
Install: pip install oxihipo==0.5.3
Added
-
Chain::for_each_range/for_each_ranges— stream one or several global
event ranges, reading only the records they touch. Reading part of a file
previously meantevent(idx)per index, which is why a per-record index could
not be exploited: a downstream cut that correctly skipped 85% of events came
out 4.5x slower through that path than a full scan.Measured on a 3 GB CLAS12 file, 21,506 events over 89 ranges (warm, best of 3):
-j 1-j 16all ranges, one call 0.24 s 0.11 s one call per range 0.62 s 0.80 s event(idx)per index0.61 s — So 5.9x against per-index reading at 16 threads and 2.6x at one. The
single-threaded margin is modest becauseeventcaches the record it last
inflated — contiguous access was already reasonable; what this adds is
parallelism across records.Pass every range in one call. Each call rebuilds the record task list and
pays a rayon dispatch, so looping spends its time on bookkeeping — at 16
threads that overhead makes the loop slower than one thread. A record
straddling a boundary is read once and its out-of-range events dropped, so
events_incounts what the ranges hold. Ranges may overlap and arrive
unsorted; indices are the same pre-filter space asread_columns(range). -
Chain::open_salvage— open a file whose 56-byte header is unusable, by
finding the records themselves. The header is bookkeeping (magic, version,
counts, where the dictionary and trailer are) and all of it is re-derivable,
because every record carries its own header and magic. A file missing its
first 56 bytes was not unreadable, only unopenable by a path that parses that
header first.Two things the scan has to handle, both found by testing rather than by
reading the format. The trailer looks like a data record — nothing in its
header says otherwise (measured:is_last_recordis 0 on both, and the
bit_infodifference is only padding), so a 120-event file came back with 121
events; it is now recognised by content, thefile::indexbank. And a
truncated tail was indexed and then unreadable, because the scan checked
that a record's header fits in the file but not the record — fixed below.What it cannot recover is the dictionary, which lives in the record right
after the header, so damage that took one usually took the other. The chain
then has an empty dictionary rather than a guess; the events are still there
and still copyable, but their banks have no names or column types.
Fixed
-
A truncated tail no longer defeats
open_salvage. The record scan checked
that a record's header fits in the file, not the record, so a killed
writer's half-written last record produced an index entry that opened fine and
then failed on read with "record extends past EOF".Salvage now stops there and keeps the intact prefix. The normal path is
unchanged and still raises, deliberately: truncation is genuine corruption,
and a reader that quietly returned a shorter file would give no way to tell.
Making the stop unconditional was the first attempt, and the binding's
test_truncated_file_raisescaught it — the difference matters only because
salvage's caller has already been told the file is damaged.
v0.5.2
Install: pip install oxihipo==0.5.2
Fixed
-
Five more bank-stream slices guarded against a corrupt offset table. 0.5.1
fixed the four columnar sites; the same&stream[record.bank_byte_range(e, b)]
pattern — a raw slice index with bounds read from the file — remained in
Event::iter_structures,OwnedEvent::bank(twice) andEventCtx::bank(twice).
These are the whole-event paths, so they are what a consumer reaches through
structures(),ev.bank()and thefor_eachcallback: a wider surface than the
columnar ones.Noneis returned instead of panicking, which already means "no
such bank in this event" at every call site.After the 0.5.1 fix a downstream fuzz test found the fifth site immediately, so
the pattern was grepped for rather than the report chased, which found the other
four at once.
v0.5.1
Install: pip install oxihipo==0.5.1
Fixed
-
A corrupt bank offset table panicked instead of erroring. Four places sliced
a decompressed bank stream with a byte range read from the record's own offset
table —&stream[rec.bank_byte_range(e, b)]— inread_columns(both the
by-bank and per-column branches) and infor_each_column(likewise). A damaged
table points past the end of the stream, and indexing a slice raw panics:
range end index 3400 out of range for slice of length 3379. Every other kind of
damage in this reader surfaces as anErr, so this was reachable fromscan,
stats,histandbanksin any downstream tool, on a file that opened cleanly.A bank whose extent does not fit is now treated as absent, which is how a bank
that is not there was already reported.Found by property-testing a downstream CLI against byte-flipped files, not by
reading the code: the offsets have to survive enough of the header to be used at
all, which is a narrow enough window that no hand-written case had hit it. The
regression test flips every byte of a by-bank file three ways and drives every
columnar entry point — which is what turned up the fourth site, after the first
three were fixed.One further raw slice in the same function,
&stream[..n * elem], was checked
and left alone:nis derived by dividing byelem, so the bound holds by
construction.
v0.5.0
Install: pip install oxihipo==0.5.0
Added
-
Chain::bank_occupancy— which banks carry data, in how many events, and
how many rows, without inflating a single bank or column. Every number is a
function of a bank's per-event byte extent, which both columnar layouts already
record in their bank-offset tables, so onLz4PerBankandLz4PerColumn
nothing beyond each record's header and offset tables is decompressed. Honors
the chain filter, takes an optional global-index range, and follows the usual
threadsconvention (0= all cores,1= sequential).This exists because the operation is easy to get wrong outside the library.
Computing it fromChain::eventscosts anOwnedEvent— a copy of every
event's bytes — and enumerating a per-column event's structures first
synthesises a whole event out of separate column streams: measured at
19 µs/event onLz4PerBankand 26 µs/event onLz4PerColumnin a downstream
tool, against roughly 1.5 µs/event for reading the presence tables.EventCtxlooks like the escape route and is not: it avoids the copy but
cannot enumerate a per-column record's banks, because that needs exactly the
synthesis it exists to avoid. A caller who tried it got 4 banks out of 71 —
fast, plausible, and wrong with no error. Putting the operation here, with a
cross-format equality test, is the fix for that class of mistake rather than
for one instance of it.Banks declared but never populated are returned with zero counts, so "never
written" stays distinguishable from "not in the dictionary". A bank opened with
no rows counts as carrying no data, which is the question being asked.Classic layouts (
None/Lz4/Lz4Best/Gzip) keep no per-bank table, so
their records are decompressed and their events walked — but with no per-event
allocation, so they gain too. -
BankOccupancyin the crate root, the per-bank result type.
v0.4.1
Install: pip install oxihipo==0.4.1
Fixed
-
for_each_columnfailed outright onLz4PerBankfiles. Its fallback
comment claimed to cover "Bytes / ByBank / chunked", but the fallback calls
decode_record_into, which expects a single whole-record payload. A by-bank
record is one LZ4 stream per bank plus a directory, so the call died with
lz4 decompress failed. Every other format worked, and the one existing test
only ever triedLz4PerColumnandLz4, so nothing caught it.There is now a by-bank branch that inflates just the requested bank's stream
and reads the column per event, mirroring the per-column opaque path — and a
test that sweeps a scalar and a jagged column on all six formats and
requires them to agree with per-event reads.Found from downstream: a CLI built on this crate could not run
statson
files its ownskimhad written, sinceLz4PerBankis a common default. -
PDG masses for light nuclei were the neutral atom, not the bare nucleus.
The table was generated fromparticle, which tabulates the atom for the
10LZZZAAAIcodes — the deuteron entry was 2.014101778 u exactly. A detector
sees a stripped ion, so every nuclear mass was heavy by Z·m_e: 0.511 MeV for
a deuteron or triton, 1.022 MeV for an alpha or He3. Small, systematic, and
wrong for the one thingpdg_massexists to do.All eight nuclear codes (Geant3 45/46/47/49 and their PDG spellings) are
corrected; nothing else moves, since a free proton has no bound electron. The
cross-check againstparticlenow subtracts Z·m_e for those codes — comparing
raw would have re-asserted the bug — and a second test pins the four values
against the literature independently of the library.
Changed
for_each_columndocuments that it ignores the chain filter. It walks the
record index directly, sowith_filterand the record-tag pushdown are both
skipped and the caller gets every value in the file. That is deliberate — the
per-column fast path has no per-event predicate to apply — but silently
returning a plausible number over the wrong event set is a trap, so the doc
now says so and points atread_columns, which is also columnar and does
honour the filter.
v0.4.0
Install: pip install oxihipo==0.4.0
Changed
-
create/recreate/updatefollow uproot. They were inverted: nothing
refused to overwrite,createclobbered, andrecreate(source, dst)meant
"decorate". Nowcreate(path)raisesFileExistsError,recreate(path)
replaces, andupdate(source, dst=None)decorates.Migration is guarded rather than silent.
recreate(source, dst)still works
with aDeprecationWarningand behaves asupdate.recreate(path)on a file
that already exists raises for one release — the old meaning decorated that
file and the new one destroys it, so acting on either guess could lose data.
Passoverwrite=Truewhen you mean the new behaviour. -
The sdist no longer advertises the wrong README.
readmeresolves relative
to the manifest directory, and maturin's sdist re-rootspy/pyproject.tomlto
the tarball root — next to the Rust README. A wheel built directly carried
the Python description while one built from the sdist carried the Rust one.
Both now resolve throughREADME-pypi.md, a symlink present at each root. -
The Python floor is back to 3.10, reversing the 0.2.1 raise to 3.13. The
floor is a support decision, not a syntax one: the package needs only
from __future__ import annotationsplus PEP 604 unions in annotations, so
3.13 excluded interpreters that run it fine — most of the installed base, for
no benefit.abi3-py310(wheels arecp310-abi3),requires-python >=3.10,
the 3.10–3.12 classifiers are back, and everypython-X.Y+badge follows.
The mypypython_versionis now3.10and CI's test job runs on 3.10, so a
3.11+ construct fails in review rather than at a user's import. -
A chain no longer requires every file to carry the same dictionary. Opening
one refused unless each file'sDictcompared equal to file 0's, and that
comparison was order-sensitive —DictderivesPartialEqover a positional
Vec<Schema>plus index tables whose values are insertion indices — so files
describing exactly the same banks in a different order were rejected, though
nothing about the format makes that order meaningful.Real run periods are not dictionary-uniform either: a pass-2 cook adds a bank,
an MC file carriesMC::Lund.ox.open("…/pass2/*/dst/*.hipo")died on that,
with no escape hatch. The chain now takes the union:keys()reports every
bank any file declares, and a bank absent from a file yields empty entries for
that file's events — which the read path already did for an absent bank.Two conflicts are still hard errors, because a reader cannot survive them: one
name describing two layouts (columns would be decoded against the wrong
schema), and one(group, item)used for two banks. The second is the
dangerous one — the columnar path locates banks by id, so a collision would
decode one file's bytes with another file's schema and return wrong numbers
rather than fail. Both errors name the files and banks involved. -
The read path goes through a
ReadAtseam instead of holding an
Arc<File>directly, so a source that is not a local file — an in-memory
image, eventually HTTP range requests — only has to supply bytes at an offset.
Entirely internal:SharedFilenever leftsrc/read/inner.rs, every caller
already went throughFileInner's two methods, and the in-place tag patch
opens its own handle, so nothing outside that file changed.The trait fills a caller-owned buffer rather than returning one (the
zero-allocation scan loop is pinned by a test) and has nolen()— the file
length is captured once at open, so no bounds check asks the source its size.
Measured against the pre-change baseline, one virtual call per multi-MB record
is not detectable:scan−1.2%/−2.2%/−6.0%,columns+1.5%/+1.2%/+3.5%,
open−5.9%/0.0%/−0.5%, all inside this machine's run-to-run spread. -
record_decompressed_sizes()reads its record headers in parallel. It runs
beforeiterate(step_size="200 MB")can plan a single batch, so on a
many-record chain the serial version was a visible stall before any data moved. -
The version badges on the GitHub README, which showed
v0.1.1and
python 3.10 | … | 3.14while 0.3.0 was current — stale across four releases.They were dynamic, on the reasoning that a live page wants "latest". That
reasoning was wrong: a shields.io badge sits behind its own Cloudflare edge
(max-age=10800) and, on GitHub, behindcamo.githubusercontent.com, and a
dynamic URL never changes, so neither cache ever refetches. A static
pypi-vX.Y.Zbadge puts the version in the URL, so each release mints a URL no
cache has seen and the correct image appears immediately.All four badge sites are now static.
scripts/release.py preparerewrites
three of them and the generated docs page reads the version from
py/pyproject.toml, so none can be forgotten.checkalso now asserts the
python-3.13+badges matchrequires-python— the mismatch that shipped in
0.2.0. -
A release no longer publishes without reaching the docs site. The
release-notes page is generated fromCHANGELOG.mdat build time, but the docs
workflow was path-filtered towebsite/**— and a release commit touches the
changelog and manifests, notwebsite/. So 0.3.0 shipped to PyPI while the
site still showed 0.2.2.CHANGELOG.mdis now in the filter.
Fixed
-
to_daskno longer builds a degenerate partition for an empty range.
entry_start == entry_stoplanding inside a record produced a zero-width
batch rather than none, so the array got a partition spanning no events and
two equal divisions — which dask requires to be increasing. The same range
past the end of the chain already raised; both now do. Found while giving
map_reducethe same batch logic. -
read_columns_atno longer re-reads a record per index. Every lookup went
throughChain::eventand its single-slot record cache, so the cost depended
entirely on the order of the list: 256 indices that happened to ascend cost
13 µs, the same 256 scattered cost 7 ms, because each one decompressed a
whole record that the next call threw away.Entries are now resolved up front and grouped by the record holding them, so
each record is read once whatever the order, and the groups run in parallel
(threads, matchingread_columns). Scattered reads are ~26× faster
(7.09 ms → 271 µs,lz4); ascending reads, which the cache already handled,
stay put. This is whatarrays(entries=[...])runs on, and a list of
interesting events found by an earlier pass is rarely sorted.Chain::read_columns_attakes a trailingthreadsargument to match its
siblings — a breaking change, called out here because the crate is pre-1.0. -
A failed
withblock no longer produces output.Writer.__exit__called
close()unconditionally, so an exception inside the block still finalised the
file — a failed run left one that opens cleanly. For the in-place
recreate(dst=None)it also ranos.replace(temp, final), overwriting the
source with a partial result. And whenclose()itself raised, that
exception replaced the user's. It now aborts, removes the partial output, and
never masks the original error. -
filtered()composes. Each call built its filter from its own arguments
alone, sof.filtered(require=…).filtered(event_tag=…)silently dropped the
require— and the record-tag clause was widened rather than dropped,
because the core unioned record tags while replacing every other clause.
Chaining now narrows:requireunions,record_tagandevent_tag
intersect, and twoevent_tag_anyclauses raise, because "any of A" and "any
of B" is not expressible as one bitmask. -
entries=no longer answers wrongly on a filtered chain. It resolves each
index through the random-access path, which addresses the file's event stream
and never consulted the filter — so it returned events the filter excludes,
and the indices did not mean what a range read means. It now raises; making
the two agree needs a decision about which index spaceentries=speaks. -
The key namespace no longer depends on how many banks matched.
single—
did the caller name one bank as a bare string? — reached only the Awkward
assembler, soarrays(["REC::Particle"], library="np")returned bare
pid/pxkeys while the same call withlibrary="ak"returned a record
namespaced by bank. A loop keyed on"BANK/col"worked until it met a file
where the glob matched a single bank. All four backends now key off the
request. -
banks=together withfilter_name=is refused.filter_namereplaces
the bank selection outright, soarrays("REC::Particle", filter_name="REC::Event*")silently returnedREC::Event. Now aTypeError. -
skim(tags=…)no longer leaves a mis-tagged file behind. The length check
ran after the skim had finished, and the shorttagswas padded with zero per
event — so the caller got an exception and a complete, silently mis-tagged
file that opened cleanly. The partial output is removed. -
The Arrow schema is declared non-nullable. It was inferred, leaving every
field nullable, so a Parquet round-trip returnedoption[var * ?float32]
instead ofvar * float32. The docs blamed Arrow for this; the schema was
ours. Values were never affected. -
composite(library="np")keeps a consistent shape.np.array(…, dtype=object)over equal-length slices collapses to a rank-2 array of boxed
scalars, so the result's shape depended on whether the file happened to have a
constant number of rows per event. -
A URL passed to
open()now reports that remote sources are unsupported,
instead of falling through to the glob branch and reporting "no such file or
directory".
Added
- **`Chain.map_reduce(fn, ......