v0.6.0
Install: pip install oxihipo==0.6.0
A correctness release. Each entry is something that was silently wrong — the
library returned success and stored the wrong bytes, or aborted the process where
it should have returned an error — rather than something that failed visibly.
Breaking
BankBuilder::finishandEventBuilder::addreturnResult. Both are
public, so this is a semver break, taken deliberately: the alternative is an
API that silently corrupts data (below). Neither is used byhipo-tools, which
builds and passes its 204 tests against this release unchanged.
Fixed
-
A bank at or past 2^24 bytes was written truncated, with no error. The
structure length word carries the data size in its low 24 bits — the top byte
is the compositeheader_sizefield — andBankBuilder::finishwrote a full
u32into it.Writer::finishreturnedOkregardless, so the loss was
invisible until the file was read back. On a one-columnIntbank:rows written data bytes rows read back 4,194,303 16,777,212 4,194,303 4,194,304 16,777,216 0 5,000,000 20,000,000 805,696 At exactly 2^24 the size masks to zero and the overflowed
0x01re-reads as
header_size = 1, so the bank comes back looking composite. Every codec was
affected — this is the structure header, not the compression. The boundary is
nowHipoError::BankTooLarge. -
Bank::readwas documented "Infallible" and panicked in release builds.
Every check in it was adebug_assert, whileColumnHandle::placeholder()is
public and safe — and thebank_row!-generatedresolve_handleshands one out
for any column a runtime schema lacks. Reading through it fell into
schema.entries()[65535]and aborted with "index out of bounds: the len is 1
but the index is 65535", withdebug_assertions = false.The documentation was the defect: it now states the real contract with a
# Panicssection, and the bounds check is unconditional with a message that
names the mistake. Returning an empty column instead was rejected —readis
the bulk accessor and callers zip parallel columns, so a zero-length result
silently truncates the loop to no rows, trading a loud abort for quietly wrong
physics.read_handle_or_defaultremains the per-row path that does accept
placeholders. -
read_columnscould hand back buffers that contradicted their own
offsets.merge_chunksstates the contractColumnBuffersowes its caller —
offsets starting at 0 and non-decreasing, each column holding exactly
total_rows * inner_lenvalues — but only as adebug_assert. A corrupted
Lz4PerColumnrecord whose row counts and column payloads disagree produces
exactly that violation, so a release build returned buffers whose data
length did not match their offsets and slicing a row read the wrong values.
The invariant is now enforced and returnsHipoError::CorruptRecord.Found by the new sweep, and only in a debug build — the release runs it had
been checked against compile the assertion out, which is precisely why it had
survived. CI runs the debug profile and caught it on the first push. -
Iterating onto a zero-event record panicked.
EventIter::next_result
refilled the current record withifrather thanwhile.advance_record
resets the event cursor to 0, so the guard was tested against the record being
left and never against the one arrived at; landing on an empty record indexed
event_offsets[1]on a one-element table. A library may returnErrfor
damaged input, but a panic leaves its caller — a CLI, a Python binding, a batch
job — no way to handle the file at all.The empty record need not come from corruption:
Writer::flush_recordwill not
emit one, but the publicWriter::write_recordtakes a prebuilt record and
checks only that it is at least a header long.
Internal
- New
tests/mutation_sweep.rs. Mutates every byte of a real file, truncates
at every length, and writes hostile values into every header word, then drives
the whole public read path over each result asserting only that nothing panics.
It is what found the iterator bug — six single-byte mutations of a 2 KB file
reached it, each the low byte of a record's event count.corruption.rs
already had an empty-record test that could not have caught it: that test also
blanks the trailer to force the scan path, and the scan drops empty records
before iteration ever sees one.