v0.6.0
Added
- Golden archive fixtures pin the on-disk format.
crates/fqxv/tests/fixtures/
now holds one small archive per on-disk layout — plain/order-k, grouped (with the
header extension record), the whole-file reorder layout, and the long-read overlap
codec — plus a manifest of what each must decode to. Until now every test in the
workspace compared a build against itself: the round-trips, the proptests, the
fuzz targets, the thread-determinism byte-comparisons, and the accession corpus
(which recompresses from FASTQ each run) are all invariant to a change that moves
the encoder and decoder together, which is exactly the change that breaks archives
already written. Regenerate with
cargo run --release -p fqxv --example make_fixtures -- crates/fqxv/tests/fixtures,
but note that regenerating is almost always the wrong response to a failure. - CI checks cross-release compatibility. A new
compatjob downloads the
previous release's CLI, compresses with it and decompresses with the PR build
(a hard failure if that breaks), then does the reverse and requires the old binary
to either reproduce the bytes exactly or refuse loudly — a silent success with
different output fails the job. - The changelog is on the documentation site. Included verbatim from this file,
so the two cannot drift. - A documented format evolution policy (
docs/design/container.md): which
mechanism a given change belongs in, the rule that any change to the footer's
shape or the block payload's stream layout must be gated by arequired_features
bit, and the project's intent on major bumps (a last resort; a format-1 read path
is retained if one ever happens).
Changed
-
Three implicit parts of the layout now fail closed. Each was a structural
convention living in a constant rather than on disk, and each would accept an
archive from a future writer and get it wrong:- A block payload carrying a fourth stream decoded its first three and dropped
the rest with no error at all — and the per-block content digests, which cover
only the streams that were read, still matched, so the archive looked healthy.
Trailing bytes after the three streams are now rejected. - A footer written at a different per-group stride (a future fourth stream
location) passed its CRC and was then walked at the wrong stride, producing an
index that could satisfy every range check:fqxv inforeported a 1,000-read
archive as 46,048 reads. The footer body length must now match its row-group
count exactly. - Unknown header flag bits (6 and 7) were ignored, so a future flag whose
meaning is purely semantic would decode under the old interpretation with every
CRC and digest still matching. Unknown bits are now refused with a new
Error::UnsupportedFlags.
All three are backward compatible: no archive any release has written trips them.
- A block payload carrying a fourth stream decoded its first three and dropped
-
Original per-slot member labels are recorded and restored. Member identity in
the container is positional, sodecompress_splitcould only number its outputs
_R1.._RG— compressing a run's_2.fastq+_4.fastqand restoring them
renamed them_R1/_R2, and a 10xI1came back as_R3. The CLI now derives
each member's slot token from the input file names (all-or-nothing: every input
must yield a distinct token, or nothing is recorded) and stores them in the
header.--mate-style auto(the new default) restores the stored labels, falling
back to_R1,_R2,…; explicitr/numstay positional. This is the first
user of the header extension region: a non-critical TLV record (tag0x01), so
a reader that predates the tag skips it and decodes byte-identical reads under
positional names. The format version deliberately stays at 1.0 — the record is
skippable, and nothing about decoding depends on it. Archives compressed without
labels still write an empty extension region.
Fixed
--estimate's help text described the wrong mechanism. It claimed to code the
leading reads with the real codecs; it measures the sample's empirical entropy and
projects from that, which is why it is so much faster than a real run. The
documentation was right and the shipped--helpstring was not. A documentation
audit against the actual CLI, bindings, and codecs corrected this and a good deal
more — thefqxv infosample output was internally impossible,-vwas
documented as debug when it is info, batch mode over several archives or a
directory was undocumented, and the container layout diagram omitted the
whole-file reference frame that sits between the header and the first block.fqxv.Index.groups()no longer panics on an allocation failure; it raises.- Unequal input read counts are rejected in both directions.
compress_multi
treated member 0's EOF as a clean end of input, so a short member 0 ended the
archive early and silently dropped the surplus reads from the longer members —
yielding a well-formed, CRC-valid archive that was quietly missing data and still
reported success (with an inflated ratio, since it is computed against the full
input size). All three interleaving sites now confirm members 1..G are also spent.
The reverse direction already errored. - A trailing partial spot is rejected in the reorder layout too. The plain
layout refuses an interleaved stream whose record count is not a multiple of the
group size, but the reorder layout only checked it at one call site, which
compress_autobypasses — so--order any/--maxon a mate-named stream with
an odd record count recordedgroup_size = 2over a stream that is not
spot-aligned and split it into mismatched files. The check moved into
encode_reordered, the choke point every reorder entry point passes through. - The streaming parser validates the
+separator line.read_raw_record
compared only the sequence and quality lengths, so a header line followed by EOF
("@name\n") parsed as a zero-length record — silently repairing a truncated file
into a valid archive — and a garbage third line was consumed and discarded. Both
were reachable only through the multi-input path (a single-file compress rejects
them at the noodles-based peek). The+line's content is still dropped;+
normalization is unchanged. - A non-empty header extension region no longer corrupts the footer.
FooterIndex::newseeded block offsets from the extension-empty header length and
the recovery scan started there, so any extension record shifted every recorded
offset and failed the footer CRC. Both now use the header's actual length. The
region had never been exercised before the member-label record.