Repository navigation
Releases
v0.2.0
The release that makes a journal a log over one object, and makes a value type one
attribute. Nothing depends on the 0.1 interface yet, so both breaks are taken now
rather than carried.
Journal (breaking)
Journal::attach(base, dir)adopts an existing object as the base of the log, and
Journal::open(dir)reopens.Journal::start(dir, tree_id)is the empty-base case
and resumes a directory that already holds a journal, so a service calls it on every
start.- The base is recorded by content id. Before the first checkpoint the tree is the base
plus the replayed records, and a base that is missing or was replaced is
Error::BaseLost. After the first checkpoint the tree comes from that checkpoint and
the base is provenance. Journal::open(dir, tree_id),Journal::createand thetree_idargument on
Journal::optionsare gone. Identity comes from the base: a plain object's tree id is
derived from its content id, a checkpoint keeps the id of the stream it belongs to.Journal::base(),Journal::last_checkpoint()andjournal::recorded_type_id(dir)
exist so a tool can report what a directory is without opening it as a value type.
CLI (breaking)
foldis removed. The journal's product is the checkpoint it writes itself.- New:
journal init <base.snap> <dir>,journal append <dir> <records.jsonl>,
journal checkpoint <dir>,journal status <dir>, all with--json. Append takes
{"prefix": .., "value": n}to insert and{"prefix": .., "withdraw": true}to
remove; the value type comes from the directory, not a flag.
Value types
#[snap_value("id")]is the one-attribute form: the id is positional,Cloneis
derived unless the caller already derives it, andrkyv(...)options pass through.
#[derive(SnapValue)]remains for a type that already derivesrkyvitself.- the attribute refuses a type that already derives
rkyv's traits, naming both ways
out, and refusesrkyv(crate = ...), which it sets itself.
A second trie with its own value type
profile::TrieSection<U>builds the body of a profile section holding a trie of
another value type, andSnapshot::section_trie::<U>(id)returns a queryable view
(len,get_exact,lpm,iter) whose values come back by value. The body records
the value type id, so a section read as the wrong type isConfigMismatchrather than
reinterpreted bytes, and both archives are 16-byte aligned because an unaligned rkyv
archive is not valid.
v0.1.0
The first version under the pre-stable numbering. The crate moved from 1.0.0-rc.4 to
0.1.0 on purpose: no service has leaned on this interface for long enough to freeze
it, and a 1.0 that has to break is worse than a 0.1 that is allowed to. The wire layout
is unchanged (v4) and is documented as the contract; the Rust API is not.
This is also the release that makes the surface small enough to describe without a
table of near-duplicates, and the one that makes the journal's durability claim true.
API
snap.owned()answers every value-returning query withT;search_ownedand
get_exact_ownedare gone, because a pair of owned methods covered two queries out
of five and left the rest to the caller's ownrkyvcall.TreeIdwithTreeId::from_name("...")(the first 16 bytes of its BLAKE3 hash).
Journal::open,open_withandrepairtakeimpl Into<TreeId>, so a plain
[u8; 16]still works and no example has to invent sixteen bytes.Journal::to_snapshot(&opts)andJournal::to_bytes(&opts): the object a
checkpoint would write, opened or as bytes, without touching the disk. This is how
a journal user reacheschildren,coveranditerwithout a second copy of the
query surface on the journal.Writer::insert_manystops at the first refused prefix.Writerdeliberately does
not implementExtend, which cannot report that.Snapshot::merge_last_winsandmerge_with: fold several objects into one, the
daily-merge step. The merge drops stream coordinates because several streams are not
one stream, and takes the union of the input windows when every input declares one.ipnetis re-exported. Unlike the archive re-export this one is complete, because no
macro expands::ipnet::paths, so a caller cannot end up on a differentipnet
version than the signatures use.- five queries removed because they were compositions of the others:
lpm_prefix,
spm_prefix,get_key_value,cover_keys,cover_values. Error::BadArchive(u16)separates "this section's body is not an archive of your
type" fromBadSectionKind, which is about the four-byte kind tag.Erroris
#[non_exhaustive], so a new refusal does not break a caller's match.AddressCounts/Count { NoEntries, All, Exact(u128) }: the old pair of options
could not tell "no entries in this family" from "this family is covered end to end"
(::/0covers 2^128 addresses, which does not fit in the count).Countprints
(all,0, the number) andCount::exact()gives the number when there is one.#[derive(SnapValue)]from the newbgpkit-snap-derivecrate (feature
derive):#[snap(type_id = "...")]replaces the hand-writtenimpl, and the
id is checked where it is written rather than when an object is finished.Journal::seq()andJournal::generation(): the sequence the next record will
carry and the stream incarnation, so a caller shipping records elsewhere can say
how far the journal has got and which stream wrote it.- a journal's value type is now pinned by its first record, not by its first sync: a
directory whose records were never synced had nostate.json, so a reader of
another type could open it and decode those bytes as its own. Snapshot::lpmtakes a prefix andSnapshot::searchtakes an address; both are
documented as the same lookup, one with the host prefix built for you.Journal::open(dir, tree)andopen_with_payload(dir, tree, payload)replace the
positionalOption<PayloadSection>;checkpoint_with(&opts, &validator)runs a
profile's own check before the object is committed;Snapshot::view()is public so
a validator can be run over an object a caller already holds.- every write path takes the value by value (
Journal::insert(prefix, value)), like
Writer::insert, instead of borrowing one the journal copies anyway.
Journal
- rotation flushes and fsyncs the outgoing segment before dropping it, and keeps the
segment if that fails. Previouslysync()returned Ok while the records in a
rotated segment had never reached stable storage. - the torn-tail cut runs before rotation, so a recovery offset recorded against the
old segment is not applied to the new one. checkpointcarries the journal's own sections only, because a checkpoint has to be
an object the journal can reopen; recovery refuses a checkpoint written under a
different payload configuration instead of opening it and losing the arena. Pruning
is best effort and never fails a committed checkpoint.childrenreturns stored keys: a query prefix with host bits answers for its
network address instead of returning a key the object does not contain.- the owned view reports a value whose
Deserializerefuses as an error, not as an
absent prefix (which madelen()disagree withiter().count()). - a zero generation is refused at open; it used to be accepted and then fail every
checkpoint. - a checkpoint that cannot be read reports the file system's error rather than
"missing" for everyio::Error, and rotation andsynckeep errno on a failed
flush. state.jsonrecords the value type and payload configuration, so reopening a
directory with another type or another payload id is refused instead of
reinterpreting the same bytes;Error::ConfigMismatchnames that case.repairrefuses a directory that still has a readablestate.json, because
rewritingnext_seqthere hides the sequence gapopenreports.- every checkpoint writes a new file name (
snapshot-<seq>-<n>.snap), so a checkpoint
with no new records cannot replace an object that is already committed. - the padding walk in
verify_bytesandSnapshot::from_bytesbounds-checks, so a
crafted object is refused instead of panicking. - a checkpoint decodes every value before it commits, because recovery decodes them on
open; an object holding an undecodable value would be unopenable for good. - recovery compares a payload section's
requiredflag as well as its id and kind. SnapshotView::sectionsis&[SectionRef]with named fields instead of a tuple; the
library returnsErrorwhere it used toexpect.
Verification
- tests: reopen after a checkpoint with a sidecar, a required sidecar refused, host
bits refused on the blob and withdraw paths, a journal crossing the 64 MiB segment
boundary reopening with both records,TreeIdstability and tree mismatch,
to_snapshot/to_byteswithout a file, andinsert_manystopping.
v1.0.0-rc.3
The first release where the API is what a developer guide can describe without
apologising for it, and the first with the write path's silent-loss cases closed.
API
- one name per query:
lpm,lpm_prefix,spm,spm_prefix,get_exact,
get_exact_owned.search_prefix,get_lpm,get_spmandSnapshot::getare
gone; a reader wanting the stored entry can no longer silently get an ancestor. Opts.created_atis unix seconds andOpts::now()exists.Opts::default()is
gone: a default would invent provenance.Snapshot::opentakes anyAsRef<Path>;Writer::finish_towrites atomically
with a per-call temp name and fsyncs.Writer::insertreturns aResultand refuses a prefix with host bits set in
every build, not only in debug.rkyvis re-exported, but a value type still needs its ownrkyvdependency:
the derive generates paths the re-export cannot satisfy.
Journal
checkpointkeeps the caller'sOpts.extrasinstead of dropping them, always
writes the payload section it was opened with (even when the arena is empty),
publishescurrent.snaponly afterstate.jsonnames the object, and removes
older snapshots instead of accumulating one per checkpoint.Journal::repair(dir, tree_id, payload)recovers a journal whosestate.json
was lost, taking the writer lock before it writes the state it is repairing.
adopt_checkpointis gone: two names for one repair confused two review rounds.insert_with_blob(prefix, blob, |offset| value)builds the handle from the
offset in the same call that decides it;insert_with_blob_atadds timestamps.Times { recv_ts, event_ts },insert_atandwithdraw_atcarry record time.- segments rotate at 64 MiB instead of growing for the life of the writer.
Error::Locked,BadOpts,BadSectionIdandBadPrefixreplace stringly-typed
variants for the cases a caller must act on.
Verification
Validator::validatenow runs after both trie archives are byte-checked, which
is what the documentation always claimed.- CI builds with
--features cliand buildstests/downstream, the out-of-crate
consumer that proves a value type can be added without touchingsrc/.
v1.0.0-rc.2
The container loses its metadata section, its footer and its JSON. The header is
now the only claim about the object, and checked rkyv access is the proof. This
release is the first one with a journal that survives restarts on its own terms,
and the first one reviewed against a written byte contract.
Format
- 192-byte header with the value type, file length, section count and table
length, coverage window, stream coordinates and content id; a crc32c over
header[0..188]. Everything up to byte 148 is hashed, so there is no
field-exclusion rule. - 64-byte section table without offsets: sections are packed in ascending id
order, 16-byte aligned, padding zero and checked. - Family sections are bound to their ids (2 and 3) in both directions: a section
whose kind is V4 at any other id is refused instead of ignored. content_idcovers the header prefix, the table and every section.- Deleted: META, the projection, the footer, per-section hashes,
WIDTH,
type_id_len,minor,header_len,table_off, per-entry offsets.
Journal
Journal<T>owns the tree, the blob arena, the writer lock and the segments.- Records carry their value bytes, so replay rebuilds the value; a record with a
blob carries the blob, so payload offsets reproduce exactly. state.jsonis the commit record: it names the checkpoint and the next
sequence number, so recovery starts from the checkpoint, replays only records
above it, and never hands out a number twice.- A hole, an unreadable segment header, a malformed checkpoint, a foreign
checkpoint, a checkpoint section the journal was not opened for, and an
unreadable state file are all errors. A torn tail is not: the next append cuts
it away and reuses the number. - Checkpoint verifies the object it built against the live tree, is itself the
validator for the section it wrote, and prunes only segments it fully covers.
Library and tooling
verify_byteschecks the container without the value type, so the CLI can
verify an object whose type belongs to another binary.Writer::finishrefuses headers a reader would reject; a value that is missing
or too wide is refused bysnap buildinstead of becoming zero.snapCLI: build, info, verify, search;infostill reads 192 bytes.
Review
The format document was rewritten as a byte contract (header, table, identity,
journal framing, section ids, payload dependency, test vector) and reviewed round
by round until the reviewer's verdict was SHIP. The nine rounds found reader
paths that could answer with less than the writer committed, state that moved
ahead of the log, and bytes the document fixed that the reader did not check.
v1.0.0-rc.1
First release candidate. The layout is frozen; the items in the README's
"Known limitations" are tracked for 1.0 final.
- 192-byte peek header replacing the earlier 64-byte prefix: magic, version,
tree id, journal generation, content id, type id, sequence, sizes, counts,
families, event window, header CRC. - Content id over the data projection and every non-META section; provenance is
excluded, so equal data from different paths hashes equally. - Journal with 48-byte segment headers, crc32c framed records, dense sequences,
a writer lock, a generation per incarnation, checkpoint verification, and
state.jsonfor restart-safe sequence numbering. snapCLI: build, verify, search, info.- Golden object test that fails on any archived-layout drift.