Repository navigation
dendro v0.1.0
First release. The crate is a segmented Parquet archive with a write-ahead
log, in a single SQLite file: rows land in the WAL and are periodically sealed
into immutable parquet segments, with the catalog, retention, crash recovery
and rewriting around them. A row is a timestamp, a wall-clock offset and opaque bytes; what the
bytes mean is the caller's, expressed through one trait.
Added
- Writer, source, stream, segment and row model, with the WAL readable rather
than a staging area, and snapshot isolation for readers. SegmentEncoder, the whole schema boundary, plus an encoder version marker
that refuses a reader whose encoder disagrees with the writer's.- Per-stream retention by time, with the size accounting a cap needs.
- Reopening an archive and resuming a source, with writer sessions recorded
and a floor that refuses an append behind where the previous session stopped. - Out-of-order appends are dropped rather than silently stored, counted per
source and logged once per stream. - The caller's time-keyed store,
caller_rows: opaque rows against
(source, stream, ts)that the archive reads by range, evicts by the same
cutoff as segments, carries verbatim in every copy, and never decodes or
merges.Transaction::insert_caller_rows,ArchiveMut::insert_caller_rows,
SourceWriter::caller_rows,Archive::read_caller_rowsand
Archive::caller_row_streams;Evicted::caller_rowscounts what
retention removed. A store row does not make a stream exist. This is the
store the schema-churn journal entry owed to a caller that keeps column
identity in a secondary index. - An opaque per-segment index slot, so a caller that builds its own index over
segment contents can keep it in the archive and stay one file. Archive::verify, reporting every problem it finds in one pass.read::describe, one call for the whole catalog plus the file's own size.- Optional wall-clock alignment for segment boundaries.
rewrite::compact, merging a stream's small segments and reclaiming the
space, measured at 18.2x read and 2.38x size between 400 segments and one.SchemaPolicy::UnionFields, opting into merging across a column set that
grew or shrank.- Rewriting: combine, trim, range-copy and opt-in column projection.
- An archive is exactly a file that carries dendro's header stamp. There is
no fallback to a catalog table and no legacy schema: a.rezrecording
from before dendro is upgraded by rezolus, which still reads it. archive::ArchiveMut, the write handle, besideArchive, which now only reads.
Archive::openis read-only (the formeropen_read_only), works on read-only
media and on a file another process is writing, and is the only open the
read paths use.ArchiveMut::createmakes a new archive;ArchiveMut::opentakes an
existing one with SQLite's exclusive locking mode, so it is refused as
Error::InUsewhile anything else holds the file, and it refuses read-only
media by name.ArchiveMutderefs toArchive. Before this, a
second read-write connection on a file the writer thread held wrote
silently and left the writer's cached sequence numbers and watermarks
wrong. Seven crate-internal methods (open_for_write,remove_archive,
next_seqs,source_time_span,total_wal_rows, the pragma readers) left
the public surface, andReadOnly::Handleis gone because a read handle
has nothing to refuse.- Each
writer_sessionsentry records the dendro crate version that
appended, underdendro. Provenance for tracing a defect to the sessions
that had it, never a gate: readability is the header's schema version. keys::PRODUCER_VERSION, a reserved metadata key for the version of the
software that produced a source's values. Written by the producer, stored
opaquely, never parsed. Distinct fromencoder, which versions the row
encoding and which dendro enforces: a producer that keeps its encoding and
changes what it measures moves every value while the encoder version stays
identical, which is the caseencodercannot see.
(rezolus #1195.)
Notes
- Minimum supported Rust version is 1.85, the floor the dependency graph
sets. Raising it is a minor-version change and is gated in CI. - The reader configuration (
--no-default-features) drops the writer thread
and is what builds forwasm32-unknown-unknown. It is a supported
configuration, tested in CI.