Skip to content

dendro v0.1.0

Choose a tag to compare

@github-actions github-actions released this 16 Sep 17:01
· 75 commits to main since this release
5108527

First release. The crate is a segmented Parquet archive with a write-ahead
log, in a single SQLite file: rows land in the WAL and are periodically sealed
into immutable parquet segments, with the catalog, retention, crash recovery
and rewriting around them. A row is a timestamp, a wall-clock offset and opaque bytes; what the
bytes mean is the caller's, expressed through one trait.

Added

  • Writer, source, stream, segment and row model, with the WAL readable rather
    than a staging area, and snapshot isolation for readers.
  • SegmentEncoder, the whole schema boundary, plus an encoder version marker
    that refuses a reader whose encoder disagrees with the writer's.
  • Per-stream retention by time, with the size accounting a cap needs.
  • Reopening an archive and resuming a source, with writer sessions recorded
    and a floor that refuses an append behind where the previous session stopped.
  • Out-of-order appends are dropped rather than silently stored, counted per
    source and logged once per stream.
  • The caller's time-keyed store, caller_rows: opaque rows against
    (source, stream, ts) that the archive reads by range, evicts by the same
    cutoff as segments, carries verbatim in every copy, and never decodes or
    merges. Transaction::insert_caller_rows, ArchiveMut::insert_caller_rows,
    SourceWriter::caller_rows, Archive::read_caller_rows and
    Archive::caller_row_streams; Evicted::caller_rows counts what
    retention removed. A store row does not make a stream exist. This is the
    store the schema-churn journal entry owed to a caller that keeps column
    identity in a secondary index.
  • An opaque per-segment index slot, so a caller that builds its own index over
    segment contents can keep it in the archive and stay one file.
  • Archive::verify, reporting every problem it finds in one pass.
  • read::describe, one call for the whole catalog plus the file's own size.
  • Optional wall-clock alignment for segment boundaries.
  • rewrite::compact, merging a stream's small segments and reclaiming the
    space, measured at 18.2x read and 2.38x size between 400 segments and one.
  • SchemaPolicy::UnionFields, opting into merging across a column set that
    grew or shrank.
  • Rewriting: combine, trim, range-copy and opt-in column projection.
  • An archive is exactly a file that carries dendro's header stamp. There is
    no fallback to a catalog table and no legacy schema: a .rez recording
    from before dendro is upgraded by rezolus, which still reads it.
  • archive::ArchiveMut, the write handle, beside Archive, which now only reads.
    Archive::open is read-only (the former open_read_only), works on read-only
    media and on a file another process is writing, and is the only open the
    read paths use. ArchiveMut::create makes a new archive; ArchiveMut::open takes an
    existing one with SQLite's exclusive locking mode, so it is refused as
    Error::InUse while anything else holds the file, and it refuses read-only
    media by name. ArchiveMut derefs to Archive. Before this, a
    second read-write connection on a file the writer thread held wrote
    silently and left the writer's cached sequence numbers and watermarks
    wrong. Seven crate-internal methods (open_for_write, remove_archive,
    next_seqs, source_time_span, total_wal_rows, the pragma readers) left
    the public surface, and ReadOnly::Handle is gone because a read handle
    has nothing to refuse.
  • Each writer_sessions entry records the dendro crate version that
    appended, under dendro. Provenance for tracing a defect to the sessions
    that had it, never a gate: readability is the header's schema version.
  • keys::PRODUCER_VERSION, a reserved metadata key for the version of the
    software that produced a source's values. Written by the producer, stored
    opaquely, never parsed. Distinct from encoder, which versions the row
    encoding and which dendro enforces: a producer that keeps its encoding and
    changes what it measures moves every value while the encoder version stays
    identical, which is the case encoder cannot see.
    (rezolus #1195.)

Notes

  • Minimum supported Rust version is 1.85, the floor the dependency graph
    sets. Raising it is a minor-version change and is gated in CI.
  • The reader configuration (--no-default-features) drops the writer thread
    and is what builds for wasm32-unknown-unknown. It is a supported
    configuration, tested in CI.