dendro v0.6.1

Added

• Frame::SourceUpdate (wire kind 6, WIRE.md §4.6): a source's labels and
  metadata after its handshake, both maps whole, and ts, the publisher's
  wall time when it saw the change. Subscriber replaces its copy's labels
  and metadata with them, except the copy's encoder and writer_sessions
  where it has them; any other key the publisher removed is removed. It
  counts them in Applied::source_updates. A FrameReader on dendro 0.3.0
  or later that does not know the kind skips it (WIRE.md §7);
  wire::decode_payload before this release refuses it.
• SourceWriter::set_identity and ArchiveMut::set_source_identity, which
  replace a source's labels and metadata in one transaction, keeping the
  source's encoder and writer_sessions where it has them, since they
  describe the file's own encoder and writer sessions; every other key,
  events included, is replaced. Transaction::set_source_identity writes
  both maps as given.

Changed

• ArchivePublisher::next reads the archive's sources on every poll. It sends
  a SourceUpdate for a source whose labels or metadata changed since they
  were last sent, ahead of that source's other frames, and a Handshake,
  with the next ordinal, for a source created after the publisher opened,
  followed by what the opening catch-up would ship for it: segments and clock
  offsets from catching_up's since (all of them for tailing), index
  entries and live rows. Before this the publisher read the sources once,
  when it opened, so a subscriber kept the identity it saw at connect and
  never received a source added later. A caller of next that assumed a
  fixed set of sources now sees handshakes after the first batch.

Fixed

• ArchivePublisher::next ships as Segment frames the sealed rows of a
  stream that had no rows when a tailing publisher opened, no segments
  when a catching_up one opened, or that did not exist then, and that was
  sealed before a poll shipped any of its rows. Such a stream lost those
  rows without an error: it had no cursor to compare the seal's watermark
  with, and the live tail held only the rows after the seal. Every stream of
  a source created after the publisher opened and handshaken while empty was
  exposed to this.

dendro v0.6.0

This is 0.6.0, not 0.5.2. Archives this release creates are schema version
5, which releases up to 0.5.1 refuse (unsupported archive schema version 5). A patch release would reach dendro = "0.5" callers on a cargo update, and a process on the new build would then write files that their
other processes, still on the old one, cannot open: the same break-by-upgrade
the minor exists to prevent (see 0.2.0 below). Error gains variants, which
#[non_exhaustive] allows, and Error::UnsupportedSchema now reports
reads: 4.

Added

• Archive-level documents: named JSON documents that belong to the archive
  rather than to a source, in a new documents table (FORMAT.md §3.6). A
  document is (kind, name) with a caller-chosen schema string naming its
  body's type and version. Writes append a revision and never overwrite one;
  the current revision is the newest, and a delete is a revision with no
  body. Written with ArchiveMut::put_document, put_document_if_current
  (which takes the uuid of the revision the caller read, and is refused as
  the new Error::DocumentConflict when another revision is current) and
  delete_document, which return the new revision's archive::RevisionId
  (id and uuid), so conditional saves chain without a read between them;
  the same three on Transaction, and on Writer for an archive being
  recorded, which an ArchiveMut cannot open. Read with
  Archive::document, documents, document_kinds and document_history
  (types archive::{Document, Deletion, Revision}). A body must be JSON
  (RFC 8259, no nesting limit). Kinds beginning dendro.
  (archive::RESERVED_KIND_PREFIX) are refused.
• rewrite::copy_documents_into, which copies every document revision of
  one archive into a destination transaction, skipping revisions the
  destination already holds (by uuid) and failing on one it holds with
  different contents. It returns archive::CopiedDocuments: the revisions
  inserted, the documents whose current revision came from this input, and
  which of those displaced a revision they do not supersede (forked, each
  an archive::Fork with the displaced and current revisions' uuids), for
  a combine that must choose or rename.
• Archive::schema_version, and Error::SchemaTooOld, returned for a
  document written into a v4 archive.

Changed

• Breaking (format): the schema version is 5 (was 4). Every archive this
  release creates has the documents table and is stamped 5; this release
  reads 4 and 5. A v4 archive reads as holding no documents, and is still
  written in place as v4 (a writer resumes it, metadata edits and compaction
  run), but no document is written into it: copy it forward with
  rewrite::copy_sources_into into a new archive, which is v5. The reason
  is in FORMAT.md §8: 0.5.1's copier drops a table it does not know, and
  documents are data people wrote.
• Breaking (format): on schema 5 the caller's per-segment index is a
  row of a new segment_indexes table, keyed like its segment, instead of
  the segments.caller_index column (FORMAT.md §3.2). SQLite stores a
  row's columns in order and spills a large BLOB onto overflow pages, so an
  index after the segment bytes was reachable only through them: reading
  every index of a cold archive cost as much I/O as reading the segments.
  Archive::read_segment_indexes now reads no segment row on v5. The table
  cascades on segment delete, so retention, compaction and
  Transaction::delete_segment remove an index with its segment. A v4
  archive keeps its indexes in the column, is written there when written in
  place, and is read through the same API; a copy into a new archive moves
  them to the table. The Rust API is unchanged: SegmentRow::caller_index
  and insert_segment_with_index carry the index as before.
• Breaking: SegmentEncoder::encode and encode_after take
  &segment::EncodeContext in place of the stream name, and
  segment::materialize takes one too. The context carries the stream
  (ctx.stream), the source's row id in the archive (ctx.source_id) and
  its uuid (ctx.source_uuid). Before, an encoder was told only the stream
  name, which one writer with several sources (two hosts recording cpu)
  repeats, so an encoder keeping per-stream state could not key it. The
  writer, a reader materializing a tail and a copy all pass the source the
  rows are in; a copy passes the source archive's, not the destination's.
  The struct is #[non_exhaustive], so fields are added without a break.
  For an implementor: replace stream: &str with ctx: &EncodeContext<'_>
  and read ctx.stream; calls an implementation makes to its own encode
  or encode_after (an encode_after that falls back to encode, say)
  pass ctx instead of the stream too. metriken-storage needed about eight
  lines in three files. A caller driving an encoder itself (a test) builds
  one with EncodeContext::new(source_id, stream). Key state that outlives
  one archive by source_uuid: a copy renumbers source_id.
• rewrite::copy_sources_into also copies the source archive's documents,
  every revision, whatever the CopySpec selects. Called once per input,
  as a combine does, it carries each input's documents; call
  copy_documents_into first for the report.

dendro v0.5.1

Added

• ArchiveMut::update_source_labels, which replaces one source's labels in
  place, as update_source_metadata replaces its metadata. The source keeps
  its uuid. An assembly of several archives uses it to tag each copied
  source (an A/B arm) without rewriting segments.

dendro v0.5.0

Changed

• Breaking: arrow and parquet are 60 (were 58). Parquet 59 and later no
  longer depend on thrift, which clears GHSA-2f9f-gq7v-9h6m (thrift 0.17).
  Arrow and parquet types appear in the public API, so a caller must use the
  same major version: segment::encode_batch takes an arrow Schema and
  RecordBatch, segment::writer_props returns a parquet
  WriterProperties, CompactSpec::writer_props and
  CopySpec::writer_props hold one, MergeInput carries parquet
  KeyValues and arrow Fields, MetadataMerge::finish returns
  KeyValues, and ColumnFilter's methods take an arrow Field.
• Breaking: the minimum supported Rust version is 1.88 (was 1.85), the
  floor arrow and parquet 60 set. Parquet 59 would clear the advisory at
  1.85; 60 matches the metriken crates, which take arrow and parquet types
  across this API and move to 60 with it.

dendro v0.4.2

Added

• SourceWriter::seal_capped(batch, max_rows): a seal that writes no
  segment with more than max_rows WAL rows. A stream with more live rows
  than that is sealed as consecutive segments of max_rows rows, oldest
  first, and a last one with the remainder (7 live rows at a cap of 2 are
  segments of 2, 2, 2 and 1), each with its own catalog entry and seq,
  all in the batch's one transaction. The cap counts WAL rows, as the
  catalog's rows does, not the parquet rows an encoder writes for them.
  Only the row cap is applied: the byte cap and the age bound remain
  triggers for when the caller seals, because the writer does not have the
  byte estimate the caller's SegmentAccount charged per row. A seal that
  is deferred (its transaction could not commit) leaves its rows live, and
  the next capped seal of the stream splits them with the rest.
• SegmentEncoder::encode_after(stream, before, rows): how a capped seal
  encodes each part after the first, with the stream's earlier live rows
  as before. The default ignores before and calls encode, which is
  right when every row decodes on its own. An encoder whose rows depend on
  earlier ones (a schema carried only by the first row after a seal) must
  implement it before its caller uses seal_capped. A part whose segment
  drops a row at an edge it shares with another part, or that yields no
  segment, abandons the split for that stream, which is then sealed as one
  segment as seal would, so a split never loses a row. That holds only
  for an encoder whose segment reports the rows it wrote: one that claims
  rows it skipped hides the gap from this check.
• Evicted::held_streams: how many streams a writer retention pass left
  out because their seal was deferred and still could not commit.

Fixed

• A writer retention pass (SourceWriter::evict_before and its variants)
  deleted the live rows of a stream whose seal had been deferred, the same
  as any other live row past the cutoff. For an encoder whose rows depend
  on an earlier one (a schema stated once at a segment's start), that cut
  the row the survivors needed, and the next seal decoded none of them.
  The writer now remembers which streams' seals deferred, seals them at the
  start of the next pass (retried like any seal), and leaves out of the
  pass any stream it still cannot seal; that stream is evicted by the
  first pass after its seal commits. The stream is left out whole, its
  sealed segments and caller rows included. While a stream is held, a
  whole-source pass runs stream by stream, so a CallerRowFloor is asked
  for every stream name rather than only the names with caller rows. A
  pass after a deferral first retries the deferred streams' seals, each in
  its own transaction, on the writer thread; after the first one that
  still cannot commit (one retry schedule: up to four busy timeouts and
  310 ms) the rest are held without being tried. A retried seal that
  fails with an error that is not retryable, such as an encoder error, is
  not held: it is logged, the pass evicts the
  stream's live rows as it did before, and a Seal of the stream that
  fails the same way stops the writer, as it always has.
• A seal put every live row of a stream into one segment, so a caller that
  checked SegmentAccount::is_due once after several rows had arrived
  (frames that queued while a slow seal ran) got segments past its
  policy's max_rows: 3- and 4-row segments against a cap of 2 at 10 Hz
  with ~350 ms seals. seal_capped with the policy's max_rows bounds
  them. seal is unchanged.

dendro v0.4.1

Added

• rewrite::compact_with and rewrite::compact_stream_with: compaction
  with a caller-supplied rewrite::MetadataMerge that decides the merged
  segment's file key-value metadata. It is a fold over the run's segments in
  order, each given once as a rewrite::MergeInput (its metadata, columns
  and row count): start with the first, extend with each next one, which
  may refuse it and so end the run before it, and finish for the segments
  merged. A segment the SchemaPolicy refuses ends the run before the merge
  is given it. A merge that keeps what it has decoded in its Run costs time
  linear in the run. compact and compact_stream use
  rewrite::IdenticalMetadata, which merges only segments whose metadata has
  the same entries; its Run is rewrite::IdenticalMetadataRun. A caller
  whose metadata changes at every segment, such as a list of the occupants
  the rows refer to, which changes whenever the population does, can merge
  across those changes by combining the lists.
  MergeInput::new builds an input by hand, for testing a merge without an
  archive; MergeInput::rows is the segment's parquet row count, which for a
  stream that writes several parquet rows per WAL row is larger than the
  catalog's SegmentMeta::rows.
• rewrite::compaction_opens (test-support): how many segments compaction
  has opened on the calling thread.

Fixed

• Compaction (rewrite::compact, rewrite::compact_stream) cataloged a
  merged segment's rows as its parquet row count. Everywhere else the
  catalog's rows counts the WAL rows a segment covers (the seal path
  writes it, and segment::materialize refuses a segment whose count
  differs), and the two differ for an encoder that writes several parquet
  rows per WAL row, such as a long layout with one parquet row per occupant
  per WAL row. On one long recording, 0.4.0's compact took a
  cpu_usage_task/occupants stream from 3,383 catalog rows to 264,286, and
  with CompactSpec::unioning_fields it also took cpu_usage_task from
  34,678 to 76,182,021 and metriken's ArchiveReader::interval() from 1 s to
  0.001 s; this release's caller merge would have done the same to
  cpu_usage_task without the fix. A merged segment now catalogs the sum of
  its inputs' rows. Segments already compacted keep the inflated count, and
  the compaction planner, which sizes runs by that count, treats each as over
  any target below it, so they are not merged again.
• Compaction (rewrite::compact, rewrite::compact_stream) did not carry
  the parquet file key-value metadata of the segments it merged into the
  merged segment's file metadata. Under StopAtChange it moved the entries
  into the merged file's embedded ARROW:schema, and under UnionFields it
  dropped them entirely, so a caller that describes a segment there (a
  layout marker, a per-column list of what the rows refer to) got a merged
  segment that read as a different kind of segment. The merged segment now
  carries its inputs' metadata, Arrow's schema entry aside, with an input's
  value replacing the writer properties' for the same key. A run stops
  where any input's metadata has different entries from the first's,
  compared without regard to their order, under either SchemaPolicy;
  before, only StopAtChange stopped there, because it compares whole
  Arrow schemas, and UnionFields merged across the change. Segments
  already compacted are not repaired.
• rewrite::project_segment_columns wrote a key twice when the writer
  properties and the segment both carried it. The segment's value now
  replaces the properties', as in compaction.

dendro v0.4.0

Changed

• Breaking: a feature archive, on by default, gates the container and
  everything that reads or writes it: archive's SQLite part, read,
  rewrite, segment, replicate::ArchivePublisher, the Error::Sqlite
  variant, Error::sqlite, Error::sqlite_code and
  From<rusqlite::Error> for Error, and the arrow, bytes, parquet, rusqlite,
  serde_json and tracing dependencies. write implies it. With no features
  the crate is the replication wire (replicate::{Frame, wire, IndexKind, IndexState, NO_INDEX_STATE}, keys, archive::{WalRow, SegmentMeta},
  seal, Error) and has no dependencies. A user that sets
  default-features = false and reads archives adds
  features = ["archive"]. Error::is_retryable and Error::is_constraint
  are false without it.

Fixed

• The docs.rs build. It failed for 0.3.3 and 0.3.4 because nightly removed
  feature(doc_auto_cfg); the crate now enables feature(doc_cfg).

dendro v0.3.4

Added

• A writer heartbeat. sources gains two nullable columns, heartbeat and
  heartbeat_interval_ns (no schema version change, FORMAT.md §3.1 and §8).
  The writer sets the interval when it adds or resumes a source and bumps
  the heartbeat of every source it has open every HEARTBEAT_INTERVAL
  (5 s), whether or not it has rows to commit, until the source is
  finalized. A writer that reopens an older archive adds the columns.
  Copies carry both values unchanged.
• SourceRow::heartbeat and SourceRow::heartbeat_interval_ns, and
  archive::HeartbeatWatch, which turns successive reads of a source into a
  WriterState: Complete, Live, Stopped (the heartbeat has not
  changed for STOPPED_AFTER_INTERVALS = 3 intervals: a killed writer or a
  copy) or Unknown (no heartbeat).
• writer::HEARTBEAT_INTERVAL, archive::STOPPED_AFTER_INTERVALS,
  Transaction::set_heartbeat, Transaction::set_heartbeat_interval,
  Transaction::beat and ArchiveMut::ensure_heartbeat_columns.
  WriterState is #[non_exhaustive]. A resumed source continues its
  heartbeat count; an interval of zero or less reads as Unknown.
• Writer::create_beating_every (feature test-support) chooses the
  heartbeat interval.

dendro v0.3.3

Added

• ColumnFilter::projects(stream), default true: a stream it returns
  false for is copied as it is by a keep_columns pass, sealed segments
  byte for byte and its tail unprojected. For a companion stream whose
  columns a field-by-field filter cannot tell from the data's.

Fixed

• project_segment_columns keeps the source file's key-value metadata
  (all but ARROW:schema, which the writer writes for the projected
  schema). It wrote the projection with only the caller's properties, so a
  segment marked by its caller (a layout, a list of what its rows refer to)
  lost the mark and could read back as another kind of segment.

dendro v0.3.2

Fixed

• Archive::vacuum_into works on the read handle. Archive::open sets
  query_only, and SQLite refused VACUUM INTO under it ("attempt to write
  a readonly database") although the statement writes only the
  destination, so a live archive, which allows a second party no other
  handle, could not be dumped this way. query_only is lifted for the one
  statement and restored on every path; the connection is still
  SQLITE_OPEN_READ_ONLY, so the archive stays unwritable.