Skip to content

dendro v0.4.2

Choose a tag to compare

@github-actions github-actions released this 13 Oct 14:53
· 14 commits to main since this release
62ee3bc

Added

  • SourceWriter::seal_capped(batch, max_rows): a seal that writes no
    segment with more than max_rows WAL rows. A stream with more live rows
    than that is sealed as consecutive segments of max_rows rows, oldest
    first, and a last one with the remainder (7 live rows at a cap of 2 are
    segments of 2, 2, 2 and 1), each with its own catalog entry and seq,
    all in the batch's one transaction. The cap counts WAL rows, as the
    catalog's rows does, not the parquet rows an encoder writes for them.
    Only the row cap is applied: the byte cap and the age bound remain
    triggers for when the caller seals, because the writer does not have the
    byte estimate the caller's SegmentAccount charged per row. A seal that
    is deferred (its transaction could not commit) leaves its rows live, and
    the next capped seal of the stream splits them with the rest.
  • SegmentEncoder::encode_after(stream, before, rows): how a capped seal
    encodes each part after the first, with the stream's earlier live rows
    as before. The default ignores before and calls encode, which is
    right when every row decodes on its own. An encoder whose rows depend on
    earlier ones (a schema carried only by the first row after a seal) must
    implement it before its caller uses seal_capped. A part whose segment
    drops a row at an edge it shares with another part, or that yields no
    segment, abandons the split for that stream, which is then sealed as one
    segment as seal would, so a split never loses a row. That holds only
    for an encoder whose segment reports the rows it wrote: one that claims
    rows it skipped hides the gap from this check.
  • Evicted::held_streams: how many streams a writer retention pass left
    out because their seal was deferred and still could not commit.

Fixed

  • A writer retention pass (SourceWriter::evict_before and its variants)
    deleted the live rows of a stream whose seal had been deferred, the same
    as any other live row past the cutoff. For an encoder whose rows depend
    on an earlier one (a schema stated once at a segment's start), that cut
    the row the survivors needed, and the next seal decoded none of them.
    The writer now remembers which streams' seals deferred, seals them at the
    start of the next pass (retried like any seal), and leaves out of the
    pass any stream it still cannot seal; that stream is evicted by the
    first pass after its seal commits. The stream is left out whole, its
    sealed segments and caller rows included. While a stream is held, a
    whole-source pass runs stream by stream, so a CallerRowFloor is asked
    for every stream name rather than only the names with caller rows. A
    pass after a deferral first retries the deferred streams' seals, each in
    its own transaction, on the writer thread; after the first one that
    still cannot commit (one retry schedule: up to four busy timeouts and
    310 ms) the rest are held without being tried. A retried seal that
    fails with an error that is not retryable, such as an encoder error, is
    not held: it is logged, the pass evicts the
    stream's live rows as it did before, and a Seal of the stream that
    fails the same way stops the writer, as it always has.
  • A seal put every live row of a stream into one segment, so a caller that
    checked SegmentAccount::is_due once after several rows had arrived
    (frames that queued while a slow seal ran) got segments past its
    policy's max_rows: 3- and 4-row segments against a cap of 2 at 10 Hz
    with ~350 ms seals. seal_capped with the policy's max_rows bounds
    them. seal is unchanged.