You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Added
SourceWriter::seal_capped(batch, max_rows): a seal that writes no
segment with more than max_rows WAL rows. A stream with more live rows
than that is sealed as consecutive segments of max_rows rows, oldest
first, and a last one with the remainder (7 live rows at a cap of 2 are
segments of 2, 2, 2 and 1), each with its own catalog entry and seq,
all in the batch's one transaction. The cap counts WAL rows, as the
catalog's rows does, not the parquet rows an encoder writes for them.
Only the row cap is applied: the byte cap and the age bound remain
triggers for when the caller seals, because the writer does not have the
byte estimate the caller's SegmentAccount charged per row. A seal that
is deferred (its transaction could not commit) leaves its rows live, and
the next capped seal of the stream splits them with the rest.
SegmentEncoder::encode_after(stream, before, rows): how a capped seal
encodes each part after the first, with the stream's earlier live rows
as before. The default ignores before and calls encode, which is
right when every row decodes on its own. An encoder whose rows depend on
earlier ones (a schema carried only by the first row after a seal) must
implement it before its caller uses seal_capped. A part whose segment
drops a row at an edge it shares with another part, or that yields no
segment, abandons the split for that stream, which is then sealed as one
segment as seal would, so a split never loses a row. That holds only
for an encoder whose segment reports the rows it wrote: one that claims
rows it skipped hides the gap from this check.
Evicted::held_streams: how many streams a writer retention pass left
out because their seal was deferred and still could not commit.
Fixed
A writer retention pass (SourceWriter::evict_before and its variants)
deleted the live rows of a stream whose seal had been deferred, the same
as any other live row past the cutoff. For an encoder whose rows depend
on an earlier one (a schema stated once at a segment's start), that cut
the row the survivors needed, and the next seal decoded none of them.
The writer now remembers which streams' seals deferred, seals them at the
start of the next pass (retried like any seal), and leaves out of the
pass any stream it still cannot seal; that stream is evicted by the
first pass after its seal commits. The stream is left out whole, its
sealed segments and caller rows included. While a stream is held, a
whole-source pass runs stream by stream, so a CallerRowFloor is asked
for every stream name rather than only the names with caller rows. A
pass after a deferral first retries the deferred streams' seals, each in
its own transaction, on the writer thread; after the first one that
still cannot commit (one retry schedule: up to four busy timeouts and
310 ms) the rest are held without being tried. A retried seal that
fails with an error that is not retryable, such as an encoder error, is
not held: it is logged, the pass evicts the
stream's live rows as it did before, and a Seal of the stream that
fails the same way stops the writer, as it always has.
A seal put every live row of a stream into one segment, so a caller that
checked SegmentAccount::is_due once after several rows had arrived
(frames that queued while a slow seal ran) got segments past its
policy's max_rows: 3- and 4-row segments against a cap of 2 at 10 Hz
with ~350 ms seals. seal_capped with the policy's max_rows bounds
them. seal is unchanged.