You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
A multi-table CDC cycle to GCS is 10–13× faster (60 tables, 330k changes,
measured against 0.28.0 on the same binlog span: 348 s → 35 s on a SQLite ledger,
376 s → 29 s on Postgres; an idle cycle 115 s → 10 s). The time was not data:
every roll uploaded each table's part, then rewrote EVERY table's run-unique
manifest, one PUT at a time. A roll now uploads all tables' parts and writes the
manifests concurrently (at most 16 at once), and rewrites a table's manifest only
when it gained a part. Parts and manifests are still durable before the
checkpoint and the ack. Peak RSS rises with the concurrency (+31% on the 60-table
stand); rollover_memory_mb bounds the change buffer, not the process.
MySQL CDC no longer fails when a flush outlasts the server's net_write_timeout (60 s by default): the server dropped the binlog dump while
rivet was uploading and the run ended in CodecError { bytes remaining on stream }
(nothing was lost; the cycle was wasted). The binlog connection now sets a session net_write_timeout of 3600 s.
--parallel-exports runs at most 16 exports at once (it started one thread
per export: a 150-table config opened 150 source and 150 state connections). The
same pool now takes a lone CDC export's pending baseline snapshots, at most 16 at
once (a 60-table first cycle: 235 s → 48 s); without the flag they stay one at a
time. A panicking export or snapshot is reported as that item's failure and the
others finish (release builds; release-min aborts).
GCS: the ADC access token is minted once per identity per process, not once per
destination (a multi-table CDC run opened one per table, ~0.4 s each).
Internals behind the same behaviour (refactor-3): the commit ledger is the only
writer of a run's counters, and one FanIn owns every parallel runner's tail
(keyset, chunked, Mongo, parallel checkpoint). Fixes a resume after a failed chunked
run under-reporting rows (50 of 150) and a Mongo worker panic that lost a durable
part. rivet run --summary-output into a missing directory, apply stopping at the
first plan rejection, and a leaked progress channel after a panic are fixed on the way.
A local multi-table CDC stream refuses tables that differ only by letter case
(Orders / orders): on a case-insensitive filesystem (macOS, Windows) their parts
overwrote each other and the lost rows were permanent while rivet validate passed.
Cloud prefixes are case-sensitive and keep both.
README rewritten around what rivet does: a diagram (sources → rivet → storage →
warehouse, light and dark), six claims, and three quickstarts — batch, CDC, and the
full run → load → compact cycle — each run as printed against a live stand.