Skip to content

v0.29.0

Choose a tag to compare

@github-actions github-actions released this 25 Sep 13:30
· 501 commits to main since this release
9df5fd8
  • A multi-table CDC cycle to GCS is 10–13× faster (60 tables, 330k changes,
    measured against 0.28.0 on the same binlog span: 348 s → 35 s on a SQLite ledger,
    376 s → 29 s on Postgres; an idle cycle 115 s → 10 s). The time was not data:
    every roll uploaded each table's part, then rewrote EVERY table's run-unique
    manifest, one PUT at a time. A roll now uploads all tables' parts and writes the
    manifests concurrently (at most 16 at once), and rewrites a table's manifest only
    when it gained a part. Parts and manifests are still durable before the
    checkpoint and the ack. Peak RSS rises with the concurrency (+31% on the 60-table
    stand); rollover_memory_mb bounds the change buffer, not the process.
  • MySQL CDC no longer fails when a flush outlasts the server's
    net_write_timeout
    (60 s by default): the server dropped the binlog dump while
    rivet was uploading and the run ended in CodecError { bytes remaining on stream }
    (nothing was lost; the cycle was wasted). The binlog connection now sets a session
    net_write_timeout of 3600 s.
  • --parallel-exports runs at most 16 exports at once (it started one thread
    per export: a 150-table config opened 150 source and 150 state connections). The
    same pool now takes a lone CDC export's pending baseline snapshots, at most 16 at
    once (a 60-table first cycle: 235 s → 48 s); without the flag they stay one at a
    time. A panicking export or snapshot is reported as that item's failure and the
    others finish (release builds; release-min aborts).
  • GCS: the ADC access token is minted once per identity per process, not once per
    destination (a multi-table CDC run opened one per table, ~0.4 s each).
  • Internals behind the same behaviour (refactor-3): the commit ledger is the only
    writer of a run's counters, and one FanIn owns every parallel runner's tail
    (keyset, chunked, Mongo, parallel checkpoint). Fixes a resume after a failed chunked
    run under-reporting rows (50 of 150) and a Mongo worker panic that lost a durable
    part. rivet run --summary-output into a missing directory, apply stopping at the
    first plan rejection, and a leaked progress channel after a panic are fixed on the way.
  • A local multi-table CDC stream refuses tables that differ only by letter case
    (Orders / orders): on a case-insensitive filesystem (macOS, Windows) their parts
    overwrote each other and the lost rows were permanent while rivet validate passed.
    Cloud prefixes are case-sensitive and keep both.
  • README rewritten around what rivet does: a diagram (sources → rivet → storage →
    warehouse, light and dark), six claims, and three quickstarts — batch, CDC, and the
    full run → load → compact cycle — each run as printed against a live stand.