Repository navigation
v0.31.0
-
A resumed chunked run with no tasks is no longer an empty success. A checkpointed chunked
run wrote its run row and its task rows in two writes. When the task write failed, the run row
stayed behind with no tasks; the next plain run resumed it, found nothing to do, exited 0 and
delivered 0 of 150 rows (measured). The run row and its tasks are now written in one
transaction, so a run opens with its tasks or not at all. -
A
keyset_incrementalexport loads as an append, not an overwrite. Each run of such an
export delivers only the keys past its anchor, butrivet loadtreated it as a full load and
replaced the warehouse table with the newest delta. Measured on a PostgreSQL source (ids
1..100, then 101..120, then no change): BigQuery and ClickHouse held 100, then 20, then 20
rows. It now loads as an incremental load ordered bychunk_by_key. A table that an earlier
release already overwrote is not repaired by the upgrade; it holds only the last delta until
the source is re-read in full. -
CDC refuses a value its column type cannot hold instead of writing NULL. The CDC sink
appended NULL when a change event's value did not fit its column (integers, Boolean, floats,
dates, timestamps, times, binary). Only a source NULL is NULL now; any other mismatch fails the
flush and names the column. -
The PostgreSQL batch reader refuses an undecodable cell instead of panicking or writing
NULL. A text or enum cell that is not valid UTF-8 fails with
RIVET_SOURCE_VALUE_UNREPRESENTABLEnaming the column (text used to panic, enum wrote NULL).
Integer overrides decode at the wire width and widen exactly; a narrowing value that does not
fit is refused by name. -
Oracle CDC re-mines when the redo log set changes under LogMiner. A log switch that reused
or archived an online redo file LogMiner had registered failed the run with ORA-01368 behind a
permissions hint, and a sequence archived between two catalog reads could be refused as lost.
The run now re-plans the logs and restarts mining after the last queued commit group, five
times with a doubling backoff, before failing with a message that names the cause. -
Oracle CDC no longer pins its low-water mark at SCN 0. A transaction with a slot but no
start SCN yet showsSTART_SCN = 0; the anchor took it as the minimum, and the next run
planned redo from SCN 0 and refused with a false "changes LOST". -
CDC refuses an unmet prerequisite before the anchor and the snapshot write. Under
initial: snapshotthe first run pinned its anchor and wrote the snapshot part, manifest and
marker before the engine's refusals ran, so a source that cannot be captured was refused only
after a part and a checkpoint were on disk. The refusals now run first (MySQL: row metadata,
binlog compression, replica re-log, routing; Oracle: container, table vetting, supplemental
logging; PostgreSQL: routing). -
The PostgreSQL missing-slot refusal names the evidence it found. When the replication slot
is gone but an earlier run left a checkpoint, a completed snapshot, or both, the refusal now
says which, and gives the recovery steps for that case. -
rivet init --mode cdcscaffolds a config that runs. SQL Server, MongoDB, a single table
and a non-publicPostgreSQL schema got a stream with no baseline under a base-and-buffer load
block, so the firstrivet compactrefused; Oracle got aload:block every command refuses.
Every CDC scaffold now takes a baseline, and none carries a load block the loader refuses. -
ClickHouse loads accept
partition:.partition: { column, granularity }maps to
PARTITION BYon the full-load table and both change logs. A part ClickHouse pulls is
footer-checked for the DateTime64 range before insert (it used to be stored clamped),
statements that are safe to repeat are retried up to five times, andload.ca_filesets the
CA bundle. -
Oracle batch: override decoders, case refusal at
check, non-Unicode national charsets.
Row decoders for int2, date, timestamp_ns, uuid, float8 and text overrides;rivet check
refuses a wrong-casecolumns:key as the run does; NVARCHAR2, NCHAR and NCLOB round-trip
byte-exact on a WE8ISO8859P1 database. -
The CDC sink builds exactly the planned column type. A column the CDC builder cannot build
is planned as server text and labelled so in the Parquet metadata; aDate32column refuses a
time of day; integer and float overrides accept decimal text through one parser. -
New:
rivet state vacuum. It shrinks a SQLite state database (VACUUM plus a truncating
checkpoint) and prints the size before and after; on a PostgreSQL state it does nothing.
Measured on a pilot state copy: 46.8 MB to 18.8 MB, with 32 concurrent runs on the same file
and no failed run. -
State schema v32. A run's load spec is stored once per distinct version instead of once
per run (a pilot state held 9,029 run rows for 154 distinct specs). The migration runs on the
first command; binaries older than this release refuse a v32 state, so upgrade every host
that shares a state together. The PostgreSQL migration replays safely over a partly migrated
schema, and the PostgreSQL state reads every integer width (a resumed parallel keyset run
could panic on an INT4). -
Dependencies.
mysql28.0.3 (28.0.0 was yanked),mongodb3.9.1 (a change-stream loop
and two CVEs),tiberius0.13 (SQL Serversql_variantand CLR types read as text).
cargo auditis clean. -
MySQL CDC
compactreads the base only in its MERGE.rivet compactmerges only the
partitions its changes name. To catch a row whose partition value changed, it used to look
every updated key up in the base on every cycle; a pilot paid 2 x 2.2 GiB per large table
per cycle for that, mostly for keys inserted and updated in the same cycle, and no row
there ever moved. Now the partitions come from the buffer alone. An UPDATE that moves a row
between two partitions of a base-and-buffer table is written byrivet runas a delete of
the old row and an insert of the new one, with a warning naming both values: the delete
carries the old partition into the buffer, so the MERGE reads it and the base keeps one row,
in its new partition. A change to or from NULL is written as an update, as before. -
Breaking: MySQL CDC refuses an UPDATE without the row's previous value on a partitioned
base-and-buffer table. Without it (binlog_row_imagenot FULL) the old partition is
unknown, so the run fails withRIVET_SOURCE_CDC_PREREQUISITEnaming
SET PERSIST binlog_row_image = FULL;, before any part is written. Other engines, and the
change-log layout, are unchanged. -
rivet compactno longer fails on a table that was never loaded. A source table that was
empty from the start loads nothing, so it has neither a base nor a__changesbuffer in the
warehouse;compactstill ranALTER TABLEon the base before looking for the buffer, and
BigQuery answeredNot found: Tableon every cycle, failing the cycle's exit status for the
other tables too (five of 153 tables in a pilot run). The buffer is now checked first, and such a
table is theCOMPACT SKIPit always should have been. -
A full load of an emptied source empties the warehouse table again. Since the change below,
amode: fullrun that exported 0 rows was consumed as "up to date" and the warehouse kept rows
that no longer exist in the source. That run now reaches the full load's replace, which leaves
the table empty (BigQuery, ClickHouse; Snowflake refuses loudly, as before). Empty incremental and
CDC runs are still consumed as a 0-row success. -
rivet loadconsumes the extraction runs that produced no files. An idle CDC cycle (no
changes) writes a manifest with no parts; the load skipped it but never recorded it, so every
later load re-read every such manifest again: a pilot with 110 idle tables read ~50 manifests per
table per cycle, one more each cycle. Such runs are now recorded as a 0-row success, and the
next load reads only the runs since. Measured on the stand over three idle cycles: 1, 2, 3
manifests re-read before; 1, 1, 1 after. -
Breaking: MySQL CDC refuses
binlog_row_metadata = MINIMAL. It used to warn and map binlog
values by position, so a column reorder orALTERdelivered them under the wrong names (source
(1, b='BBB', a='AAA')arrived asb='AAA', a='BBB'). A server that is not at FULL, or that
predates the variable (MySQL before 8.0.1), now fails the run at open with
RIVET_SOURCE_CDC_PREREQUISITE, before a first run pins its anchor. The message names
SET PERSIST binlog_row_metadata = FULL;. A rows event written under MINIMAL is refused when read,
even after the server is switched to FULL, withRIVET_SOURCE_CDC_UNDECODABLEnaming the table
and the binlog position. No part is written and the checkpoint does not move. Recovery: delete the
checkpoint so the next run anchors afresh first, then re-snapshot.rivet doctorfails the "CDC
binlog server config" check for the same settings. The CDC sink no longer maps any nameless image
by position. -
rivet runapplies the type policy before it reads data. A column whose mapping islossy
orunsupportedused to be reported only byrivet check --type-report, and the run wrote it
without a word. Batch and CDC runs, includingrivet cdc --output, now log one warning per
such column before the first row is read. The warning has the wordingcheckprints. The new
rivet run --strictrefuses the run instead, before any part or checkpoint is written, and
names every such column withRIVET_TYPE_UNSAFE_MAPPING(exit 1).rivet check --type-report --strictnow refuses with the same code, socheckandrunagree on the verdict.