Repository navigation
v0.30.0
-
Breaking: a source URL whose scheme names another engine is refused.
source.typeis the
engine rivet anchors and types the stream as, and the URL's scheme was never compared with it:
type: mysqlwith apostgresql://URL got as far as the driver. Config load now refuses the
pair withRIVET_CONFIG_SOURCE_URL_SCHEME_MISMATCH(exit 1). A scheme rivet does not know
(jdbc:…) is still left to the driver. -
Breaking: a
cursor_columnorcolumns:key that matches a result column only when case is
ignored is refused. MySQL and SQL Server (case-insensitive collations) run
cursor_column: updatedatagainst a column the result set namesupdatedAt. rivet matched the
name exactly, found nothing, recorded no cursor, and every run re-exported the whole table
with exit 0. Acolumns:override spelled that way was skipped just as silently. Both are now
refused when the schema arrives, naming the exact spelling (RIVET_CONFIG_CURSOR_COLUMN_CASE,
RIVET_CONFIG_COLUMN_OVERRIDE_CASE, exit 1). This applies on every engine. Acolumns:key that
names no column in any case still passes, because multi-table override maps rely on that. -
Breaking: a MySQL keyset key under a
uuidcolumn override is refused. The override reads
BINARY(16)as a 16-byte UUID, but the page cursor was rendered as hyphenated hex text, which
MySQL compares against the raw bytes. Pages skipped and repeated rows while the run reported
success. An explicit or auto-selected keyset key with that override now fails at plan time with
RIVET_CONFIG_KEYSET_KEY_UUID_OVERRIDE(exit 1). PostgreSQL casts the literal touuidand is
unaffected. -
Breaking: a
columns:override whose type the column's value cannot be read as is refused,
not written as NULL. On PostgreSQL and SQL Server, adecimal(p,s)override on an integer
column produced a column that was entirely NULL, and on SQL Server auuidoverride on
nvarchar(36)did the same. The exit was 0 and both value checksums agreed on the NULL. Integers
under adecimaloverride are now scaled exactly, and SQL Serveruuidtext is parsed. Any
other mismatch fails naming the column (RIVET_SOURCE_OVERRIDE_WIRE_MISMATCH, exit 1). On
MySQL, auuidoverride now refuses a value it cannot parse instead of nulling it. -
Breaking: a PostgreSQL URL with a remote
host=orhostaddr=query parameter needs TLS. The
plaintext gate looked only at the URL's authority, while the driver also dials the query's
hostaddr=(in preference to the authority) andhost=. So
postgresql://u:p@localhost/db?hostaddr=203.0.113.5passed as loopback and sent credentials and
rows to the remote address in cleartext. Everyhost=/hostaddr=value must now be loopback
or a unix-socket path, or the connection needs atls:block. -
Breaking: MySQL
TIMEoutside 00:00–24:00 is refused, not written as garbage. A table
holding durations that exported with exit 0 before now fails every run. A MySQLTIMEis a
duration of up to ±838:59:59, and a ParquetTIMEholds one day.838:59:59read back as
'1$:59:59', a negative value made the file unreadable, and unparseable text became NULL, all
with exit 0. Such a value now fails withRIVET_SOURCE_VALUE_UNREPRESENTABLE(exit 5), naming a
cast that keeps it (TIME_TO_SEC(col),CAST(col AS CHAR)). -
Oracle Database as a batch source (preview).
source.type: oraclewith an
oracle://user:password@host:1521/SERVICEURL runsfull,incremental, range- and
keyset-chunked(alsoparallel > 1),time_windowandpartition_byexports. It uses
Oracle's pure-Rust thin driver, so no Oracle client install is needed.rivet initscaffolds
from the schema's catalog and never picks a key or cursor that the planner or the run would
refuse. rivet pins its session to UTC with ISO NLS formats andNLS_SORT = NLS_COMP = BINARY.
ATIMESTAMP(7..9)column is read at microseconds and refused as an incremental cursor
(RIVET_SOURCE_CURSOR_FINER_THAN_MICROSECOND). Known limits (docs/reference/oracle.md): the
driver is a pinned beta behind the default-onoraclecargo feature; TLS verifies only against
the public CA bundle built into the driver (notls.ca_file, no wallet); only 23ai/26ai Free
are tested. -
SQL Server keyset and range exports no longer skip or repeat rows under a non-default
collation or date format. A keyset page bound was sent asN'…', which avarcharkey under
a SQL collation compares by different rules than its ownORDER BY: 67 of 180 keys lost. A
timestamp range bound inYYYY-MM-DD HH:MM:SSform is read day-first under a British login
(DATEFORMAT dmy), and a window exported 39 rows for 24. An ASCII bound is now a plain'…'
literal and timestamps are sent in theTform, which does not depend onDATEFORMAT. -
Timestamp bounds keep their fraction and offset on PostgreSQL and SQL Server. The probe that
reads a range or keyset bound dropped the fraction on both engines, and the+00offset of a
PostgreSQLtimestamptz. Under a session in another zone the bound then named a different
instant (measured on PostgreSQL in Asia/Tokyo: 999 of 1000 rows, then 1914 rows for 1499 ids).
A SQL ServerDATETIMEbound was cut instead of rounded to its 1/300 s tick, and the newest row
stayed behind (199 of 200). -
Every PostgreSQL connection rivet opens runs in UTC with ISO formats. Only the export
transaction was pinned (SET LOCAL TimeZone = 'UTC'). The sampler, the bound probes,check
andinitran in the role's zone andDateStyle, so one run used two zones (the cause of the
Tokyo loss above). Each connection now setsTimeZone,DateStyle,IntervalStyleand
bytea_outputat connect, except behind a transaction-mode pooler, where a sessionSETwould
leak to other clients. Stored values do not move. Calendar arithmetic in your ownquery:
(current_date,ts::dateon atimestamptz) now counts UTC days everywhere, as it already
did inside the export. -
A keyset crash anchor is resumed only by the runner shape that wrote it. After a crash,
changingparallel:made the other runner continue the anchor: 1501 of 2000 rows
(sequential → parallel), or 2600 for 2000 (parallel → sequential). A plain keyset run now
warns and starts a fresh pass. Akeyset_incrementalrun switched toparallel:after a
sequential crash is refused instead, because the sequential cursor had already moved past
pages no manifest names (RIVET_STATE_KEYSET_SEQUENTIAL_ANCHOR_UNFINISHED, exit 5). Re-run
once withparallel: 1. -
--splitunits no longer reuse a stale checkpoint. A fresh--splitre-run re-sampled its
windows but kept each unit's keyset anchor and crashed chunk run, so units resumed against moved
windows: 525000 rows over 450000 ids, or a plan-fingerprint refusal on every run. A unit that
runs over a new window now drops both.rivet state reset-chunks(and--stuck-checkpoints)
now also accept the<export>#<n>and<parent>__<value>names that the chunk refusals print
for split units andpartition_bychildren. -
A chunked resume re-exports a chunk whose part it cannot find. Resuming into a prefix that
held a foreign or stale manifest declared every recorded part without looking, so a part deleted
between attempts was named in aSuccessmanifest. A chunk whose part record was never written
was skipped as completed and left out. Every resume now checks the destination and re-exports
what it cannot account for. -
A chunk that fails and is retried within one run is counted once. Its failed attempt's
parts stayed in theSuccessmanifest beside the retry's, so the chunk's leading rows appeared
twice andtotal_rowscounted them twice. Those parts are now pruned before the manifest is
written. In the parallel checkpoint runner, a chunk that failed and then completed on retry
still failed the run; now the chunk ledger alone decides. When chunks remain unfinished, the
error names each one's last error. -
A MySQL
FLOATkeyset key or cursor no longer re-reads its boundary row. The saved value
was the shortestf32text (0.1), which MySQL compares as aDOUBLEbelow the row's own
0.10000000149011612, so every page boundary and every incremental run re-read the last row.
The exact widened value is saved now. -
A
?or#in a URL password no longer merges different servers into one cursor scope. The
state key cut the URL's query before removing the credentials, so
postgresql://app:k?9@h1/aand…@h2/bboth keyed aspostgres://app:k. The two shared one
incremental cursor and crash anchor, and a prefix of the password was stored in the state DB.
Keys of well-formed URLs are unchanged, so no existing cursor moves. The same password shapes,
and a?password=query parameter, are now redacted fromplan.jsontoo. -
PostgreSQL CDC decodes
DOMAINcolumns instead of writing NULL.test_decodinglabels a
domain cell with the domain's name, so every integer, date or boolean domain column was NULL in
the change log. The reader now resolves each domain to its base type at open (nested domains
included). -
PostgreSQL CDC refuses a value it cannot decode instead of writing NULL. A BC date, a
time '24:00:00'or an unparseable value intest_decodingoutput became NULL with exit 0.
It is now refused, likeinfinityalready was. The reader also pinsTimeZone = 'UTC'. Under a
named server zone, an instant before that zone adopted a whole-minute offset rendered with a
seconds offset (+00:53:28). That was refused on every later run as bad source data. -
SQL Server CDC refuses a captured value it has no decoder for. An
xmlorsql_variant
cell (and any other type without a decoder) was written as NULL with exit 0. It now fails with
RIVET_SOURCE_CDC_CELL_UNSUPPORTED(exit 5). To go on, leave the column out of the capture
instance. -
A DROP and ADD COLUMN of the same arity no longer fills the new column with a neighbour's
value. On PostgreSQL and SQL Server CDC, a missing column was read by position whenever the
change image had as many columns as the schema. AfterDROP a+ADD c, older images put
b's value intoc. They now leavecNULL. The CDC reference no longer claims that a column
added while a run is open is captured: that run's values for it are dropped and acknowledged,
so re-snapshot the table to recover them. -
MySQL CDC writes
YEAR 0000as 0, not 1900, as the batch path does. The binlog decoder
adds 1900 to the stored byte, and 1900 is never a legalYEAR. -
SQL Server CDC advances a quiet table's checkpoint. A table with no changes kept its
checkpoint at its last captured LSN. Once the cleanup job moved the retention floor past that
LSN, the next run failed with a log-gap refusal and asked for a re-snapshot, although nothing
had been lost. After a clean drain, the checkpoint now moves to the bound read at the start of
the run. -
The SQL Server CDC anchor is written in lowercase hex, like every change event's LSN. The
BigQuery and Snowflake dedup views order__pos.lsnas a case-sensitive string, so after a
re-baseline an older event could outrank the new anchor and the view served the stale row.
Anchors already written to a warehouse stay uppercase. -
MySQL CDC fails the resume when it cannot run its
GTID_SUBSETcheck. A failed query read
as "contained" with no warning, which switched off the one identity check that catches a
RESET MASTERon the same server. The query error now fails the resume. A checkpoint with a
GTID set that gets no containment answer resumes with a warning. -
PostgreSQL
time '24:00:00'and atimestampinfinityare refused in batch exports.
24:00:00wrapped to00:00:00with exit 0, and both checksum sides agreed. A
'infinity'::timestamp(without time zone) panicked the process (exit 101, no summary). Both
now fail withRIVET_SOURCE_VALUE_UNREPRESENTABLE(exit 5), naming the column. -
A SQL Server
sql_variantcolumn is refused instead of crashing the export. The driver
cannot parse the type and panicked (exit 101; a release build aborts) on every export or type
probe that touched it. rivet now asks the server for the result's column types first and
refuses withRIVET_SOURCE_VALUE_UNREPRESENTABLE, naming a cast. If that probe is unavailable,
the export runs as before. -
A
rivet loadof CDC base+buffer writes only the newest unloaded baseline. If two
baseline generations were waiting (a snapshot re-run after a crash, or a re-baseline before any
load), both were written into the base: every key twice, with a count gate that grew to match.
The older generation is now recorded as superseded and not read. -
rivet loadno longer disowns its own table after a skip or refusal under the same run id.
An up-to-date skip or a pre-write refusal overwrote the ledger row that proved the table was
rivet's, and the next load refused it as foreign ("drop or rename it"). A load whose source
identity cannot be read from the ledger now refuses before writing. It used to skip the check
that stops one source from replacing another's rows. -
A load pinned to an older run's columns no longer refuses the newer runs it listed. The pin
loads spec-less newer runs with the older run's columns, but it recorded the older run's finish
time, so those same runs were refused as late on every retry. -
A partition column whose name contains
|keeps its partition budget. The footer note split
on every|, sorivet loadcould not parse it and fell back to the row count. It could then
refuse a part the writer had kept within budget. Notes written before the fix still parse. -
An incremental load after an idle run no longer fails. A run that exported nothing after
the first pass made the nextrivet loadfail withno Parquet URIs to append. -
Duplicates found before
unique_max_entriesis reached fail the run. Once a unique column
hit its cap, only the capped warning was emitted: ids[1,1,2,3,4]at cap 3 exited 0 with one
duplicate already proven. It now fails as "at least N duplicate values". -
rivet validate --depth fullno longer reports corruption on healthy exports with NULL
keys. On re-read, Parquet can leave stale values in leading NULL slots, and the keyed
checksum hashed them. MySQL and SQL Server incremental exports whose nullable cursor sorts NULLs
first failed validation. Existing manifests stay valid. -
A run opens far fewer source connections. Each probe (forensics, both harm snapshots,
reconcile, the chunk planner and, on CDC, the schema resolver and row-image checks) opened its
own connection. It now shares one metadata connection per run. When that connection has dropped
during a long run, it is reopened once. Measured per run through a counting proxy against
0.29.0: PostgreSQL full 6 → 2, chunked 7 → 3, keyset 7 → 2; MySQL and SQL Server full 4 → 2,
chunked 5 → 3, keyset 5 → 2. For a bounded CDC run, measured with the server's own counters:
PostgreSQL 10 → 2, MySQL 12 → 2, SQL Server 6 → 2. Each new PostgreSQL backend warms its
catalog cache, so fewer connections also means fewer catalog reads on the source. -
SQL Server chunked and keyset exports work for a
db_datareaderlogin. The row estimate
readsys.dm_db_partition_stats, which needsVIEW DATABASE STATE, so every chunked or keyset
export by such a login failed. It now readssys.partitions. -
A multi-export run exits with the most serious failure's class. Ranking is now integrity >
internal > refusal > schema drift > retryable > usage. A refusal (5) or internal error (6) used
to lose to a retryable failure, so a scheduler retried a protective stop. A combined failure
took its exit class from the text of every other export's error: a permanent failure beside
one mentioning "timeout" exited 2. A child that panicked (exit 101) was passed through as 101;
it now counts as internal (6). A typed stop (integrity, drift, refusal) is never retried because
its message contains a transient-sounding word. -
MongoDB
partition_byis refused when the config is read.rivet checkpassed it, and
rivet runpanicked while expanding partitions, taking every other export in the invocation
with it. NowRIVET_CONFIG_SOURCE_MODE_UNSUPPORTED. -
rivet checkgrades a BigQuerydecimalwith more than 38 integer digits as Fail.
BIGNUMERICholds at most 38 integer digits, sodecimal(50,0), MySQLDECIMAL(65,0)and
PostgreSQLnumeric(76,0)were reported OK, and the load then failed at BigQuery after the
extract had run. -
Discovery reads the keys and indexes that really exist. On SQL Server, a UNIQUE index over
(tenant_id NOT NULL, ext_ref NULL)was taken for a single-column key ontenant_id, so
keyset paging skipped rows at page boundaries with status success. A composite primary key
gave an arbitrary one of its columns as the rangechunk_column.rivet initon PostgreSQL
counted a column as indexed when it sat in any position of any index. Only a column that leads
a valid, complete btree counts now. Preflight no longer applies whole-table statistics to a
partition-extended,APPLY, flashback or dblink read. -
rivet initgives CDC tables whose names collide distinct slots and checkpoints. Names that
reduce to the same identifier (order-itemsandorder_items,Ordersandorders,
non-Latin names) got one checkpoint and one slot, and init's own config was refused with
RIVET_CONFIG_CDC_RESOURCE_CONFLICT. Only colliding names get a hash suffix, so names that
already worked keep their checkpoint and slot. -
Recovery advice that failed when followed has been corrected. The chunked recovery block no
longer chainsreset-chunks && run --resume, which always failed. The rerun warning no longer
sends a completed (_SUCCESS) prefix to--resume, which refuses it. The resume command in the
run report names the failed export and appears only for a checkpointed strategy. The
cursor-owner hint carries the required-c <config>. The PostgreSQL slot-missing refusal lists
every done-signal to clear, not only the checkpoint. The invalid-checkpoint refusals say to
re-anchor first, then re-snapshot. Thepartition_by+load:refusal names only remedies
that work, and the load-adoption refusal saysALTER TABLE … ADD COLUMNinstead of renaming
the table aside, which dropped its rows from the view.--forcewithout--resumeis described
truthfully. -
The plaintext-state warning checks the host. It stayed silent for a remote PostgreSQL state
whose password or database name containedlocalhost. -
rivet doctorandrivet check --type-reporthonour themongo:block, and the MCP
server's MySQL tools accept a URL withsslmode=again. -
Documentation: the exit-code table in
docs/reference/cli.mdlists 5 (refusal) and 6
(internal). The README's CDC quickstart says which sources get a first-run baseline, and the
CDC reference describes the per-transaction caps and opt-in spilling. Thepartition_by
reference lists every combination that config load refuses. -
rivet compactreads the base only for keys whose day moved, not for every key. To find a
key's OLD partition the day probe semi-joined the whole base on every compaction — a full read of
its key and partition columns, 44% of one pilot's BigQuery bill (18.9 GiB per run on its largest
table). The days now come from the buffer first; the base is read only for updated, deleted or
NULL-op (incremental) keys whose row those days do not contain, through the key column
(IN UNNEST) or, past a script variable's size, the old semi-join. Inserted keys are not looked
up. Measured on a 50M-row base: 763 MiB → 46 MiB with no moved key, 102 MiB with one; the day
set is identical. -
A count mismatch on an incremental or CDC load exits 3 (integrity), like a full load. The
append and base+buffer gates raised an uncoded error, so the exit class fell to the transient-
error text match: exit 1 in general, and exit 2 (retry) when the table name contained a
network word such asdns— a scheduler retried a data-integrity failure. Every load-integrity
gate now carriesRIVET_LOAD_COUNT_MISMATCH, and an incremental base+buffer load no longer
labels its failure "CDC". -
MySQL CDC writes a zero
TIMESTAMP(0000-00-00 00:00:00) as NULL, not1970-01-01. The
binlog carries a zero TIMESTAMP as epoch 0, and the CDC decoder turned it into the epoch instant,
while the batch path (and so every snapshot) wrote NULL: the same source value landed two ways in
one table, and a row that CDC touched after its snapshot silently changed from NULL to 1970. No
real TIMESTAMP can be 0 — MySQL's range starts at1970-01-01 00:00:01UTC — so 0 is the zero
date and is now NULL on both paths. Found on a pilot comparing rivet with a cursor pipeline.
Rows already loaded keep the epoch until they change again or the table is re-snapshotted. -
A load whose state ledger dies before it can mark a table as being written no longer writes it.
Thewritingmarker that keeps a table rivet's own across a crash was written best-effort: when
the ledger connection dropped first, the load wrote the table anyway and reportedLOAD OK, and
the next load refused that table as foreign for ever. The marker is now required — without it the
load fails before the warehouse write, and the next load finishes the job. -
rivet initon Oracle keysets a table whose only key is a UNIQUE NOT NULL column. It looked
only for a primary key, so such a table (a common shape) fell to a full scan while the same table
on MySQL — which reports that key asPRI— was keysetted. A single-column UNIQUE key on a NOT
NULL column now stands in when there is no primary key, and is recorded as the load key. -
A Postgres state behind pgBouncer (transaction mode) keeps one run of an export at a
time (state v31). The run lease of a checkpointed export and the per-table lease of
rivet loadwere session advisory locks. Behind a transaction-mode pooler two rivets
could be handed the same server connection and both take the lock: two runs of one
export ran together (measured: both exit 0). The lease is now astate_leaserow taken in
one statement, renewed by a heartbeat and expired by the state server's clock. A holder
that died on the same host is taken over at once; one that died on another host blocks
the next run until its lease expires (RIVET_STATE_LEASE_TTL_S, default 30 s). A SQLite
state is unchanged. The previous release cannot open a v31 state DB. -
Oracle CDC (preview), through LogMiner.
mode: cdcon an Oracle source now mines the
redo logs fromCDB$ROOTfor the configured tables of one pluggable database
(ADR-0037), with no GoldenGate licence. A run drains to the SCN current at its start and
resumes from a checkpoint of two SCNs, so a transaction open across two runs arrives whole.
The checkpoint records the database, its incarnation and the PDB, and a run refuses one
from elsewhere. A checkpoint that needs a deleted archive is refused as a loss. Each captured
table needsADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; a table without it, or with a type
the preview does not capture (LOBs among them), is refused by name. ATRUNCATEof a captured table or partition is refused once the
changes before it are delivered (RIVET_SOURCE_CDC_TRUNCATED, exit 5), and every re-run
stops there until you re-anchor and re-snapshot. Continuous capture (until_current: false,
rivet cdc --stream) is refused (RIVET_CONFIG_CDC_CONTINUOUS_UNSUPPORTED), and an Oracle
CDC export under aload:block is refused when the config is read. -
On a shared Postgres state, the state server stamps when a run started. Runs were
ranked by the writer's own clock, so a run that crashed on a host whose clock ran ahead
outranked the run that replaced it, andrivet load --gc-orphanskept treating the prefix
as still being written until that clock caught up (cleanup deferred, nothing lost). The
start is nowclock_timestamp()on the state server; a SQLite state keeps the local
clock, which is the only one it has. -
A timestamp ClickHouse cannot hold is refused, not clamped. ClickHouse
DateTime64holds
1900-01-01 to 2299-12-31, and reading Parquet it clamps anything past that to the nearest end
with no error (measured on 24.8: 9999-12-31 stored as 2299-12-31 23:00; the
date_time_overflow_behaviorsetting does not reach the Parquet reader).rivet loadnow reads
each part's footer statistics and refuses a part holding such a value before inserting anything
(RIVET_LOAD_VALUE_OUT_OF_TARGET_RANGE), or a timestamp column the footer cannot bound (no
min/max statistics); an all-NULL row group no longer hides a value past the range in another. A load pulled through a named collection is not
inspected yet; the type report says so. -
A MongoDB CDC export under a ClickHouse
load:is refused when the config is read. It was
refused only byrivet load, afterrivet runhad extracted the whole stream. -
The type report no longer claims a bare
Decimalfor a negative-scale decimal on ClickHouse
(ClickHouse reads a bareDecimalasDecimal(10, 0)): it is declaredDecimal(p+|s|, 0),
which holds the whole numbers exactly, and refused past precision 76. -
Two configs sharing a Postgres state DB no longer read each other's incremental cursor
(state v30). The cursor and the keyset crash anchor were keyed by the export name alone.
Two configs with the same export name on one state DB — one template rolled out per
database — shared one row. The second config continued from the first one's cursor and
exported nothing, exit 0 (measured: 0 of 100 rows). Both are now keyed by the SOURCE too
(engine, host, port and database — never credentials), so a destination that changes
between runs, a{date}path for one, keeps its cursor. A cursor written before v30 is
continued by the first run of that export and by no other config. The previous release
cannot open a v30 state DB; it says so and writes nothing. -
Breaking: two new exit codes, and failures carry a registered code. A protective stop,
where rivet refuses so as not to lose, duplicate or overwrite data, now exits 5. A
broken invariant, which is a bug, exits 6. Codes 1–4 keep their meanings. Every coded
failure names a stableRIVET_<FAMILY>_<NAME>code with a kind (usage, environment,
refusal, integrity, internal) and one line saying what to do. The code appears in
--json-errorsascodeand as a[CODE]prefix. The full list is
docs/reference/errors.md, generated from the registry (rivet schema errors). The first
codes are a newer state DB, a foreign cursor, an Oracle cursor finer than a microsecond, a
load count mismatch, a table the change log cannot adopt, and the internal spill and
value-converter failures. The rest move over a
batch at a time; an offline ratchet keeps the uncoded ones from growing. A scheduler that
treated every non-zero exit as "fix the config" should now treat 5 as "a human decides"
and 6 as "report it". -
The release gate now grades what rivet costs the source database. It compared wall time
and memory with the previous release, never the load on the source. A new stage exports the
same table with both binaries on every engine and fails if rows scanned or temp spills grow
past the previous release. Harm now also records PostgreSQL logical-decoding spill and decoded
bytes and SQL Server logical reads and tempdb work tables. A live test proves on every engine
that a CDC drain reads the change log and not the table. -
max_file_sizeholds on parquet exports again (#307). Parquet writes bytes only when
a row group closes, and arivet initconfig sizes row groups automatically (~128 MB), so a
256 KB cap produced one 1.1 MB part with no warning. Row groups are now capped at a quarter
ofmax_file_sizeand a batch is written a group at a time, so parts rotate within about a
quarter of the cap.parquet.row_group_rowsunder the defaultautostrategy is now said to
be ignored, namingfixed_rows. Enforcing the cap made a keyset page rotate into several
parts, which exposed a resume that duplicated the finished parts of a page interrupted
mid-rotation when the cap changed between the crash and the resume; such a page is now
re-read whole. -
Breaking:
chunk_denseis removed. It paged byROW_NUMBER() OVER (ORDER BY chunk_column), recomputed per chunk, so a concurrent insert or delete skipped or
duplicated rows even on a unique key. Nobody used it. A config that still sets
chunk_dense: trueis refused at load with a pointer tochunk_by_key(keyset) or
chunk_columnrange chunking;chunk_dense: falseis still accepted. A chunked
plan.jsonwritten by an older rivet failsapply's integrity check; re-run
rivet plan. -
CDC keeps each change part inside a load job's partition budget. Under the
log_view
layout the change log is partitioned, and BigQuery writes at most 4,000 partitions per load
job. The CDC drain wrote each flush as one part with no partition count in its footer, so a
flush of changes to rows scattered over more than 4,000 days could be refused at load. The
drain now cuts a flush at the budget and notes each part, like the batch writer (verified
on MySQL, PostgreSQL and SQL Server). MongoDB's change parts carry no date column, and the
run now says so instead of shipping them unbudgeted in silence. -
Chunked exports record their partition count in every part again. The chunked,
parallel-chunked and parallel-checkpoint runners closed each part without the
rivet.partition_bucketsfooter note, sorivet loadfell back to the part's min/max
date span — a wider bound than the part really holds, which can refuse a load that fits.
Every runner now closes parts the same way, and a live test reads every part's footer. -
SQL Server CDC refuses a checkpoint from another database or from before a restore.
The checkpoint held only an LSN, which every database accepts: pointed at another server's
database, a run resumed there, exited 0 and captured nothing, skipping that database's
changes. The checkpoint now records the database'sfamily_guidandrecovery_fork_guid.
rivet refuses a different family (another database) or a changed fork (aRESTORE), and
still follows an availability-group failover, whose replicas share both. -
A CDC run sends its notifications and says which batch settings it ignores. A failed CDC
run never reachednotifications:, so anon: [failure]webhook stayed silent while the
stream was down. It now fires like a batch run's.quality:,parquet:,compression*,
max_file_size,shape_drift_warn_factorandskip_emptyapply only to a CDC export's snapshot baseline,
andrun --validate/--reconcilelikewise check only the baseline; each run now warns,
naming the ones that were set. A CDC run now also records the source server it ran against
(even when it fails) and the source-harm counters of its window, with the same DIAGNOSIS line
a batch run prints. -
The release gate grades what it used to wave through. A run that bypasses the
run-integrity facade only warns in a release build, and the gate now fails on that warning.
Withoutcargo-llvm-covthe offline battery still runs and is graded instead of skipping. -
Reading CDC from a replica is verified on every engine. Live tests and release-gate rows
now cover a PostgreSQL 16 standby (continuous mode; the default bounded mode still refuses,
and its message no longer claims a standby cannot host a slot), a SQL Server read-scale
availability group's readable secondary, and a MongoDB secondary, beside MySQL's re-logging
replica. The CDC reference says, per engine, what the replica needs and what proves it. -
MySQL CDC refuses a replica that does not re-log what it applies. With
log_replica_updates = OFF(MySQL's default) a replica's binlog holds none of the primary's
changes; reading it captured nothing and exited 0. rivet now refuses such a replica at start.
MySQL only: the other engines have no such setting to get wrong. -
SQL Server incremental exports work on a legacy
DATETIMEcursor. The saved boundary was
rendered with six fractional digits, whichDATETIMErejects, so every run after the first
failed withConversion failed. A rivet-rendered timestamp cursor is now typedDATETIME2(7),
and aDATETIMEvalue is read to its nearest microsecond (its 1/300 s tick has no exact one) in
batch and CDC alike, so the boundary row is not re-exported. -
rivet compactcarries a column the source gained into the base. After anADD COLUMN
at the source, the BigQuery MERGE named the new column while the base lacked it and failed
with a rawUnrecognized name, on this and every later compact. The base now gets
ADD COLUMN IF NOT EXISTSbefore any MERGE. -
CDC refuses a captured table or collection that was dropped. MySQL skipped
DROP TABLE
and MongoDB skippeddrop/rename/dropDatabase, so the destination kept the removed rows
live and a table re-created under the same name continued the old history as one table. Both
now fail naming the object and the recovery order; the same DDL on something nobody captures
is still skipped. -
MySQL CDC refuses a change to a captured table that was logged as a SQL statement. A
writer session onbinlog_format=STATEMENT(or MIXED) put its DML in the binlog as text,
which the row reader skipped: the run exited 0 with the change missing. It now fails naming
the table and the fix; the same statement on a table nobody captures is still skipped. -
Parallel keyset incremental no longer skips text keys the source collation ranks higher.
Whether anything lay past the anchor was decided by a byte compare in rivet while the source
orders by its collation: on anen_USdatabase 500 new keysC…after an anchorb…read as
"nothing new" and the run exported 0 of them with status success. The source now answers the
question itself (MAX(key) … WHERE key > anchor). -
cleanup_sourceno longer deletes parts an extract committed during the load. The load
checked for a running extract before appending, then deleted the whole prefix after it; an
extract that started and committed in between lost its parts after the source position had
moved past them (measured: 51 of 52 rows reached BigQuery). Cleanup now deletes exactly the
files of runs already loaded (or superseded, on a full load) that nobody is writing. -
on_schema_driftnow applies to CDC exports. A CDC run never consulted it: a column
retyped between runs wrote parts with different types into one prefix and exited 0 under
fail. Each captured table's schema is now checked before the stream reads a change, so a
refusal acknowledges nothing and switching towarncaptures every deferred change. -
rivet loadinto ClickHouse (preview) (ADR-0035; the loader started from
@ssyusyukalov's #145).load: { target: clickhouse, url, database, user, password_env }, orrivet init --clickhouse-url … --clickhouse-database …. A
CDC table lands in<table>__changes, aReplacingMergeTreekeyed on the primary
key with a version decoded from the source position (PostgreSQL LSN, MySQL binlog
file + offset, SQL Server LSN), behind aFINALview that flags deletes, so the
log collapses itself and needs norivet compact. Full loads swap in a filled
table; incremental loads get a change log and a latest-cursor view. By default
rivet sends each part over HTTP; withnamed_collection:ClickHouse reads the
bucket itself. The export may be staged on GCS, S3 or Azure. Not supported:
MongoDB CDC,partition:,layout: base_buffer. -
destination.oneshot_budget_mb(by @ssyusyukalov, #143): the RAM cap on
single-PUT uploads, until now fixed at 64 MB, is configurable. On GCS and Azure a
part that fits uploads in one PUT and gets a store-computedContent-MD5that
validatechecks; larger parts stream with size-only verification (S3 is always
size-only, but a single PUT still saves the sequential multipart round trips:
measured 3–4.5× faster per part at 50 ms RTT).0streams every part. The cap
is shared by every destination with the same value, including each table of a
CDC export, so it never multiplies by table count. -
The one-shot upload budget no longer grows with every streamed part. A part
too big for the budget was refused, but the refusal credited its size back to the
pool, so after one large streamed part the next one was read into memory whole:
the 64 MB cap was not a cap (since 2026-06). Measured on MinIO and Azurite with a
budget of 0: the first 10.8 MB part streamed, the next two went up as single PUTs.