Skip to content

v0.30.0

Choose a tag to compare

@github-actions github-actions released this 29 Sep 19:39
· 446 commits to main since this release
216de2d
  • Breaking: a source URL whose scheme names another engine is refused. source.type is the
    engine rivet anchors and types the stream as, and the URL's scheme was never compared with it:
    type: mysql with a postgresql:// URL got as far as the driver. Config load now refuses the
    pair with RIVET_CONFIG_SOURCE_URL_SCHEME_MISMATCH (exit 1). A scheme rivet does not know
    (jdbc:…) is still left to the driver.

  • Breaking: a cursor_column or columns: key that matches a result column only when case is
    ignored is refused.
    MySQL and SQL Server (case-insensitive collations) run
    cursor_column: updatedat against a column the result set names updatedAt. rivet matched the
    name exactly, found nothing, recorded no cursor, and every run re-exported the whole table
    with exit 0. A columns: override spelled that way was skipped just as silently. Both are now
    refused when the schema arrives, naming the exact spelling (RIVET_CONFIG_CURSOR_COLUMN_CASE,
    RIVET_CONFIG_COLUMN_OVERRIDE_CASE, exit 1). This applies on every engine. A columns: key that
    names no column in any case still passes, because multi-table override maps rely on that.

  • Breaking: a MySQL keyset key under a uuid column override is refused. The override reads
    BINARY(16) as a 16-byte UUID, but the page cursor was rendered as hyphenated hex text, which
    MySQL compares against the raw bytes. Pages skipped and repeated rows while the run reported
    success. An explicit or auto-selected keyset key with that override now fails at plan time with
    RIVET_CONFIG_KEYSET_KEY_UUID_OVERRIDE (exit 1). PostgreSQL casts the literal to uuid and is
    unaffected.

  • Breaking: a columns: override whose type the column's value cannot be read as is refused,
    not written as NULL.
    On PostgreSQL and SQL Server, a decimal(p,s) override on an integer
    column produced a column that was entirely NULL, and on SQL Server a uuid override on
    nvarchar(36) did the same. The exit was 0 and both value checksums agreed on the NULL. Integers
    under a decimal override are now scaled exactly, and SQL Server uuid text is parsed. Any
    other mismatch fails naming the column (RIVET_SOURCE_OVERRIDE_WIRE_MISMATCH, exit 1). On
    MySQL, a uuid override now refuses a value it cannot parse instead of nulling it.

  • Breaking: a PostgreSQL URL with a remote host= or hostaddr= query parameter needs TLS. The
    plaintext gate looked only at the URL's authority, while the driver also dials the query's
    hostaddr= (in preference to the authority) and host=. So
    postgresql://u:p@localhost/db?hostaddr=203.0.113.5 passed as loopback and sent credentials and
    rows to the remote address in cleartext. Every host=/hostaddr= value must now be loopback
    or a unix-socket path, or the connection needs a tls: block.

  • Breaking: MySQL TIME outside 00:00–24:00 is refused, not written as garbage. A table
    holding durations that exported with exit 0 before now fails every run. A MySQL TIME is a
    duration of up to ±838:59:59, and a Parquet TIME holds one day. 838:59:59 read back as
    '1$:59:59', a negative value made the file unreadable, and unparseable text became NULL, all
    with exit 0. Such a value now fails with RIVET_SOURCE_VALUE_UNREPRESENTABLE (exit 5), naming a
    cast that keeps it (TIME_TO_SEC(col), CAST(col AS CHAR)).

  • Oracle Database as a batch source (preview). source.type: oracle with an
    oracle://user:password@host:1521/SERVICE URL runs full, incremental, range- and
    keyset-chunked (also parallel > 1), time_window and partition_by exports. It uses
    Oracle's pure-Rust thin driver, so no Oracle client install is needed. rivet init scaffolds
    from the schema's catalog and never picks a key or cursor that the planner or the run would
    refuse. rivet pins its session to UTC with ISO NLS formats and NLS_SORT = NLS_COMP = BINARY.
    A TIMESTAMP(7..9) column is read at microseconds and refused as an incremental cursor
    (RIVET_SOURCE_CURSOR_FINER_THAN_MICROSECOND). Known limits (docs/reference/oracle.md): the
    driver is a pinned beta behind the default-on oracle cargo feature; TLS verifies only against
    the public CA bundle built into the driver (no tls.ca_file, no wallet); only 23ai/26ai Free
    are tested.

  • SQL Server keyset and range exports no longer skip or repeat rows under a non-default
    collation or date format.
    A keyset page bound was sent as N'…', which a varchar key under
    a SQL collation compares by different rules than its own ORDER BY: 67 of 180 keys lost. A
    timestamp range bound in YYYY-MM-DD HH:MM:SS form is read day-first under a British login
    (DATEFORMAT dmy), and a window exported 39 rows for 24. An ASCII bound is now a plain '…'
    literal and timestamps are sent in the T form, which does not depend on DATEFORMAT.

  • Timestamp bounds keep their fraction and offset on PostgreSQL and SQL Server. The probe that
    reads a range or keyset bound dropped the fraction on both engines, and the +00 offset of a
    PostgreSQL timestamptz. Under a session in another zone the bound then named a different
    instant (measured on PostgreSQL in Asia/Tokyo: 999 of 1000 rows, then 1914 rows for 1499 ids).
    A SQL Server DATETIME bound was cut instead of rounded to its 1/300 s tick, and the newest row
    stayed behind (199 of 200).

  • Every PostgreSQL connection rivet opens runs in UTC with ISO formats. Only the export
    transaction was pinned (SET LOCAL TimeZone = 'UTC'). The sampler, the bound probes, check
    and init ran in the role's zone and DateStyle, so one run used two zones (the cause of the
    Tokyo loss above). Each connection now sets TimeZone, DateStyle, IntervalStyle and
    bytea_output at connect, except behind a transaction-mode pooler, where a session SET would
    leak to other clients. Stored values do not move. Calendar arithmetic in your own query:
    (current_date, ts::date on a timestamptz) now counts UTC days everywhere, as it already
    did inside the export.

  • A keyset crash anchor is resumed only by the runner shape that wrote it. After a crash,
    changing parallel: made the other runner continue the anchor: 1501 of 2000 rows
    (sequential → parallel), or 2600 for 2000 (parallel → sequential). A plain keyset run now
    warns and starts a fresh pass. A keyset_incremental run switched to parallel: after a
    sequential crash is refused instead, because the sequential cursor had already moved past
    pages no manifest names (RIVET_STATE_KEYSET_SEQUENTIAL_ANCHOR_UNFINISHED, exit 5). Re-run
    once with parallel: 1.

  • --split units no longer reuse a stale checkpoint. A fresh --split re-run re-sampled its
    windows but kept each unit's keyset anchor and crashed chunk run, so units resumed against moved
    windows: 525000 rows over 450000 ids, or a plan-fingerprint refusal on every run. A unit that
    runs over a new window now drops both. rivet state reset-chunks (and --stuck-checkpoints)
    now also accept the <export>#<n> and <parent>__<value> names that the chunk refusals print
    for split units and partition_by children.

  • A chunked resume re-exports a chunk whose part it cannot find. Resuming into a prefix that
    held a foreign or stale manifest declared every recorded part without looking, so a part deleted
    between attempts was named in a Success manifest. A chunk whose part record was never written
    was skipped as completed and left out. Every resume now checks the destination and re-exports
    what it cannot account for.

  • A chunk that fails and is retried within one run is counted once. Its failed attempt's
    parts stayed in the Success manifest beside the retry's, so the chunk's leading rows appeared
    twice and total_rows counted them twice. Those parts are now pruned before the manifest is
    written. In the parallel checkpoint runner, a chunk that failed and then completed on retry
    still failed the run; now the chunk ledger alone decides. When chunks remain unfinished, the
    error names each one's last error.

  • A MySQL FLOAT keyset key or cursor no longer re-reads its boundary row. The saved value
    was the shortest f32 text (0.1), which MySQL compares as a DOUBLE below the row's own
    0.10000000149011612, so every page boundary and every incremental run re-read the last row.
    The exact widened value is saved now.

  • A ? or # in a URL password no longer merges different servers into one cursor scope. The
    state key cut the URL's query before removing the credentials, so
    postgresql://app:k?9@h1/a and …@h2/b both keyed as postgres://app:k. The two shared one
    incremental cursor and crash anchor, and a prefix of the password was stored in the state DB.
    Keys of well-formed URLs are unchanged, so no existing cursor moves. The same password shapes,
    and a ?password= query parameter, are now redacted from plan.json too.

  • PostgreSQL CDC decodes DOMAIN columns instead of writing NULL. test_decoding labels a
    domain cell with the domain's name, so every integer, date or boolean domain column was NULL in
    the change log. The reader now resolves each domain to its base type at open (nested domains
    included).

  • PostgreSQL CDC refuses a value it cannot decode instead of writing NULL. A BC date, a
    time '24:00:00' or an unparseable value in test_decoding output became NULL with exit 0.
    It is now refused, like infinity already was. The reader also pins TimeZone = 'UTC'. Under a
    named server zone, an instant before that zone adopted a whole-minute offset rendered with a
    seconds offset (+00:53:28). That was refused on every later run as bad source data.

  • SQL Server CDC refuses a captured value it has no decoder for. An xml or sql_variant
    cell (and any other type without a decoder) was written as NULL with exit 0. It now fails with
    RIVET_SOURCE_CDC_CELL_UNSUPPORTED (exit 5). To go on, leave the column out of the capture
    instance.

  • A DROP and ADD COLUMN of the same arity no longer fills the new column with a neighbour's
    value.
    On PostgreSQL and SQL Server CDC, a missing column was read by position whenever the
    change image had as many columns as the schema. After DROP a + ADD c, older images put
    b's value into c. They now leave c NULL. The CDC reference no longer claims that a column
    added while a run is open is captured: that run's values for it are dropped and acknowledged,
    so re-snapshot the table to recover them.

  • MySQL CDC writes YEAR 0000 as 0, not 1900, as the batch path does. The binlog decoder
    adds 1900 to the stored byte, and 1900 is never a legal YEAR.

  • SQL Server CDC advances a quiet table's checkpoint. A table with no changes kept its
    checkpoint at its last captured LSN. Once the cleanup job moved the retention floor past that
    LSN, the next run failed with a log-gap refusal and asked for a re-snapshot, although nothing
    had been lost. After a clean drain, the checkpoint now moves to the bound read at the start of
    the run.

  • The SQL Server CDC anchor is written in lowercase hex, like every change event's LSN. The
    BigQuery and Snowflake dedup views order __pos.lsn as a case-sensitive string, so after a
    re-baseline an older event could outrank the new anchor and the view served the stale row.
    Anchors already written to a warehouse stay uppercase.

  • MySQL CDC fails the resume when it cannot run its GTID_SUBSET check. A failed query read
    as "contained" with no warning, which switched off the one identity check that catches a
    RESET MASTER on the same server. The query error now fails the resume. A checkpoint with a
    GTID set that gets no containment answer resumes with a warning.

  • PostgreSQL time '24:00:00' and a timestamp infinity are refused in batch exports.
    24:00:00 wrapped to 00:00:00 with exit 0, and both checksum sides agreed. A
    'infinity'::timestamp (without time zone) panicked the process (exit 101, no summary). Both
    now fail with RIVET_SOURCE_VALUE_UNREPRESENTABLE (exit 5), naming the column.

  • A SQL Server sql_variant column is refused instead of crashing the export. The driver
    cannot parse the type and panicked (exit 101; a release build aborts) on every export or type
    probe that touched it. rivet now asks the server for the result's column types first and
    refuses with RIVET_SOURCE_VALUE_UNREPRESENTABLE, naming a cast. If that probe is unavailable,
    the export runs as before.

  • A rivet load of CDC base+buffer writes only the newest unloaded baseline. If two
    baseline generations were waiting (a snapshot re-run after a crash, or a re-baseline before any
    load), both were written into the base: every key twice, with a count gate that grew to match.
    The older generation is now recorded as superseded and not read.

  • rivet load no longer disowns its own table after a skip or refusal under the same run id.
    An up-to-date skip or a pre-write refusal overwrote the ledger row that proved the table was
    rivet's, and the next load refused it as foreign ("drop or rename it"). A load whose source
    identity cannot be read from the ledger now refuses before writing. It used to skip the check
    that stops one source from replacing another's rows.

  • A load pinned to an older run's columns no longer refuses the newer runs it listed. The pin
    loads spec-less newer runs with the older run's columns, but it recorded the older run's finish
    time, so those same runs were refused as late on every retry.

  • A partition column whose name contains | keeps its partition budget. The footer note split
    on every |, so rivet load could not parse it and fell back to the row count. It could then
    refuse a part the writer had kept within budget. Notes written before the fix still parse.

  • An incremental load after an idle run no longer fails. A run that exported nothing after
    the first pass made the next rivet load fail with no Parquet URIs to append.

  • Duplicates found before unique_max_entries is reached fail the run. Once a unique column
    hit its cap, only the capped warning was emitted: ids [1,1,2,3,4] at cap 3 exited 0 with one
    duplicate already proven. It now fails as "at least N duplicate values".

  • rivet validate --depth full no longer reports corruption on healthy exports with NULL
    keys.
    On re-read, Parquet can leave stale values in leading NULL slots, and the keyed
    checksum hashed them. MySQL and SQL Server incremental exports whose nullable cursor sorts NULLs
    first failed validation. Existing manifests stay valid.

  • A run opens far fewer source connections. Each probe (forensics, both harm snapshots,
    reconcile, the chunk planner and, on CDC, the schema resolver and row-image checks) opened its
    own connection. It now shares one metadata connection per run. When that connection has dropped
    during a long run, it is reopened once. Measured per run through a counting proxy against
    0.29.0: PostgreSQL full 6 → 2, chunked 7 → 3, keyset 7 → 2; MySQL and SQL Server full 4 → 2,
    chunked 5 → 3, keyset 5 → 2. For a bounded CDC run, measured with the server's own counters:
    PostgreSQL 10 → 2, MySQL 12 → 2, SQL Server 6 → 2. Each new PostgreSQL backend warms its
    catalog cache, so fewer connections also means fewer catalog reads on the source.

  • SQL Server chunked and keyset exports work for a db_datareader login. The row estimate
    read sys.dm_db_partition_stats, which needs VIEW DATABASE STATE, so every chunked or keyset
    export by such a login failed. It now reads sys.partitions.

  • A multi-export run exits with the most serious failure's class. Ranking is now integrity >
    internal > refusal > schema drift > retryable > usage. A refusal (5) or internal error (6) used
    to lose to a retryable failure, so a scheduler retried a protective stop. A combined failure
    took its exit class from the text of every other export's error: a permanent failure beside
    one mentioning "timeout" exited 2. A child that panicked (exit 101) was passed through as 101;
    it now counts as internal (6). A typed stop (integrity, drift, refusal) is never retried because
    its message contains a transient-sounding word.

  • MongoDB partition_by is refused when the config is read. rivet check passed it, and
    rivet run panicked while expanding partitions, taking every other export in the invocation
    with it. Now RIVET_CONFIG_SOURCE_MODE_UNSUPPORTED.

  • rivet check grades a BigQuery decimal with more than 38 integer digits as Fail.
    BIGNUMERIC holds at most 38 integer digits, so decimal(50,0), MySQL DECIMAL(65,0) and
    PostgreSQL numeric(76,0) were reported OK, and the load then failed at BigQuery after the
    extract had run.

  • Discovery reads the keys and indexes that really exist. On SQL Server, a UNIQUE index over
    (tenant_id NOT NULL, ext_ref NULL) was taken for a single-column key on tenant_id, so
    keyset paging skipped rows at page boundaries with status success. A composite primary key
    gave an arbitrary one of its columns as the range chunk_column. rivet init on PostgreSQL
    counted a column as indexed when it sat in any position of any index. Only a column that leads
    a valid, complete btree counts now. Preflight no longer applies whole-table statistics to a
    partition-extended, APPLY, flashback or dblink read.

  • rivet init gives CDC tables whose names collide distinct slots and checkpoints. Names that
    reduce to the same identifier (order-items and order_items, Orders and orders,
    non-Latin names) got one checkpoint and one slot, and init's own config was refused with
    RIVET_CONFIG_CDC_RESOURCE_CONFLICT. Only colliding names get a hash suffix, so names that
    already worked keep their checkpoint and slot.

  • Recovery advice that failed when followed has been corrected. The chunked recovery block no
    longer chains reset-chunks && run --resume, which always failed. The rerun warning no longer
    sends a completed (_SUCCESS) prefix to --resume, which refuses it. The resume command in the
    run report names the failed export and appears only for a checkpointed strategy. The
    cursor-owner hint carries the required -c <config>. The PostgreSQL slot-missing refusal lists
    every done-signal to clear, not only the checkpoint. The invalid-checkpoint refusals say to
    re-anchor first, then re-snapshot. The partition_by + load: refusal names only remedies
    that work, and the load-adoption refusal says ALTER TABLE … ADD COLUMN instead of renaming
    the table aside, which dropped its rows from the view. --force without --resume is described
    truthfully.

  • The plaintext-state warning checks the host. It stayed silent for a remote PostgreSQL state
    whose password or database name contained localhost.

  • rivet doctor and rivet check --type-report honour the mongo: block, and the MCP
    server's MySQL tools accept a URL with sslmode= again.

  • Documentation: the exit-code table in docs/reference/cli.md lists 5 (refusal) and 6
    (internal).
    The README's CDC quickstart says which sources get a first-run baseline, and the
    CDC reference describes the per-transaction caps and opt-in spilling. The partition_by
    reference lists every combination that config load refuses.

  • rivet compact reads the base only for keys whose day moved, not for every key. To find a
    key's OLD partition the day probe semi-joined the whole base on every compaction — a full read of
    its key and partition columns, 44% of one pilot's BigQuery bill (18.9 GiB per run on its largest
    table). The days now come from the buffer first; the base is read only for updated, deleted or
    NULL-op (incremental) keys whose row those days do not contain, through the key column
    (IN UNNEST) or, past a script variable's size, the old semi-join. Inserted keys are not looked
    up. Measured on a 50M-row base: 763 MiB → 46 MiB with no moved key, 102 MiB with one; the day
    set is identical.

  • A count mismatch on an incremental or CDC load exits 3 (integrity), like a full load. The
    append and base+buffer gates raised an uncoded error, so the exit class fell to the transient-
    error text match: exit 1 in general, and exit 2 (retry) when the table name contained a
    network word such as dns — a scheduler retried a data-integrity failure. Every load-integrity
    gate now carries RIVET_LOAD_COUNT_MISMATCH, and an incremental base+buffer load no longer
    labels its failure "CDC".

  • MySQL CDC writes a zero TIMESTAMP (0000-00-00 00:00:00) as NULL, not 1970-01-01. The
    binlog carries a zero TIMESTAMP as epoch 0, and the CDC decoder turned it into the epoch instant,
    while the batch path (and so every snapshot) wrote NULL: the same source value landed two ways in
    one table, and a row that CDC touched after its snapshot silently changed from NULL to 1970. No
    real TIMESTAMP can be 0 — MySQL's range starts at 1970-01-01 00:00:01 UTC — so 0 is the zero
    date and is now NULL on both paths. Found on a pilot comparing rivet with a cursor pipeline.
    Rows already loaded keep the epoch until they change again or the table is re-snapshotted.

  • A load whose state ledger dies before it can mark a table as being written no longer writes it.
    The writing marker that keeps a table rivet's own across a crash was written best-effort: when
    the ledger connection dropped first, the load wrote the table anyway and reported LOAD OK, and
    the next load refused that table as foreign for ever. The marker is now required — without it the
    load fails before the warehouse write, and the next load finishes the job.

  • rivet init on Oracle keysets a table whose only key is a UNIQUE NOT NULL column. It looked
    only for a primary key, so such a table (a common shape) fell to a full scan while the same table
    on MySQL — which reports that key as PRI — was keysetted. A single-column UNIQUE key on a NOT
    NULL column now stands in when there is no primary key, and is recorded as the load key.

  • A Postgres state behind pgBouncer (transaction mode) keeps one run of an export at a
    time (state v31).
    The run lease of a checkpointed export and the per-table lease of
    rivet load were session advisory locks. Behind a transaction-mode pooler two rivets
    could be handed the same server connection and both take the lock: two runs of one
    export ran together (measured: both exit 0). The lease is now a state_lease row taken in
    one statement, renewed by a heartbeat and expired by the state server's clock. A holder
    that died on the same host is taken over at once; one that died on another host blocks
    the next run until its lease expires (RIVET_STATE_LEASE_TTL_S, default 30 s). A SQLite
    state is unchanged. The previous release cannot open a v31 state DB.

  • Oracle CDC (preview), through LogMiner. mode: cdc on an Oracle source now mines the
    redo logs from CDB$ROOT for the configured tables of one pluggable database
    (ADR-0037), with no GoldenGate licence. A run drains to the SCN current at its start and
    resumes from a checkpoint of two SCNs, so a transaction open across two runs arrives whole.
    The checkpoint records the database, its incarnation and the PDB, and a run refuses one
    from elsewhere. A checkpoint that needs a deleted archive is refused as a loss. Each captured
    table needs ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; a table without it, or with a type
    the preview does not capture (LOBs among them), is refused by name. A TRUNCATE of a captured table or partition is refused once the
    changes before it are delivered (RIVET_SOURCE_CDC_TRUNCATED, exit 5), and every re-run
    stops there until you re-anchor and re-snapshot. Continuous capture (until_current: false,
    rivet cdc --stream) is refused (RIVET_CONFIG_CDC_CONTINUOUS_UNSUPPORTED), and an Oracle
    CDC export under a load: block is refused when the config is read.

  • On a shared Postgres state, the state server stamps when a run started. Runs were
    ranked by the writer's own clock, so a run that crashed on a host whose clock ran ahead
    outranked the run that replaced it, and rivet load --gc-orphans kept treating the prefix
    as still being written until that clock caught up (cleanup deferred, nothing lost). The
    start is now clock_timestamp() on the state server; a SQLite state keeps the local
    clock, which is the only one it has.

  • A timestamp ClickHouse cannot hold is refused, not clamped. ClickHouse DateTime64 holds
    1900-01-01 to 2299-12-31, and reading Parquet it clamps anything past that to the nearest end
    with no error (measured on 24.8: 9999-12-31 stored as 2299-12-31 23:00; the
    date_time_overflow_behavior setting does not reach the Parquet reader). rivet load now reads
    each part's footer statistics and refuses a part holding such a value before inserting anything
    (RIVET_LOAD_VALUE_OUT_OF_TARGET_RANGE), or a timestamp column the footer cannot bound (no
    min/max statistics); an all-NULL row group no longer hides a value past the range in another. A load pulled through a named collection is not
    inspected yet; the type report says so.

  • A MongoDB CDC export under a ClickHouse load: is refused when the config is read. It was
    refused only by rivet load, after rivet run had extracted the whole stream.

  • The type report no longer claims a bare Decimal for a negative-scale decimal on ClickHouse
    (ClickHouse reads a bare Decimal as Decimal(10, 0)): it is declared Decimal(p+|s|, 0),
    which holds the whole numbers exactly, and refused past precision 76.

  • Two configs sharing a Postgres state DB no longer read each other's incremental cursor
    (state v30).
    The cursor and the keyset crash anchor were keyed by the export name alone.
    Two configs with the same export name on one state DB — one template rolled out per
    database — shared one row. The second config continued from the first one's cursor and
    exported nothing, exit 0 (measured: 0 of 100 rows). Both are now keyed by the SOURCE too
    (engine, host, port and database — never credentials), so a destination that changes
    between runs, a {date} path for one, keeps its cursor. A cursor written before v30 is
    continued by the first run of that export and by no other config. The previous release
    cannot open a v30 state DB; it says so and writes nothing.

  • Breaking: two new exit codes, and failures carry a registered code. A protective stop,
    where rivet refuses so as not to lose, duplicate or overwrite data, now exits 5. A
    broken invariant, which is a bug, exits 6. Codes 1–4 keep their meanings. Every coded
    failure names a stable RIVET_<FAMILY>_<NAME> code with a kind (usage, environment,
    refusal, integrity, internal) and one line saying what to do. The code appears in
    --json-errors as code and as a [CODE] prefix. The full list is
    docs/reference/errors.md, generated from the registry (rivet schema errors). The first
    codes are a newer state DB, a foreign cursor, an Oracle cursor finer than a microsecond, a
    load count mismatch, a table the change log cannot adopt, and the internal spill and
    value-converter failures. The rest move over a
    batch at a time; an offline ratchet keeps the uncoded ones from growing. A scheduler that
    treated every non-zero exit as "fix the config" should now treat 5 as "a human decides"
    and 6 as "report it".

  • The release gate now grades what rivet costs the source database. It compared wall time
    and memory with the previous release, never the load on the source. A new stage exports the
    same table with both binaries on every engine and fails if rows scanned or temp spills grow
    past the previous release. Harm now also records PostgreSQL logical-decoding spill and decoded
    bytes and SQL Server logical reads and tempdb work tables. A live test proves on every engine
    that a CDC drain reads the change log and not the table.

  • max_file_size holds on parquet exports again (#307). Parquet writes bytes only when
    a row group closes, and a rivet init config sizes row groups automatically (~128 MB), so a
    256 KB cap produced one 1.1 MB part with no warning. Row groups are now capped at a quarter
    of max_file_size and a batch is written a group at a time, so parts rotate within about a
    quarter of the cap. parquet.row_group_rows under the default auto strategy is now said to
    be ignored, naming fixed_rows. Enforcing the cap made a keyset page rotate into several
    parts, which exposed a resume that duplicated the finished parts of a page interrupted
    mid-rotation when the cap changed between the crash and the resume; such a page is now
    re-read whole.

  • Breaking: chunk_dense is removed. It paged by ROW_NUMBER() OVER (ORDER BY chunk_column), recomputed per chunk, so a concurrent insert or delete skipped or
    duplicated rows even on a unique key. Nobody used it. A config that still sets
    chunk_dense: true is refused at load with a pointer to chunk_by_key (keyset) or
    chunk_column range chunking; chunk_dense: false is still accepted. A chunked
    plan.json written by an older rivet fails apply's integrity check; re-run
    rivet plan.

  • CDC keeps each change part inside a load job's partition budget. Under the log_view
    layout the change log is partitioned, and BigQuery writes at most 4,000 partitions per load
    job. The CDC drain wrote each flush as one part with no partition count in its footer, so a
    flush of changes to rows scattered over more than 4,000 days could be refused at load. The
    drain now cuts a flush at the budget and notes each part, like the batch writer (verified
    on MySQL, PostgreSQL and SQL Server). MongoDB's change parts carry no date column, and the
    run now says so instead of shipping them unbudgeted in silence.

  • Chunked exports record their partition count in every part again. The chunked,
    parallel-chunked and parallel-checkpoint runners closed each part without the
    rivet.partition_buckets footer note, so rivet load fell back to the part's min/max
    date span — a wider bound than the part really holds, which can refuse a load that fits.
    Every runner now closes parts the same way, and a live test reads every part's footer.

  • SQL Server CDC refuses a checkpoint from another database or from before a restore.
    The checkpoint held only an LSN, which every database accepts: pointed at another server's
    database, a run resumed there, exited 0 and captured nothing, skipping that database's
    changes. The checkpoint now records the database's family_guid and recovery_fork_guid.
    rivet refuses a different family (another database) or a changed fork (a RESTORE), and
    still follows an availability-group failover, whose replicas share both.

  • A CDC run sends its notifications and says which batch settings it ignores. A failed CDC
    run never reached notifications:, so an on: [failure] webhook stayed silent while the
    stream was down. It now fires like a batch run's. quality:, parquet:, compression*,
    max_file_size, shape_drift_warn_factor and skip_empty apply only to a CDC export's snapshot baseline,
    and run --validate / --reconcile likewise check only the baseline; each run now warns,
    naming the ones that were set. A CDC run now also records the source server it ran against
    (even when it fails) and the source-harm counters of its window, with the same DIAGNOSIS line
    a batch run prints.

  • The release gate grades what it used to wave through. A run that bypasses the
    run-integrity facade only warns in a release build, and the gate now fails on that warning.
    Without cargo-llvm-cov the offline battery still runs and is graded instead of skipping.

  • Reading CDC from a replica is verified on every engine. Live tests and release-gate rows
    now cover a PostgreSQL 16 standby (continuous mode; the default bounded mode still refuses,
    and its message no longer claims a standby cannot host a slot), a SQL Server read-scale
    availability group's readable secondary, and a MongoDB secondary, beside MySQL's re-logging
    replica. The CDC reference says, per engine, what the replica needs and what proves it.

  • MySQL CDC refuses a replica that does not re-log what it applies. With
    log_replica_updates = OFF (MySQL's default) a replica's binlog holds none of the primary's
    changes; reading it captured nothing and exited 0. rivet now refuses such a replica at start.
    MySQL only: the other engines have no such setting to get wrong.

  • SQL Server incremental exports work on a legacy DATETIME cursor. The saved boundary was
    rendered with six fractional digits, which DATETIME rejects, so every run after the first
    failed with Conversion failed. A rivet-rendered timestamp cursor is now typed DATETIME2(7),
    and a DATETIME value is read to its nearest microsecond (its 1/300 s tick has no exact one) in
    batch and CDC alike, so the boundary row is not re-exported.

  • rivet compact carries a column the source gained into the base. After an ADD COLUMN
    at the source, the BigQuery MERGE named the new column while the base lacked it and failed
    with a raw Unrecognized name, on this and every later compact. The base now gets
    ADD COLUMN IF NOT EXISTS before any MERGE.

  • CDC refuses a captured table or collection that was dropped. MySQL skipped DROP TABLE
    and MongoDB skipped drop/rename/dropDatabase, so the destination kept the removed rows
    live and a table re-created under the same name continued the old history as one table. Both
    now fail naming the object and the recovery order; the same DDL on something nobody captures
    is still skipped.

  • MySQL CDC refuses a change to a captured table that was logged as a SQL statement. A
    writer session on binlog_format=STATEMENT (or MIXED) put its DML in the binlog as text,
    which the row reader skipped: the run exited 0 with the change missing. It now fails naming
    the table and the fix; the same statement on a table nobody captures is still skipped.

  • Parallel keyset incremental no longer skips text keys the source collation ranks higher.
    Whether anything lay past the anchor was decided by a byte compare in rivet while the source
    orders by its collation: on an en_US database 500 new keys C… after an anchor b… read as
    "nothing new" and the run exported 0 of them with status success. The source now answers the
    question itself (MAX(key) … WHERE key > anchor).

  • cleanup_source no longer deletes parts an extract committed during the load. The load
    checked for a running extract before appending, then deleted the whole prefix after it; an
    extract that started and committed in between lost its parts after the source position had
    moved past them (measured: 51 of 52 rows reached BigQuery). Cleanup now deletes exactly the
    files of runs already loaded (or superseded, on a full load) that nobody is writing.

  • on_schema_drift now applies to CDC exports. A CDC run never consulted it: a column
    retyped between runs wrote parts with different types into one prefix and exited 0 under
    fail. Each captured table's schema is now checked before the stream reads a change, so a
    refusal acknowledges nothing and switching to warn captures every deferred change.

  • rivet load into ClickHouse (preview) (ADR-0035; the loader started from
    @ssyusyukalov's #145). load: { target: clickhouse, url, database, user, password_env }, or rivet init --clickhouse-url … --clickhouse-database …. A
    CDC table lands in <table>__changes, a ReplacingMergeTree keyed on the primary
    key with a version decoded from the source position (PostgreSQL LSN, MySQL binlog
    file + offset, SQL Server LSN), behind a FINAL view that flags deletes, so the
    log collapses itself and needs no rivet compact. Full loads swap in a filled
    table; incremental loads get a change log and a latest-cursor view. By default
    rivet sends each part over HTTP; with named_collection: ClickHouse reads the
    bucket itself. The export may be staged on GCS, S3 or Azure. Not supported:
    MongoDB CDC, partition:, layout: base_buffer.

  • destination.oneshot_budget_mb (by @ssyusyukalov, #143): the RAM cap on
    single-PUT uploads, until now fixed at 64 MB, is configurable. On GCS and Azure a
    part that fits uploads in one PUT and gets a store-computed Content-MD5 that
    validate checks; larger parts stream with size-only verification (S3 is always
    size-only, but a single PUT still saves the sequential multipart round trips:
    measured 3–4.5× faster per part at 50 ms RTT). 0 streams every part. The cap
    is shared by every destination with the same value, including each table of a
    CDC export, so it never multiplies by table count.

  • The one-shot upload budget no longer grows with every streamed part. A part
    too big for the budget was refused, but the refusal credited its size back to the
    pool, so after one large streamed part the next one was read into memory whole:
    the 64 MB cap was not a cap (since 2026-06). Measured on MinIO and Azurite with a
    budget of 0: the first 10.8 MB part streamed, the next two went up as single PUTs.