Repository navigation
v0.32.0
Upgrading from 0.31.x or earlier — read these first:
- Exit codes follow the code's class. A coded refusal exits 5, an integrity stop 3, an internal
error 6, a usage error 1, an environment fault 2 when it is transient and 1 otherwise. Several
stops that exited 1 now exit 5; each is named in its entry below. A scheduler that retries on any
non-zero exit should stop retrying on 5. - The state schema moves from v32 to v37 on the first run. Before migrating, rivet now leaves a
copy of the old state (.rivet_state.db.v32.<stamp>.bakbeside a SQLite state, schema
rivet_state_backup_v32in a PostgreSQL state) and the way back to the previous release is in
docs/reference/state.md, "Rolling back an upgrade". If the copy cannot be written, the
migration does not run (RIVET_STATE_BACKUP_FAILED). - A relative
destination.pathis anchored to the config file's directory, not the working
directory. A run started from another directory while the export's files still sit under the old
place is refused (RIVET_DEST_RELATIVE_PATH_MOVED) until the files are moved or the path is made
absolute. - A dropped source table is refused
RIVET_SOURCE_TABLE_NOT_FOUNDon every SQL engine; remove
the export fromexports:if the table was dropped on purpose.
Known issue in this release (fixed in 0.32.1):
-
rivet loadof a batch export reads the manifests under its whole prefix, including another
export's nested under it.rivet initon SQL Server writes a batch prefixexports/<t>/and a
CDC prefixexports/<t>/cdc/; with both in one bucket,rivet loadof the batch export takes the
stream's snapshot and change parts as its own and applies their deletes to the batch table
(exit 0). Present since at least 0.31.0. What to do: never place one export's prefix inside
another's; in a configrivet initgenerated for SQL Server, move the CDC export'sprefixto a
sibling (for examplecdc/<t>/) before the first run. -
A plan applied after UTC midnight wrote under the planning day.
rivet planresolved
{date}in the destination into the sealed plan, so a plan made before midnight and applied
after it wrote under the planning day while the data and the part files were of the apply
(P-26, sincerivet planexists). Now the sealed plan keeps{date}as written and
rivet applyresolves it to the UTC day it runs on, through the same resolverrivet run
uses: a plan made before midnight UTC and applied after now writes under the apply date.
rivet plan --date YYYY-MM-DDfreezes that day into the plan instead. A plan sealed by an
earlier release holds an already-resolved path and applies as it did. What to do:
nothing; pass--datetorivet planwhen the planning day is the one wanted. -
rivet applyanswers a replay of an applied plan in the summary document, and gains--summary-output/--json.
A replay of a full, chunked or keyset plan that already applied to exit 0 (0.31.0, #616) answered
only in prose on stderr: no summary file, no JSON line, so a scheduler could not tell "already
applied by run X" from "succeeded, result missing" without parsing text. Nowrivet apply
takes--summary-output <PATH>and--jsonlikerivet run(in both plan-artifact and config
modes), and the replay writes the same document a run writes, built from the run that applied the
plan: itsrun_id, rows, files and bytes from itsexport_metricsrow,status: "already_applied"
andanswered_by_run: "<run_id>". Everyper_export[]entry now carriesanswered_by_run
(nullunless the entry answers a replay); the stderr line is unchanged, the exit stays 0, and
the replay still writes no run, manifest or metrics row. ADR-0039 lists the new status word and
key. What to do: a scheduler readsstatuson the entry; astatustable that treats an
unknown word as failed needsalready_appliedadded. -
An upgrade could not be rolled back: the first run of a newer rivet migrated the state schema in
place, and the previous release then refused it (RIVET_STATE_SCHEMA_NEWER, exit 5) with no copy
of what it had written. Release 0.31.0 knows schema v32; this release moves it to v37 (0.31.0
and earlier). Now a migration first writes a copy of the state as the previous release left it,
inside the migration lock and before its first statement: on SQLite the file
<state>.v<old>.<UTC time>.bakbeside the state (.rivet_state.db.v32.20261017T101530Z.bak), on
PostgreSQL the schemarivet_state_backup_v<old>in the state database, every state table with its
rows, keys, indexes and defaults, in the migration's own transaction. One log line names the copy.
A copy that cannot be written (disk full, a read-only directory, a role withoutCREATEon the
database) stops the migration withRIVET_STATE_BACKUP_FAILED(exit 1); the state is untouched and
the previous release still opens it. What to do: nothing to upgrade. To roll back, follow
Rolling back an upgrade: stop the schedulers,
install the previous release, put the copy back (SQLite: replace the file; PostgreSQL: the SQL
block there) and run once. Progress written after the upgrade is lost, so the rows delivered since
are delivered again, at least once; what that means per mode and forrivet loadis written there. -
Behaviour change: a replay of a sealed plan that already applied re-ran it, and an incremental plan could be applied only once.
rivet apply plan.jsonrun again after that plan's apply had exited 0 (a scheduler retry whose
task process died after rivet finished) extracted the whole table a second time formode: full,
chunked and keyset plans. An incremental plan made after a run had stored a cursor was refused
RIVET_PLAN_ARTIFACT_STALE(exit 5, "cursor has drifted") on every apply after its first, by its
own cursor commit, and any plan older than 24 h was refused (0.31.0 and earlier). Now a
full, chunked or keyset plan is a point-in-time snapshot: the run that applies it to exit 0
records the plan'splan_id(state schema v37,run_status.plan_id), and a replay of it exits 0
without extracting, printingapply: plan <plan_id> of export '<export>' was already applied by run <run_id> (...)with that run's rows, files and bytes; it writes no run, manifest or metrics
row. An incremental plan (mode: incremental,keyset_incremental) is a reusable description:
every apply of it, a retry included, reads the cursor from state and delivers what arrived since,
new rows or none; a cursor moved by an earlier apply of the same plan is not drift, and the 24 h
limit no longer applies to it. A cursor moved by another run is still refused
RIVET_PLAN_ARTIFACT_STALE, and a plan whose apply failed, was interrupted or died still
replays. What to do: nothing for a retry. To extract a full, chunked or keyset plan again,
make a new plan withrivet plan, or pass--force. Plans applied before the upgrade recorded no
plan_id, so a replay of one behaves as before. -
An export whose source table had been dropped failed with the raw driver error and no code.
rivet runover a table dropped since the config was generated ended with the engine's own words
as itsError:line and noRIVET_*code on MySQL and PostgreSQL (Error: MySqlError { ERROR 1146 (42S02): Table 'rivet.drop_probe_b' doesn't exist }, exit 1), orRIVET_SOURCE_QUERY_FAILED
on SQL Server and Oracle;rivet checkrefused the same table asRIVET_SOURCE_QUERY_FAILED
(0.30.0 and 0.31.0). Now one function recognises each engine's own verdict (MySQL 1146,
PostgreSQL SQLSTATE 42P01, SQL Server 208, Oracle ORA-00942) on every path that reads the table
(the batch read, the parallel runners, the chunk planner's probe, the CDC snapshot leg,check
andplan), and the export stops asError: [RIVET_SOURCE_TABLE_NOT_FOUND] export 'drop_probe_b': the table it reads does not exist (drop_probe_b, read by itsquery:). Correct the export'stable:orquery:, or remove the export fromexports:if the table was dropped on purpose: MySqlError { ERROR 1146 (42S02): Table 'rivet.drop_probe_b' doesn't exist }(exit 1). The exit
stays 1 and the other exports of the same run still deliver (1 success · 1 failed), so alerting
on the exit code sees no change;rivet plankeepsRIVET_PLAN_SOURCE_UNREADABLEas its outer
code, as it does for a missing MongoDB collection, with this refusal as the cause. What to do:
remove the export fromexports:if the table was dropped on purpose, or correct itstable:. -
A relative
destination.pathfollowed the working directory, not the config. A local
export whosepathis relative (./output/orders, whatrivet initwrites) was resolved
against the directory rivet was started from, while its cursor and state live beside the
config: the same config run from a cron entry and from a shell delivered ids 1..10 under one
directory and 11..13 under the other, each run green (P-27). Now a relative localpath
is anchored to the config file's directory, resolved once at config load and used the same
way byrun,plan/apply,validate,reconcile,repair,state,doctorand
load; a cursor stored under the old relative spelling continues under the anchored path.
Transition guard: a command started from another directory, whose working-directory
place still holds the export's files (itsmanifest.json) while the directory beside the
config holds none, is refused withRIVET_DEST_RELATIVE_PATH_MOVED(exit 5) naming both
directories, before anything is read or written; the check is one decision in the config
loader, sorun,load,reconcile,repair,validateand every other command that
resolves the path meet it, and it costs local file reads only. What to do: nothing for
a run started from the config's directory, where the path resolves to the same place as
before. After the refusal, move the files to the named directory beside the config (the
stored cursor continues there), or makedestination.pathabsolute to keep the old location. -
reconcileandrepairread a stored chunk run on an editedchunk_column. Neither
command compared the chunk plan the stored run was planned under with the config, so after a
chunk_column(orquery,chunk_size,chunk_count,chunk_by_days) edit they re-read the
stored windows on the new column, and arepair --executesuperseded parts: the manifest listed
39 rows for 40 source ids, exit 0 (P-21). Now both commands run the checkrivet run --resumealready runs and refuse withRIVET_STATE_CHUNK_PLAN_CHANGED(exit 5) before any window
is read, naming the remedy. What to do: restore the chunk settings the run had to reconcile or
repair it, orrivet state reset-chunks -c <config> --export <name>abandons it and the next run
starts over. Exit codes: areconcileorrepairunder a changed chunk plan exited 0 or 2 and
exits 5 now. -
Behaviour change: a Snowflake full load emptied its table before the COPY. The load ran
CREATE OR REPLACE TABLEand thenCOPYas one script, so a COPY that failed (an unreadable
part, a lost client, a killedsnow) left the table with 0 rows until a later load succeeded
(0.31.0 and earlier; measured: 20 rows before, 0 after a COPY of a corrupted part). Now the
parts land in a new table<table>__rivet_load_<16 hex>, the landed row count is checked against
the runs, and oneCREATE OR REPLACE TABLE <table> CLONE <staging> COPY GRANTSputs it in place;
any failure before that statement leaves the table as it was and drops the staging table. A table
that existed before the upgrade is replaced like any other. Grants made on the table now survive a
full load (COPY GRANTS); before, every full load dropped them. What to do: nothing; a load
killed between the COPY and the clone can leave a<table>__rivet_load_*table, which no view or
load reads: drop it. -
A coded stop met by one chunk of a checkpointed export lost its code and its exit class. A
range-chunked export withchunk_checkpoint: true(with or withoutparallel:) recorded a
failed chunk's cause as text inchunk_taskonly, so the run always ended as the incomplete run
RIVET_RUN_UNITS_FAILEDwith exit 1 and a--resumehint, and a--resumemet the same stop and
ended the same way, so a scheduler resumed a refusal forever. Present through 0.31.0. Now the
run prints the code of the most stop-worthy coded chunk failure and exits by its class, as the
same export does withoutchunk_checkpoint; the message lists every failed chunk under the resume
remedy, and a--resumethat meets the stop again refuses by the same code. The exits that
change: a refusal in a chunk (e.g.RIVET_SOURCE_VALUE_UNREPRESENTABLE) 1 → 5; an integrity
stop (e.g.RIVET_DEST_PART_ROWS_MISMATCH) 1 → 3; an internal one 1 → 6; a coded usage or
environment stop 1 or 2 → its own code's exit. A chunk whose stored window does not parse now
stops asRIVET_STATE_CHUNK_WINDOWS_BROKEN(exit 5) on the parallel checkpoint runner too. Chunk
failures that carry no code still exit asRIVET_RUN_UNITS_FAILED. The sequential runner's
message changed fromchunk checkpoint incomplete (N tasks not completed); fix errors and ...to
N chunk task(s) of export '...' not completed; fix the cause, then ... --resume, or inspect rivet state chunks ..., followed by each failed chunk's cause. What to do: a job that matched
RIVET_RUN_UNITS_FAILEDor exit 1 for these stops matches the stop's own code and exit class
instead. -
A stop caused by another live rivet process exited 5, "a human decides", while its remedy was to wait and run again.
RIVET_STATE_RUN_IN_PROGRESS(a run,apply, a CDC drain,rivet state reset*orrivet repairbeside a live run of the export),RIVET_LOAD_TABLE_BUSYandRIVET_STATE_LEASE_LOST(arivet loadorrivet compactbeside another of the same table) exited 5, so a scheduler that does not retry a refusal (the Airflow provider, cron with retries keyed on the exit code) failed the task for good, although the same command succeeds once the other process ends (an overlapping trigger, an orphaned child of a killed parent). Behaviour change: these stops now have the kindbusyand exit 2 (retryable); the codes and the texts are unchanged, the stop still writes nothing and never counts as ownership of the table (RIVET_STATE_LEASE_LOSTis journaledrefused;RIVET_LOAD_TABLE_BUSYstops before a load_run row exists and journals nothing). Arivet runof several exports where one is busy and another fails without a code now exits 2 (the busy stop ranks above an uncoded failure). A stop about what a dead run left behind keeps exit 5. What to do: a scheduler that branches on these codes needs no change; one that branches on exit 5 to catch an overlap matches exit 2 with the code instead. -
The
dest retriesline of a--parallel-export-processesrun counted a CDC child's retries
more than once. Each child sends its process's running total of destination retries with every
run it finishes, and a CDC child withinitial: snapshotfinishes one run per snapshot leg and then
its own; the parent added every one of those totals, so a run whose leg retried R times and whose
drain retried D times printed 2R + D, and more with more legs (0.31.0 and earlier with two or more
legs; with one leg since the snapshot-leg summary change in this release). Now the parent adds
only what each child's total grew by since that child's previous report. What to do: nothing. -
rivet runon a PostgreSQL source crashed when the process had no file descriptor left for the
driver's runtime. Under a lowRLIMIT_NOFILEthe PostgreSQL client's own runtime build hit
Too many open filesand the process ended with a panic (exit 101, noError:line), while
MySQL, SQL Server, Oracle and MongoDB failed loudly under the same cap (0.31.0 and earlier).
Now every PostgreSQL dial (the source, CDC, the Postgres state,init,doctor,check, the
MCP server) first proves the process can start a runtime and is refused
RIVET_SOURCE_RUNTIME_UNAVAILABLE(an environment stop) when it cannot. What to do: raise the
open-files limit of the process that runs rivet. -
An Azure container removed while
rivet runwas writing into it was retried for minutes. Each
chunk's upload was retried as a transient error (Azurite answers its 404 with akeep-alive: timeout=5header, and the retry classifier read the word "timeout" in the store's answer as a
timeout), so a run of ten chunks took over nine minutes to fail; a bucket removed on S3 or GCS
failed in seconds (0.31.0 and earlier). Now a store's own final answer (not found, denied,
invalid, unsupported) is permanent by its type, whatever words its headers carry, and a container
that answered once but is gone by the time a write, listing or read fails is asked about again and
refusedRIVET_DEST_CONTAINER_NOT_FOUND. What to do: nothing. -
The run summary of a CDC export's first run counted the stream only, not its snapshot. A
mode: cdcexport withcdc.initial: snapshotwrites each table's baseline through a snapshot leg
(<export>__snapshot_<table>) and then drains the stream; itsper_export[]entry in
--summary-output,--jsonandrun_aggregatecarried the drain's counts alone, so a run that wrote
the whole table readrows: 0, files: 0(and so did a scheduler task built on it), and a failed leg
showed up as an entry named after the leg instead of the CDC export (0.31.0 and earlier). Now the
CDC export's entry adds what its snapshot legs wrote in the same run torows,files,bytesand
bytes_read, whether the exports run in one process or as--parallel-export-processeschildren,
and a failed leg fails the entry under the CDC export's name. The legs still have
their ownexport_metricsrows, sorivet metricscounts each write once, andduration_msstays the
drain's. Under--parallel-export-processesthe parent also took each export's entry as the newest
finished metric row of the export's NAME, so on a shared state database a failed run was reported as
another config's success of the same export name, with that run's id (0.31.0 and earlier); now
each child reports its own run id and the parent reads that run's row, and a child with none is
reported failed. What to do: nothing; summaries written by earlier runs are not rewritten. -
A coded stop met by one parallel worker lost its code and its exit class. A keyset export
withparallel:(chunk_by_key), a range-chunked export withparallel:(chunk_column,
withoutchunk_checkpoint) and a MongoDB export withparallel:kept each worker's failure as
text only, so the run always printedRIVET_RUN_UNITS_FAILEDand exited 1, or 2 when any
worker's text read as transient; a scheduler then retried a refusal it should have stopped on.
Present through 0.31.0. Now the run prints the code of the most stop-worthy coded worker
failure and exits by its class, as the same export does with one worker; the message still
lists every failed unit, the deciding one last. The exits that change: a refusal in a worker
(e.g.RIVET_SOURCE_VALUE_UNREPRESENTABLE) 1 or 2 → 5; an integrity stop (e.g.
RIVET_DEST_PART_ROWS_MISMATCH) → 3; an internal one → 6; a usage one (e.g.
RIVET_SOURCE_OVERRIDE_WIRE_MISMATCH,RIVET_SOURCE_VALUE_TOO_LARGE) 2 → 1 when a sibling's
text read as transient; an environment one keeps its own code and exits 2 only when its own
text is transient. Worker failures that carry no code still exit asRIVET_RUN_UNITS_FAILED.
What to do: a job that matchedRIVET_RUN_UNITS_FAILEDor exit 1/2 for these stops matches
the stop's own code and exit class instead. -
A stored incremental cursor moved past rows the export never delivered was continued from.
Withlast_cursor_valueedited above the last delivered value, the next run exported only the
rows past it and exited 0, while the destination's ownmanifest.jsonended below it (every
version through 0.31.0). Now a run that continues from a stored cursor (incremental,
keyset_incremental) compares it with thecursor_highof the success manifest at its
destination and refuses a cursor past it withRIVET_STATE_CURSOR_PAST_MANIFEST(exit 5); a
cursor equal or below, a destination with no success manifest of this export, or one that keeps
no objects (a warehouse load) proceed as before. What to do:rivet state reset -c <config> --export <name>starts the cursor over with a full pass. -
rivet could only be installed from a tarball, Homebrew or cargo, which a managed Airflow worker
(Cloud Composer, MWAA, Astronomer) does not allow. Now a release can publish the same
binaries asrivet-cliwheels on PyPI (manylinux_2_28x86_64 / aarch64 and macOS arm64 /
x86_64;rivetandrivet-mcpland in the environment'sbin, andrivet_cli.path()returns
the installedrivet). What to do: nothing; once a release is published there,
pip install rivet-cliinstalls the binary. -
The Linux release binaries did not start on Debian 12, RHEL 8/9 or the official Airflow images.
They were built on Ubuntu 24.04 and needed GLIBC_2.38 / 2.39:version 'GLIBC_2.39' not found
(0.31.0 and earlier releases built the same way). Now every release also ships
rivet-<version>-x86_64-unknown-linux-gnu-glibc2.28.tar.gzand…-aarch64-…-glibc2.28.tar.gz,
built in manylinux_2_28, which run on any glibc 2.28 or newer; the release checks that they need
nothing newer and that they start on Rocky Linux 8 and 9, Amazon Linux 2023, Debian 12 and
Airflow 2.10 / 3.3 before it publishes. What to do: on an older distribution or in an Airflow
image, take the-glibc2.28asset. -
A
rivet loadstopped while selecting its runs left no ledger row, and a missing load spec hid a state-DB fault. A stop raised before any warehouse write but outside the load's typed refusal seam (a--pool --splitprefix holding two split generations,RIVET_LOAD_SPLIT_GENERATIONS_MIXED; a shared prefix; an unreadable loaded set) propagated untyped and unrecorded, sorivet state loadsshowed no attempt at all, and the one text underRIVET_LOAD_SPEC_MISSINGcovered both "no column types recorded" and "the state DB could not be read". Since 0.27.0. Now every stop out of run selection is journaledrefused(never counted as rivet's own attempt) and the next cycle refuses again by the same code; a state DB that cannot answer for the load spec stops asRIVET_STATE_BACKEND_FAILED, and a missing spec staysRIVET_LOAD_SPEC_MISSING. What to do: nothing. -
The Airflow provider package moved to its own repository.
integrations/airflow(the
airflow-provider-rivetpackage: operators forrun,loadandcompact, retry by error class,
CDC) left the rivet repository for
panchenkoai/airflow-provider-rivet, with its
history, so the Rust repository does not publish to PyPI. Now the scheduler contract fixtures
(tests/fixtures/scheduler/) stay here and the provider's CI reads them from a pinned rivet
checkout. What to do: install the provider from its own repository; nothing changes in rivet. -
An empty table under
chunk_by_daysstopped the run instead of exporting zero rows. The date
planner read the NULLminof a table with no row as an unreadable bound and stopped with
min(<column>) returned NULL — table may be empty(since date-native chunking in 0.2.0-beta.3, through
0.31.0; exit 5RIVET_SOURCE_PROBE_UNREADABLEonce the stop carried a code), while the same table
range-chunked on an integer key exported zero rows with exit 0. Now both planners read the key's
span through one function: a table with no row plans one empty window, so the run exits 0 and writes
_SUCCESSand a manifest of 0 rows, as the range run does; a bound absent over a table that has rows
is still refusedRIVET_SOURCE_PROBE_UNREADABLE, and a NULL key is still refused
RIVET_PLAN_CHUNK_KEY_NULL. What to do: nothing. -
A keyset run stopped before its first part blocked the remedy its refusal named. A resumed
MongoDB keyset run refused for an_ida$gtseek cannot page over (mixed BSON types, NaN) had
already claimed the export's resume anchor, so the full scan the refusal recommends was refused
RIVET_STATE_INTERRUPTED_RUN_OWNER_MISMATCH(exit 5) on every run, through 0.31.0. The same held
for any keyset export whose run failed before it recorded a part. Now a run that ends with a
resume anchor under which no part was recorded releases it, so the next run of any mode starts
fresh; an anchor with recorded parts is kept for the resume as before. The refusal
(RIVET_SOURCE_KEYSET_KEY_UNORDERED) now names every key to remove:page_size,resumeand
parallel(resume: truewithoutpage_sizeis itself refused). What to do: nothing for a
new refusal. An export already stuck on an anchor a 0.31.0 run left: run therivet state reset
command theRIVET_STATE_INTERRUPTED_RUN_OWNER_MISMATCHrefusal prints (that run recorded no part). -
Every remaining error rivet raises outside its source readers carried no
RIVET_*code. The
CLI, destination, plan, init, state, preflight, format, manifest, MCP, type-override and TLS
stops (152 sites) printed prose only, so a scheduler could match on wording alone, since the
codes were introduced. Now every stop carries a registered code ([CODE]prefix on the text
line,codein--json-errors), with 28 new codes indocs/reference/errors.md; message text
is unchanged. A stop's exit follows its code's class, which moves these stops from exit 1 to the
class exit:rivet init -oover an existing config (RIVET_INIT_CONFIG_EXISTS, 5); an export
whose destination prefix holds a manifest of the other mode (RIVET_DEST_PREFIX_MODE_MISMATCH,
5); a Full load over a--pool --splitprefix spanning two split generations
(RIVET_LOAD_SPLIT_GENERATIONS_MIXED, 5);rivet loadfor an export with no recorded column
types (RIVET_LOAD_SPEC_MISSING, 5); a state migration that did not reach the binary's schema
version, an empty state path, a run aggregate that does not serialise
(RIVET_INTERNAL_STATE_INVARIANT, 6); a destination backend method the backend does not
implement, a destination key with no file name, a CSV cell type the writer never rejected
(RIVET_INTERNAL_DEST_INVARIANT, 6); arow_hashcolumn gone from the batch or read as the
wrong array kind, a worker panic, aCOUNT(*)that is not a number, a committed cursor with no
column to record it under (RIVET_INTERNAL_RUN_INVARIANT, 6); an Oracle current schema the
server did not return, a MySQL MCP query with no result set (RIVET_INTERNAL_SOURCE_INVARIANT,
6). What to do: a script that keyed on exit 1 for one of these stops treats 5 as "a human
decides, do not retry unchanged" and 6 as "a bug, report it"; the text is unchanged, so a script
matching text keeps working. Other stops keep their exit: usage codes exit 1 and environment
codes exit 2 when transient, else 1, as before. -
Most errors raised while reading a source carried no
RIVET_*code. A malformed source
URL, a CDCtable:that names no single base table, a value the column type cannot hold, a
SQL Server statement the server refused, a MySQL server with no binlog, a spill log that
would not flush, a pgoutput message in a shape the reader never produces, and the other stops
raised undersrc/source/(PostgreSQL, MySQL, SQL Server, Oracle, MongoDB and the shared CDC,
spill, checksum and validate modules) printed only their text, and most exited 1 whatever
their kind (0.31.0 and earlier). Now every error raised there carries a registered code,
with the message text unchanged:RIVET_CONFIG_SOURCE_URL_INVALID,
RIVET_CONFIG_SOURCE_OPTION_UNSUPPORTED,RIVET_CONFIG_SOURCE_FEATURE_MISSING,
RIVET_SOURCE_CDC_TABLE_UNRESOLVED,RIVET_SOURCE_CDC_TABLE_UNSUPPORTED,
RIVET_SOURCE_CDC_TRANSACTION_TOO_LARGE,RIVET_SOURCE_QUERY_COLUMN_UNREFERENCEABLE(usage,
exit 1);RIVET_SOURCE_KEYSET_KEY_UNORDERED(refusal, exit 5);RIVET_SOURCE_RUNTIME_UNAVAILABLE,
RIVET_SOURCE_CONNECT_FAILED,RIVET_SOURCE_QUERY_FAILED,RIVET_SOURCE_CDC_SPILL_UNLOCKABLE,
RIVET_SOURCE_CDC_SPILL_UNWRITABLE(environment, exit 2 when transient, else 1);
RIVET_INTERNAL_SOURCE_INVARIANT(internal, exit 6). Existing codes now also cover source
sites (RIVET_TYPE_UNSAFE_MAPPING,RIVET_SOURCE_VALUE_UNREPRESENTABLE,
RIVET_SOURCE_OVERRIDE_WIRE_MISMATCH,RIVET_SOURCE_VALUE_TOO_LARGE,
RIVET_SOURCE_COLUMNS_CHANGED_UNDER_READ,RIVET_SOURCE_CDC_PREREQUISITE,
RIVET_SOURCE_CDC_CELL_UNSUPPORTED,RIVET_SOURCE_CDC_UNDECODABLE,RIVET_SOURCE_CDC_TRUNCATED,
RIVET_SOURCE_CDC_CHECKPOINT_INVALID,RIVET_CONFIG_CDC_CAPTURE_INSTANCE_MISMATCH,
RIVET_CONFIG_MODE_SETTING_INVALID,RIVET_CONFIG_EXPORT_SOURCE_INVALID,
RIVET_CONFIG_VALUE_OUT_OF_RANGE,RIVET_LOAD_RESERVED_COLUMN,RIVET_STATE_CURSOR_INCOMPARABLE,
RIVET_DEST_CONTROL_ARTIFACT_INVALID,RIVET_VALIDATE_UNVERIFIED,RIVET_INTERNAL_SPILL,
RIVET_INTERNAL_TYPE_BUILDER,RIVET_INTERNAL_VALUE_CONVERTER). An Oracle driver error is
coded by phase (connect against statement) with the server's words kept; a MySQL anchor refused
because the identity query failed keeps that query's own code; a pgoutput transaction poisoned
by a refusal re-raises the original cause with its code. Seedocs/reference/errors.md.
What to do: nothing; a script matching error text keeps working. A scheduler that keys on
the exit code sees 5 where these stops used to exit 1: a batch value the column's type cannot
hold (a PostgreSQL multi-dimensional array or an array of temporal/uuid/bytea elements, a
NUMERIC NaN or infinity under a decimal target, a DECIMAL that does not fit the declared
scale on PostgreSQL or MySQL, a SQL Server MONEY that is non-finite or past 2^53, a SQL Server
decimal that overflows on rescale, an Oracle date, time or NUMBER outside its column type); a
CDC cell the decoder has no faithful reading for (a deferred PostgreSQL infinity or
unchanged-TOAST cell, a pgoutput column type with no binary decoder or no known OID, an
Oracle value the miner cannot decode); a MySQL XA PREPARE or statement-logged DML on a
captured table, an Oracle row written without ALL COLUMNS supplemental logging, and a MongoDB
event past 16 MB (RIVET_SOURCE_CDC_UNDECODABLE); a MongoDB change stream invalidated
(RIVET_SOURCE_CDC_TRUNCATED); a malformed SQL Server resume LSN or an unrecognised MongoDB
resume-token checkpoint (RIVET_SOURCE_CDC_CHECKPOINT_INVALID); a MongoDB_idspan a seek
cannot page over, and a stored MongoDB keyset cursor token that no longer decodes
(RIVET_STATE_CURSOR_INCOMPARABLE); a manifest column checksum that is not a number
(RIVET_DEST_CONTROL_ARTIFACT_INVALID, underrivet validate). It sees 6 where a source
reader invariant broke: a truncated or unexpected pgoutput message, a row before its Relation,
a row image whose arity differs from its relation, a text-form cell on a binary slot, a
spilled row that decodes to no change, a malformed slot LSN or position, a SQL Server open-time
LSN orsys.databasesanswer in a shape the server never gives, a MySQL query with no result
set, an Oracle SCN the server did not return or a mined row of an unexpected table, a MongoDB
resume token that is not a document or a non-row operation reaching the row decoder, a column
builder given a type it cannot build (RIVET_INTERNAL_TYPE_BUILDER), a value checksum that
changed between read and Arrow build or a MONEY render that did not parse
(RIVET_INTERNAL_VALUE_CONVERTER), and a spill frame that is truncated, oversized, nested too
deep, not UTF-8, not a date-time, not JSON or followed by trailing bytes
(RIVET_INTERNAL_SPILL). Environment stops keep their exit: 2 when the failure is transient,
else 1, as before. -
Most errors raised while running, planning, applying, reconciling, repairing and validating
an export carried noRIVET_*code. An unknown--export, a stale plan artifact, a NULL
chunk key, an unfinished chunk checkpoint run, a committed part gone from the destination, a
damagedmanifest.json, failed chunks or keyset ranges, and the other stops raised under
src/pipeline/printed only their text, and most exited 1 whatever their kind (0.31.0 and
earlier). Now every error raised there carries a registered code, with the message text
unchanged:RIVET_CLI_EXPORT_UNKNOWN,RIVET_CLI_RUN_ID_EXPORT_MISMATCH,
RIVET_CLI_OPTION_UNSUPPORTED,RIVET_CLI_REPORT_INVALID,RIVET_PLAN_NOTHING_TO_PLAN,
RIVET_PLAN_ARTIFACT_NO_CREDENTIALS,RIVET_PLAN_PARTITION_TOKEN_UNEXPANDED,
RIVET_PLAN_REJECTED,RIVET_PLAN_CHUNK_KEY_SPARSE,RIVET_CONFIG_CHUNK_COLUMN_NOT_INTEGER,
RIVET_CONFIG_SETTLE_COLUMN_NOT_TEMPORAL,RIVET_CONFIG_SPLIT_PREFIX_UNSTABLE,
RIVET_CONFIG_EXPORT_NAME_RESERVED,RIVET_CONFIG_BATCH_MEMORY_EXCEEDED,
RIVET_CONFIG_CDC_CAPTURE_INSTANCE_MISMATCH,RIVET_SOURCE_VALUE_TOO_LARGE,
RIVET_STATE_CHUNK_RUN_MISSING,RIVET_DEST_MANIFEST_MISSING(usage, exit 1);
RIVET_PLAN_ARTIFACT_STALE,RIVET_PLAN_CHUNK_KEY_NULL,RIVET_SOURCE_KEYSET_KEY_COARSE,
RIVET_STATE_CHUNK_RUN_UNFINISHED,RIVET_DEST_COMMITTED_PART_GONE,
RIVET_DEST_CONTROL_ARTIFACT_INVALID(refusals, exit 5);RIVET_DEST_PART_ROWS_MISMATCH
(integrity, exit 3);RIVET_RUN_UNITS_FAILED,RIVET_RUN_RECONCILE_UNVERIFIED,
RIVET_DEST_MANIFEST_NOT_WRITTEN,RIVET_DEST_OBJECT_NOT_FOUND,RIVET_DEST_UPLOAD_CORRUPTED,
RIVET_DEST_PREFIX_UNLISTABLE,RIVET_CLI_OUTPUT_UNWRITABLE,RIVET_CLI_INPUT_UNREADABLE,
RIVET_CLI_EXECUTABLE_UNRESOLVED,RIVET_CLI_TEST_FAULT_INJECTED(environment, exit 2 when
transient, else 1);RIVET_INTERNAL_RUN_INVARIANT(internal, exit 6). Some existing codes now
also cover pipeline sites (RIVET_SOURCE_PROBE_UNREADABLE,RIVET_STATE_CHUNK_WINDOWS_BROKEN,
RIVET_SOURCE_CDC_PREREQUISITE,RIVET_CONFIG_MODE_SETTING_INVALID,
RIVET_CONFIG_CDC_SHAPE_INVALID,RIVET_LOAD_KEY_COLUMN_INVALID,RIVET_CONFIG_NO_EXPORTS,
RIVET_CONFIG_FILE_UNREADABLE,RIVET_VALIDATE_UNVERIFIED,RIVET_STATE_NOT_WRITABLE). An
error rebuilt around another (the chunked planner's probe aroundresolve_url, a worker's
state-database open,state reset/state acceptaround the source) keeps the inner error's
code instead of dropping it. Seedocs/reference/errors.md. What to do: nothing; a script
matching error text keeps working. A scheduler that keys on the exit code sees 5 where these
stops used to exit 1: a plan artifact older than 24 h or with a drifted cursor applied without
--force; NULL values in a range, date or--pool --splitchunk key; a keyset page that ends
on the key it started after; a plain run meeting an unfinished chunk checkpoint run; a
committed part gone from the destination on resume; amanifest.jsonthat does not parse or
exceeds the read cap; a min/max/partition bound the planner cannot read, or a keyset key value
it cannot read from a page's last row (RIVET_SOURCE_PROBE_UNREADABLE); a stored chunk window
bound that is not an integer (RIVET_STATE_CHUNK_WINDOWS_BROKEN). It sees 3 where a part
written undervalidate: trueholds fewer rows than counted, and 6 where a run invariant
broke (aCOUNT(*)with no row, a chunk run recorded with no tasks, a reconcile over a
non-chunked plan, an encode worker that stopped or panicked, a retry loop with no error to
report). -
Range chunking refused an Oracle
NUMBER(19)key. ANUMBER(19)primary key (the type a
64-bit integer key usually has on Oracle) was read as not integer-family, somode: chunkedover it
exited 1 (0.31.0 and earlier). NowNUMBER(p, 0)up to precision 19 is an integer range key for
the planner andrivet check; a value past the 64-bit range is refused when the bounds are read,
with a message that says so.rivet initstill proposes keyset for such a key, which takes any
value. What to do: nothing. -
Most
rivet loadandrivet compacterrors carried noRIVET_*code. A refused warehouse
object, an unsafe identifier, a missing key, an incomplete run under the prefix, a failed
BigQuery job and the other stops raised undersrc/load/printed only their text, and most exited
1 whatever their kind (0.31.0 and earlier). Now every error raised there carries a registered
code, with the message text unchanged:RIVET_LOAD_IDENTIFIER_UNSAFE,
RIVET_LOAD_KEY_COLUMN_INVALID,RIVET_LOAD_PRIMARY_KEY_MISSING,RIVET_LOAD_RESERVED_COLUMN,
RIVET_LOAD_LOOKALIKE_COLUMN,RIVET_LOAD_TARGET_COLLISION,RIVET_LOAD_STAGING_PREFIX_INVALID,
RIVET_LOAD_DESTINATION_TYPE_UNSUPPORTED,RIVET_LOAD_PARTITION_BUDGET_EXCEEDED,
RIVET_LOAD_FILE_LIMIT_EXCEEDED,RIVET_LOAD_COLUMN_TYPE_UNSUPPORTED,RIVET_LOAD_TLS_CA_INVALID
(exit 1);RIVET_LOAD_EMPTY_RUN_UNSUPPORTED,RIVET_LOAD_RUN_NOT_LOADABLE,
RIVET_LOAD_STAGING_OBJECT_UNSAFE,RIVET_LOAD_SOURCES_SHARED,RIVET_LOAD_COLUMN_RESPELLED,
RIVET_LOAD_TARGET_OBJECT_CONFLICT,RIVET_LOAD_COMPACT_BASE_MISSING,
RIVET_LOAD_CHANGELOG_PARTITION_CHANGED,RIVET_LOAD_BUFFER_NOT_COMPACTED,
RIVET_LOAD_REBASELINE_OVER_CHANGES,RIVET_LOAD_TABLE_BUSY(refusals, exit 5);
RIVET_LOAD_MANIFEST_INCONSISTENT,RIVET_LOAD_SOURCE_FILE_MISMATCH,
RIVET_LOAD_COPY_COUNT_MISMATCH(integrity, exit 3);RIVET_LOAD_LEDGER_UNREADABLE,
RIVET_LOAD_RUN_LANDED_DURING_LOAD,RIVET_LOAD_WAREHOUSE_AUTH_FAILED,
RIVET_LOAD_WAREHOUSE_ACCESS_DENIED,RIVET_LOAD_WAREHOUSE_OBJECT_NOT_FOUND,
RIVET_LOAD_WAREHOUSE_REJECTED,RIVET_LOAD_WAREHOUSE_UNAVAILABLE(environment, exit 2 when
transient, else 1);RIVET_LOAD_WAREHOUSE_ANSWER_UNREADABLEandRIVET_INTERNAL_LOAD_INVARIANT
(internal, exit 6). Some existing codes now also cover load sites (RIVET_CONFIG_NO_EXPORTS,
RIVET_CONFIG_MODE_SETTING_INVALID,RIVET_CONFIG_SOURCE_MODE_UNSUPPORTED,
RIVET_CONFIG_LOAD_PARTITION_UNSUPPORTED,RIVET_LOAD_TARGET_NOT_RIVETS). A Snowflake or
ClickHouse stop before the first statement (an empty newest run, more than 1,000 files, an unsafe
cluster_by, a CDC log without a key) is now journaledrefused, notfailed, so it no longer
counts as rivet having loaded the table. Seedocs/reference/errors.md. What to do: nothing;
a script matching error text keeps working. A scheduler that keys on the exit code sees 5, 3 or 6
where these stops used to exit 1. -
A float
chunk_columnunderquery:lost its fractional keys with exit 0. Withtable:the
planner refused a non-integer range key; withquery:it could not read a catalog, only warned,
and integerBETWEENwindows skipped every key between two whole numbers (adouble precisionkey
delivered 40 of 43 rows) (0.31.0 and earlier). Now the run reads the query's result type before
it plans the windows and refuses a float or scaled-decimal key with
RIVET_CONFIG_CHUNK_COLUMN_NOT_INTEGER(exit 1), the same code thetable:refusal now carries.
What to do: usechunk_by_key:(keyset) or an integer column for such a key. -
Most config errors carried no
RIVET_*code. A missing config file, broken YAML, an
incompletesource:block, an unseturl_env, a mode without its setting and the other
refusals raised while a config is read printed only their text, so a scheduler or a script could
match them by wording alone (0.31.0 and earlier). Now every error raised undersrc/config/
carries a registered code, with the message text unchanged:RIVET_CONFIG_FILE_UNREADABLE,
RIVET_CONFIG_YAML_INVALID,RIVET_CONFIG_SOURCE_CONNECTION_INVALID,RIVET_CONFIG_SECRET_UNSET,
RIVET_CONFIG_EXPORT_SOURCE_INVALID,RIVET_CONFIG_MODE_SETTING_INVALID,
RIVET_CONFIG_DESTINATION_AUTH_INVALID,RIVET_CONFIG_DESTINATION_ENDPOINT_UNSAFE,
RIVET_CONFIG_VALUE_OUT_OF_RANGE,RIVET_CONFIG_CDC_SHAPE_INVALID,
RIVET_CONFIG_CSV_COMPRESSION_UNSUPPORTED,RIVET_CONFIG_TLS_VERIFICATION_DOWNGRADE,
RIVET_CONFIG_LOAD_BLOCK_INVALID,RIVET_CONFIG_PATH_UNSAFE,RIVET_CONFIG_ROW_HASH_INVALIDand
RIVET_CONFIG_COLUMN_OVERRIDE_TABLE_UNKNOWN(seedocs/reference/errors.md). The exit code stays- What to do: nothing; a script matching config error text keeps working.
-
rivet runinto a bucket that does not exist failed with the store's raw error. The upload
ended withNotFound (persistent) at write, context: { uri: ... }and noRIVET_*code (exit 1),
whilerivet validateand every read already named the missing bucket (0.31.0 and earlier).
Now a write the store answers "not found" asks whether the bucket exists, and a missing one is
refusedRIVET_DEST_CONTAINER_NOT_FOUNDnaming it, on S3, GCS and Azure. What to do: nothing. -
Two exports of one name on a shared state refused each other even when they read different
sources. Since every run took the export's run lease (keyed by the export name), the
shared-state deployment the state docs describe (several configs, one export name, one PostgreSQL
state, different sources) ran one export at a time and refused the rest with
RIVET_STATE_RUN_IN_PROGRESS. Now the run lease is keyed by the export and its source, as the
cursor is: runs of one name on different sources run side by side, a second run on the same source
is still refused. A run under its lease closes only its own source's deadrunningrows; state
schema v36 records the source on eachrun_statusrow (rows from before v36 have none and are
closed by any run of the export). What to do: nothing. -
rivet initcould not scaffold a Snowflakeload:block. It had flags for BigQuery and
ClickHouse only, so a Snowflake config needed a hand-writtenload:block (0.31.0 and earlier).
Now--snowflake-connection,--snowflake-warehouse,--snowflake-database,
--snowflake-schemaand--snowflake-storage-integration(all five, with--gcs-bucket) write
atarget: snowflakeblock withpk: auto,cluster_by: auto,cleanup_source: trueand a
per-tablepartition:guess atday(a leadingDATE_TRUNCclustering key on Snowflake). A
missing companion flag,--s3-bucket, no bucket, or a second warehouse's flags are refused at the
flag. What to do: nothing. -
The end-of-run summary of a failed export re-read its error text from every
;. The step
that strips the inlinerivet …recovery hint was quadratic in the message: a 512 KiB message
made of;took 21 s (×4 per doubling) before the summary printed (0.31.0 and earlier).
Now the cut is found in one pass each way; the summary of the same message prints in
milliseconds. What to do: nothing. -
rivet compacton a BigQuery base partitioned by hour planned one MERGE job per 166 days of
the key's whole value span. A row holding the end-of-time sentinel9999-12-31(or any date
outside BigQuery's partitioned days, 1960-01-01 to 2159-12-31) stretched the span to the sentinel:
17,544 serial MERGE jobs for two changed rows (0.27.0 through 0.31.0; monthly and yearly keys planned
fewer). Now the days outside the partitioned range, which BigQuery keeps in one unpartitioned
partition, take one MERGE per side, and only the span of values inside the range is stepped.
What to do: nothing. -
rivet init --mode cdcon SQL Server named a capture instance that did not exist. It wrote
capture_instance: <schema>_<table>by convention, so a table whose CDC was enabled under another
instance name got a config that failed on its first run (0.31.0 and earlier). Now init reads
cdc.change_tablesand writes the instance it holds (the newest of two); a table without CDC gets
the conventional name with thesp_cdc_enable_tablecall that creates it. What to do: nothing. -
A SQL Server password holding
@reached the state database. In the keyword/value URL form
(sqlserver://host;user=sa;password=p@ss;…) the@inside the password was read as the end of a
userinfo, so the source key rivet stores beside every cursor kept the password's tail (0.31.0 and
earlier). Now keyword passwords are masked before anything is cut from the URL. What to do:
if such a password was in use, the progress stored under the old key is not found and the next run
of an incremental export reads its table again; rotate that password, since the state database
held part of it. -
A MySQL CDC stream refused a
TIMEpast 24 hours with the batch remedy, "cast it in a query",
which a stream has no query for. The check also ran by the column's native type, so a
columns: { col: string }override could not carry the duration as text (0.31.0 and earlier).
Now the refusal (RIVET_SOURCE_VALUE_UNREPRESENTABLE, exit 5) names the two remedies a stream
has: the column declaredstringincolumns:, which carries the value as text
(800:00:00.000000), or the column left out of the capture followed by the re-baseline recipe.
What to do: nothing. -
mode: cdcon a standalone MongoDB failed with the driver's raw dump. The run exited 1 with
Error code 40573 … RawDocumentBuf { data: "7d00…" }and noRIVET_*code (0.31.0 and earlier).
Now it is refusedRIVET_SOURCE_CDC_PREREQUISITE, naming the standalone and the ways out (a
replica-set member, a one-member replica set, ormode: full). What to do: nothing. -
rivet runrefused a MongoDB collection that does not exist without its code.rivet check
refused itRIVET_SOURCE_COLLECTION_NOT_FOUND;rivet runprinted the same sentence with no
RIVET_*code (exit 1) (0.31.0 and earlier). Now both raise it through one function andrun
carries the code. What to do: nothing. -
A
cursor_columnspelled in another case than its column failed with the raw database error on
PostgreSQL and Oracle. MySQL and SQL Server refused it withRIVET_CONFIG_CURSOR_COLUMN_CASE; on
PostgreSQL and Oracle the quoted name failed the query first, so the run exited 1 with
column "UPDATED_AT" does not exist/ORA-00904and no code (0.31.0 and earlier). Now a failed
incremental query reads the column names of the export's query, and a cursor that matches one only
when case is ignored is refused withRIVET_CONFIG_CURSOR_COLUMN_CASEon every engine, naming the
spelling to use. What to do: nothing. -
Two
rivet runs of onemode: cdcexport at once both delivered the stream. A batch run
holds the export's run lease and a second run beside it is refused; a CDC run took no lease, so
the second read the same changes again (MySQL, SQL Server, Oracle and MongoDB delivered duplicate
change rows; on PostgreSQL the first run died on the busy slot with an uncoded error) (0.31.0 and
earlier). Now a CDC run holds the run lease for its whole stream and a second one is refused
withRIVET_STATE_RUN_IN_PROGRESS(exit 5) before it opens the source. What to do: nothing. -
rivet apply --poolwrote its own lines into adestination: stdoutexport. The predicted
and actual makespan lines (and the wave headers of a multi-wave run) went to stdout, mixed into the
exported CSV or JSON bytes (0.31.0 and earlier). Now they go to stderr, like every other run
line. What to do: a script that read thePool:lines from stdout reads them from stderr. -
A run that died (killed, out of memory, out of disk) stayed
runningin the state forever.
Itsrun_statusandexport_metricsrows were closed only by the run itself, or by hand with
rivet state finish-run;rivet state runsandrivet metricskept showing it in progress
(0.31.0 and earlier; a chunk-checkpoint resume closed the rows of the run it adopted, nothing
else did). Now a run that takes the export's run lease stamps that export's earlierrunning
rowsinterruptedin the same transaction as its own start: while it holds the lease no other
run of the export can be alive. What to do: nothing;state finish-runstill closes a row by
hand. -
An Oracle CDC checkpoint the retained redo no longer reaches was refused without a code.
When the archived logs between the checkpoint and the oldest available log are gone (or a
redo sequence is missing), the run stopped with the data-loss text and the re-baseline recipe
but as a plain error: exit 1, noRIVET_*code, so a scheduler could not tell it from a
transient failure and retried it forever (0.31.0 and earlier). Now it is refused as
RIVET_SOURCE_CDC_LOG_GAP(exit 5), like the same loss on PostgreSQL, MySQL and SQL Server.
What to do: nothing; the recipe in the text re-baselines the stream. -
partition_byrefused a PostgreSQL table with a NUMERIC(p,s) column. Each partition is read
through a window that wraps the query, and the type hints PostgreSQL needs for NUMERIC precision
were looked up in that wrapper, which names no table: the run failed withprecision/scale unavailable from query metadata and catalog lookup, although the same export without
partition_byran (0.31.0 and earlier; a--splitunit wraps the query the same way). Now
the plan keeps the query before the window (catalog_query) and every reader resolves type hints
from it. What to do: nothing. -
A stored incremental cursor of another type stopped every run with a raw driver error.
When the stored cursor is not a value the cursor column can be compared with (the state was
edited, or the column changed type), PostgreSQL, SQL Server and Oracle failed every run with
their own conversion error (invalid input syntax for type bigint,Error converting data type varchar to bigint), exit 1, no code and no remedy (0.31.0 and earlier; MySQL coerces the text
and delivers). Now the run is refused withRIVET_STATE_CURSOR_INCOMPARABLE(exit 5), which
names the cursor, the column andrivet state reset. What to do: nothing; an export stuck on
this error is led out byrivet state reset -c <config> --export <name>. -
A MySQL table without a primary key could be paged on a different unique key from one
run to the next. With nochunk_by_key:the planner pages such a table by keyset on "the
first" single-column unique NOT NULL key, and the list came back in whatever order the
server's catalog returned it; a table with two such keys (id,ext_id) was paged onid
by one run and onext_idby the next, so a run that resumed or continued stored progress
was refused (RIVET_STATE_CURSOR_OWNER_MISMATCH, exit 5) though nothing had changed. Seen
on main CI on 2026-10-14 in a crash-then-resume cell. Now the keys are ordered by rivet:
the primary key first, then by column name, on every engine. What to do: nothing; an
export refused that way runs afterrivet state reset, as the text says. Pin the key with
chunk_by_key:where the choice matters. -
A load could be refused as "another
rivet load... is writing" right after its own
process released that table. SQLite state only. The load lease is anflock, and it was
released by closing the file; a process spawned on another thread at that moment kept a copy
of the descriptor until itsexec, and with it the lock. The next load of the same table in
this process was refused (exit 5) though nothing was writing it. Now the lease is unlocked
explicitly before the file is closed. What to do: nothing; a refused load can be run again. -
A resumed range-chunk run trusted the checkpoint and the parts it continued. A
checkpoint or a committed part changed between the crash and the resume was continued
as it stood, and the run exited 0 over wrong data. Measured on 0.31.0 with 40 source rows,
each case through bothrivet runandrivet run --resume:- a pending chunk task marked
completed: 35 rows delivered; - a task whose upper bound was lowered, leaving a hole between two windows: 38 rows;
- two stored windows that share keys: 43 rows, 3 of them twice;
- a committed part replaced by its sibling's bytes: 35 distinct ids, one chunk's rows
twice and another's missing; a truncated part was declared as it was; - a run planned under another
chunk_sizewas refused with no code (exit 1), after
rewritingmanifest.json. - Now a committed part that is gone, or is not the size the state DB logged, has its
chunk read again, and so does a chunk markedcompletedwith no part. Keys between the
stored windows are planned again. Windows that share keys are refused
(RIVET_STATE_CHUNK_WINDOWS_BROKEN, exit 5), and so is a changed chunk plan
(RIVET_STATE_CHUNK_PLAN_CHANGED, exit 5); both texts namerivet state reset-chunks,
and the plan refusal also names restoring the settings. Underverify: contenteach
committed part is read back too: a row count other than the logged one or a key outside
its chunk's window has the chunk read again, which catches a part replaced at its own
size. - What to do: nothing for runs that were not edited by hand. An export resumed by 0.31.0
or earlier after its state DB or destination was edited may hold the wrong rows; run it
again whole.
- a pending chunk task marked
-
rivet loadinto Snowflake stored atimestampcolumn as a wrong date, and several
loads failed outright. Snowflake target only; BigQuery and ClickHouse loads are not
touched.- Wrong values with exit 0 (measured on 0.24.2; the statements are the same through
0.31.0). A column the source types as a timestamp without a time zone (PostgreSQL
timestamp, MySQLdatetime) was read from the Parquet part as its raw microsecond
integer and stored as that many seconds:2026-10-13 21:25:02.145628landed as year
56785866. The load printedLOAD OKand the row count matched. Atimestamptzcolumn
was stored correctly. - Loads that failed in 0.31.0 and earlier: a table with a
TIMEcolumn (Failed to cast variant value ... to TIME); a table with a binary column,byteaorVARBINARY
(expecting BINARY ... but got VARIANT); any table whose destinationprefixis written
without a trailing/(Remote file ... was not found: the stage URL and the file name
were joined without a slash). - Loads that failed from 0.26.0 through 0.31.0: every load onto a free table name
(object-kind probe ran but KIND_ was not in snow's output), and the cycle of an
incremental or CDC table whose export gained a column (a syntax error atADD COLUMN).
A first CDC cycle whose table name is 21 to 30 characters long could also stop on a
missing change log, because the "does not exist" answer was searched in a linesnow
wraps at 80 columns. - Now each part is read through a Parquet format rivet spells inline, with logical
types on: a timestamp, atimestamptzand aTIMEland with their microseconds, a
binary column lands as its bytes, and a prefix with or without the trailing/loads.
Whether a table or a change log exists is asked of the schema catalog, never read from
an error text. - What to do: a Snowflake table loaded by 0.31.0 or earlier holds wrong values in
every timestamp-without-zone column. Amode: fullexport replaces its table on the next
rivet runandrivet load. An incremental or CDC table keeps the rows already in its
<table>__changeslog; decide per table whether to rebuild it from a whole pass. - Also changed: rivet no longer creates or reads the named file format
rivet_pq. One
left in the schema by an earlier release is unused and can be dropped; the loading role
no longer needsCREATE FILE FORMAT. rivet compacton a Snowflake target is unchanged: it exits 0 withskipped — this warehouse keeps a change log behind a view and never compacts.
- Wrong values with exit 0 (measured on 0.24.2; the statements are the same through
-
Breaking: a second run of an export while one is alive is refused, whatever the mode.
Every source; a SQLite or PostgreSQL state database. Reproduced on rivet 0.31.0 with
PostgreSQL.- What was wrong in 0.31.0 and earlier. Only a run with
chunk_checkpoint(or a MongoDB
resume) held its export's run lease. Two runs of one export started together in any
other mode both exited 0 and both wrote: the destination held every row twice, and both
manifests saidsuccess. Measured withmode: fullover 200 rows: 400 rows delivered.
rivet repair --executebeside a live checkpointed run exported that run's pending chunks
a second time.rivet state resetwas accepted while an incremental run was reading, and
that run's manifest then described its delta as a first pass. - Now every run takes the lease. One of the two runs exits 5 with
RIVET_STATE_RUN_IN_PROGRESSand writes nothing; the other delivers (200 rows in the
measurement above). It is the later one, except that a first run still creating its
state database may reach the lease no earlier than a run started beside it.rivet repair --executeandrivet state resetget the same refusal beside a live run. - Also changed: the refusal a second run got beside a live checkpointed run was exit 1
with no code; it is exit 5 withRIVET_STATE_RUN_IN_PROGRESS. The text still starts
chunk checkpoint run ... still in progress in another live rivet process. - What to do if a run now refuses: wait for the live run of that export, or stop it.
A scheduler that may start an export while its previous run is still going should treat
exit 5 with this code as "skipped, a run is in progress". Two exports with different names
never block each other; a--splitunit and apartition_bychild have their own names. - On a PostgreSQL state database every run now opens the lease keeper's connection (one
more connection per rivet process), where only checkpointed runs did. - The lease is released by the operating system (SQLite state) or expires by its keeper
(PostgreSQL state) when the process dies,kill -9included; nothing has to be cleaned up. - New file beside a SQLite state database: every run leaves an empty
.rivet_state.db.lease-chunk-run-<export>next to.rivet_state.db, one per export. Only
checkpointed runs and loads left such files before. They hold no data and may be deleted
while no rivet process runs; the next run creates its own again. - A state database or state directory that is read-only still refuses the run with
RIVET_STATE_NOT_WRITABLE(exit 1), now also when it is the lease file that cannot be
created.
- What was wrong in 0.31.0 and earlier. Only a run with
-
Fixed: a
mode: chunkedconfig the planner refuses on a large table is refused on a small
one too. PostgreSQL, MySQL, SQL Server and Oracle; exports with thetable:shortcut.
Reproduced on rivet 0.31.0 with PostgreSQL.- What was wrong in 0.31.0 and earlier. While the source's row estimate was at most
chunk_size, the planner returned a single pass before it checked the shape. So a config
with aDECIMALchunk_column, or with nochunk_columnover a table that has no key to
page by, ran with exit 0 on a small table and was refused once the table outgrew one chunk.
Measured on a 3-row table withchunk_column: amount(DECIMAL(10,2)),chunk_size: 4:
0.31.0 exits 0 and writes one file. - Now the shape is checked first: the same run exits 1 with
chunk_column 'amount' ... is not an integer-family column, twice in a row, and writes nothing. A keyless table gets
chunked mode found no safe shape. On MySQL an auto-selected unique key under auuid
override is refused on a small table as on a large one
(RIVET_CONFIG_KEYSET_KEY_UUID_OVERRIDE). - What to do if a run now refuses: follow the message: name an integer
chunk_column,
setchunk_by_key:, or usemode: full. - Unchanged: a config whose shape is valid still runs as one pass on a small table.
- What was wrong in 0.31.0 and earlier. While the source's row estimate was at most
-
Fixed: a
mode: chunkedexport whose table fits in one chunk is refused as what it is
beside an unfinished checkpoint run. Applies to PostgreSQL, MySQL, SQL Server and Oracle.
Amode: chunkedexport that sets none ofchunk_checkpoint,chunk_count,chunk_by_days
andchunk_by_keyruns as one pass while the source's row estimate is at mostchunk_size.- Before: beside an unfinished checkpoint run of that export, such a run was told
this export now runs asfull`` andrestore thechunked` settings`, which the config still had; a table
near `chunk_size` got that message or the checkpoint one as the estimate moved. - Now it is told what any run of its mode without the checkpoint is told:
this export now runs without the checkpoint that run was opened with, with the remediesrestore the checkpoint setting (chunk_checkpoint: true)andrivet state reset-chunks -c <config> --export <name>. Both were run on PostgreSQL, MySQL and SQL Server. - Unchanged: the code (
RIVET_STATE_INTERRUPTED_RUN_OWNER_MISMATCH), exit 5, nothing read or
written; what such an export runs as (one pass,modefullinrivet plan, the run
summary andexport_metrics); the message amode: fullexport gets. - Also changed: beside an unfinished run of the other paging mode (a
chunk_by_keyrun, then
chunk_columnwith no checkpoint on a small table) the message saysnow runs aschunked``, where it saidfull. rivet plan --format jsonwrites"single_pass_of": "Chunked"(or"Keyset") in
resolved_planfor such an export; the field is not part of the plan's checksum.
- Before: beside an unfinished checkpoint run of that export, such a run was told
-
Fixed:
rivet validateno longer passes a prefix whose manifest is gone while_SUCCESS
is there. Every source and destination. Reproduced on rivet 0.31.0 with PostgreSQL and a
local destination.- What was wrong in 0.31.0 and earlier. With
manifest.jsonremoved from a complete
export,validateprintedstatus: legacy_run (no manifest at this prefix — a pre-0.7.0 export ...)and exited 0, sorivet validate && deploywent on over parts nothing had
verified._SUCCESSis written only after the manifest, so a marker without a manifest
is a prefix that lost its manifest, not an old export. - Now the same command exits 3 with
RIVET_VERIFY_MANIFEST_MISSING(a new code), on
every pass, and names the two ways out: restoremanifest.json, or run the export again.
A run of the export writes the prefix anew andvalidatepasses after it. - If
_SUCCESSitself cannot be read (an I/O error, not absence),validatereports
RIVET_VERIFY_SUCCESS_READ_ERRORand exits 1, where it exited 0. - Unchanged: a prefix with neither a manifest nor
_SUCCESSis stilllegacy_run, exit 0.
- What was wrong in 0.31.0 and earlier. With
-
Fix: a
FROMinside an SQL comment is no longer read as the table aquery:reads.
PostgreSQL and SQL Server exports with a hand-writtenquery:; the preflight probe of
rivet checkon every SQL engine. Reproduced on rivet 0.31.0 with PostgreSQL and SQL Server.- What was wrong in 0.31.0 and earlier. To give a
NUMERIC/DECIMALcolumn its
declared precision and scale, rivet reads the table name out of the query text and asks
the catalog. The reader did not skip comments, so a-- ...line or a/* ... */block
holding the wordfrombefore the query's ownFROMnamed the table. Measured on a
table with anumeric(10,2)column:-- rows from the last week(no table of that name): the export failed, exit 1,
1 column(s) have no safe Rivet mapping, no file written. The same query without
the comment exporteddecimal(10,2).-- taken from legacy, wherelegacyhas that column asnumeric(20,6): exit 0, and
the column was written asdecimal(20,6), the other table's type. The values were
right.- the same with
legacyholdingnumeric(6,0): the export failed, exit 1,
cannot parse DECIMAL "12345.67" as decimal(scale=0). - A comment AFTER the table name (
FROM t -- from b) made the reader give up, with the
first outcome. - SQL Server, the same table as
decimal(10,2): every one of these exported, exit 0,
with another type for the column. With no table of the commented name, or with the
comment after the table name, the column was written asdecimal(38,2); with
legacyholdingdecimal(20,6)it was written asdecimal(20,6).
No wrong value was delivered in any of these; the column's declared type was.
- Now one reader finds the outermost
FROMfor every caller, and a comment counts as
whitespace wherever it stands. Every query above exportsdecimal(10,2)on both engines. - Also read now, where the reader gave up before: a query ending in
;, a table name
with a non-ASCII letter (FROM café), and on SQL Server a double-quoted name
(FROM "s"."t"). The preflight probe no longer takesfrom xinside a string literal
for the table, and readss . tass.t. - Unchanged: a join, a comma list, a subquery or a function call in
FROMstill
gives no table, and the column needs acolumns:override as before. A MySQL#
comment is not skipped:#is an operator in PostgreSQL, and the reader is told no
engine. - Stored progress (unreleased builds only): the stream of a query's stored progress is
read by the same reader, so a query with such a comment now names its real table there.
No released version stores a stream.
- What was wrong in 0.31.0 and earlier. To give a
-
Fix: an Oracle export whose table loses a column while it is being read stops with an
error instead of a panic. Oracle only; every batch mode. Seen once in CI on main
(07e3e9b7..97d3f57b) as exit 101 of a range-chunked run; not reported from the field.- What was wrong in 0.30.0 and 0.31.0. Each Oracle read describes its query's columns,
then runs the query. When the query needs no column rewritten (no LOB, no type read
through a conversion) it is sent as written, so a column dropped between the two
statements came back as a narrower result, and the reader indexed that result by the
wider description:index out of bounds, exit 101 (an abort in a release build). The run
ended without recording its failure: itsrun_statusandexport_metricsrows stayed
running. - Now that read stops with
RIVET_SOURCE_COLUMNS_CHANGED_UNDER_READ(kind:
environment), the run is recorded as failed, and the next run reads the table as it is. - Proof and its limit. A unit test pins the column layout on a result narrower than
its probe; it panicked before the change. The window between the two statements is a
few milliseconds, so no live cell hits it on demand: the existing cell
a_column_dropped_under_a_live_run_delivers_or_fails_loudly_oraclereaches it by chance. - Unchanged: a query whose columns are rewritten names each column, and Oracle itself
refuses a dropped one (ORA-00904). A column ADDED between the two statements is still
delivered as an extra column of that read.
- What was wrong in 0.30.0 and 0.31.0. Each Oracle read describes its query's columns,
-
Fix: masking a URL password in a log line or an error text takes one pass over the
text. Applies to every command and every engine: each emitted log line and each error
text goes through this scan.- What was wrong in 0.31.0 and earlier. The scan for
scheme://user:password@started
a new read at every byte: from each letter to the end of the run of letters, digits and
+ . -it stands in, and from each://to the next whitespace. The time grew with the
square of the longest such run. Measured on that function compiled alone with
optimisations: a run of 16 KiB took 0.1 s, 64 KiB 1.1 s, 256 KiB 18.6 s. The text
came out correct; only the time was wrong. Whether a real driver message holds a run
that long (a hex or base64 value quoted in an error) was not observed in the field: the
defect was found by reading the code under mutants that hung its tests. - Now the same 256 KiB take 0.7 ms and 16 MiB take 34 ms. What is masked is
unchanged: the new scan and the previous one give byte-identical output on 400,000
generated texts of URL punctuation, whitespace and multi-byte characters, and that
comparison stays in the test suite. - Also fixed: a secret query parameter with a name longer than 64 characters was not
masked (?<65+ characters>_token=..., since 0.25.0). The length limit is removed;
a name of any length ending in a secret word is masked. - Removed, no behaviour change: two config checks that no config could reach, because
an earlier check refuses the same input first — the..check of a local destination
path/prefix(every destination's..segment is refused before it), and "cdc mode
reads the transaction log, not a query" (aquery:besidetable:is refused as two
sources, and aquery:alone as a missingtable:). The messages an operator sees are
the ones they saw before.
- What was wrong in 0.31.0 and earlier. The scan for
-
Fix: a PostgreSQL run through a transaction-mode pooler no longer leaves session settings
on the pooled server connection. PostgreSQL batch connections andmode: cdc. Reproduced
on rivet 0.31.0 behind pgBouncer 1.26.0 in transaction mode, with pool sizes 1 and 5.- What was wrong in 0.30.0 and 0.31.0. Each connection rivet opened for
run,check,
plan,doctor,init, a PostgreSQL state database andrivet-mcpsent
SET TimeZone = 'UTC'; SET DateStyle = 'ISO, MDY'; SET IntervalStyle = 'postgres'; SET bytea_output = 'hex'as session settings, unless it detected a transaction pooler.
The detection compares the backend pid of two statements, and an idle pooler answers both
from one server connection, so the settings were sent and stayed on the connection the
pooler then gave to its other clients.- Who was affected: other clients of the same pooler, not rivet. pgBouncer puts back the
parameters it tracks for each client (TimeZone,DateStyleandIntervalStylein
1.26), so the setting another client read differently isbytea_output. Measured on a
database whose default isescape: after onerivet run, another client's
bytea::textcame back inhex. A pooler that tracks fewer parameters hands on more
of the four. mode: cdchad no pooler detection at all: every connection itstest_decodingreader
opened sent the same four andSET extra_float_digits = 3, a refused run included (the
reader has set session formats since 0.19.1, bygit log; measured on 0.31.0). Through
pgBouncer on a database whose defaults areescapeandextra_float_digits = 0, one CDC
run leftbytea_output = hexandextra_float_digits = 3on the pooled connection for
the pooler's other clients.- rivet's own output was correct.
- Who was affected: other clients of the same pooler, not rivet. pgBouncer puts back the
- Now a batch connection sends no session-level
SET. The export, the bound and count
probes of a run, the range and plan probes ofcheckandplan, and the table
statistics ofrivet-mcpset the four formats withSET LOCALinside the transaction
that reads, on a direct connection and behind a pooler alike, and nothing outlives the
transaction. On a direct connection the delivered files and thecheckoutput are the
same as in 0.31.0 (compared under a role whose defaults are Asia/Tokyo,German, DMY,
sql_standardandescape).- Behind a pooler busy enough for 0.31.0 to detect it, the probes and
checkran in the
role's zone and formats while the export ran in UTC. They now read in UTC as well. mode: cdcsends no session-levelSETeither. Each read of the slot runs in a
transaction of its own that sets the four formats andextra_float_digits = 3with
SET LOCAL(two round trips more per read of the slot, which holds up to one part's
changes). On a direct connection the delivered changes are the same as in 0.31.0
(one window of inserts, updates and deletes overfloat8,float4,timestamptz,
timestamp,date,bytea,intervalandnumeric, read under a role whose defaults
are Asia/Tokyo,German, DMY,sql_standard,escapeandextra_float_digits = 0).- Checked by hand through pgBouncer 1.26.0 in transaction mode with one server
connection, in front of awal_level = logicalserver: a refused run, a run that
read changes and a run that found none each left the five settings and their
pg_settings.sourceas they were, on the same backend. The test suite covers a
refused capture through the pooler and, offline, the statements the reader sends;
it has no pooler in front of a logical server.
- Checked by hand through pgBouncer 1.26.0 in transaction mode with one server
- A setting an earlier release left behind stays on the pooled server connection until
the pooler closes it (pgBouncer:server_lifetime, one hour by default) or is restarted.
- Behind a pooler busy enough for 0.31.0 to detect it, the probes and
- What was wrong in 0.30.0 and 0.31.0. Each connection rivet opened for
-
Fix: a
query:holding a non-ASCII character no longer ends the run in a panic.
Applies to every source engine. Reproduced on rivet 0.31.0 with PostgreSQL, MySQL,
SQL Server and MongoDB.- What was wrong in 0.31.0 and earlier. Two readers of the query text cut it at a
byte offset, and a cut inside a multi-byte character stopped the process: exit 101,
byte index N is not a char boundary, on every run.- Every export, in every mode: a multi-byte character across byte 6 of the query (a
comment or a CTE name that opens it:/* <non-ASCII comment> */ SELECT ...,
WITH é AS (...) SELECT ...), or across byte 4 of what followsSELECT *
(SELECT *, 'été' AS label FROM t). On MongoDB the same text crashed in place of the
refusal a hand-writtenquery:gets there. - Range-chunked exports (
chunk_column), and the value-partition and split planners
that probe a query's bounds the same way: any non-ASCII character before the
query's firstFROM(SELECT id, 'café' AS label FROM t). - A non-ASCII character anywhere else (a
WHEREliteral, a table name afterFROM)
was read correctly. - Nothing was written to the destination: no part, no manifest, no
_SUCCESS, no run
summary. The state database kept onerun_statusrow inrunningfor each crashed
run and the export's schema baseline. - The cut at the query's start dates from 0.6.0 and every export's query has gone
through it since 0.18.0; the scan forFROMdates from 0.16.0.
- Every export, in every mode: a multi-byte character across byte 6 of the query (a
- Now both readers compare bytes and never cut the text inside a character. Such a
query runs and delivers what its ASCII spelling delivers; on MongoDB it is refused by
name like anyquery:. An export that crashed needs nothing but the upgrade: run after
two crashes of 0.31.0 on the same state, this build exited 0 and delivered every row.
Therunningrows the crashes left stay inrun_status, behind the newer run. - Unchanged: what is recognised in an ASCII query. The
SELECT * FROM tand plain
column-list shortcuts, the position of the outerFROMand every SQL text rivet builds
from a query are byte-identical for 575,270 query texts that did not crash before
(568,212 of them ASCII), compared between the previous code and this one. A query with a
non-ASCII identifier (SELECT id, café FROM t) is not taken for the bare-table
shortcut: its bounds are read from the query, wrapped asFROM (<query>) AS _rivet.
- What was wrong in 0.31.0 and earlier. Two readers of the query text cut it at a
-
rivet load --pool Nkeeps at most 16 storage requests in flight for the whole
process. Every table loaded at once used to send its storage requests (list, read,
delete) with no shared limit, each request in flight on a connection of its own: a load
of 16 tables with--pool 16held 46 connections to Cloud Storage and 145 open files,
measured on macOS with a release build. It now holds 16 and 117, in the same time
(15.0 s before, 14.5 s after, one run each). The budget counts requests, not
connections: a connection can outlive its request for a moment, and a debug build
sampled every 50 ms held up to 22. A pool larger than the budget waits for a free
request instead of opening more connections. The export side (rivet runwriting to a
bucket) is unchanged. No config key: the budget is a constant. -
Fix (behaviour change): on a PostgreSQL state,
rivet loadandrivet compactstop
instead of writing a table under a lease nothing renews. Applies only with
RIVET_STATE_URL=postgresql://...; a SQLite state takes its lease as a file lock and is
unchanged. No state table, config key or TTL changed, and 0.31.0 and this release
exclude each other on the same lease rows.- What could happen in 0.31.0 and earlier. Every lease (one per table being loaded
or compacted, one more per table withcleanup_source, one per checkpointed export
run) was renewed by a thread of its own on a connection of its own, opened after the
lease was granted. When that connection could not be opened the load went on, printed
LOAD OKand exited 0; the lease lapsed afterRIVET_STATE_LEASE_TTL_S(30 s by
default), and a secondrivet loadorrivet compactof the same table could then
run beside the first. It took three conditions together: a PostgreSQL state; the
renewing connection failing to open (the process short of file descriptors, where the
PostgreSQL client crashed its thread, or the state server out of connections for the
role); and a load of that table still running after the TTL while another one started.
A lease another process had taken after a stall was logged and not acted on. - Now one connection and one thread per process renew every lease the process
holds, and they are started before the first lease is granted.- If that connection cannot be opened, the table is not loaded: the command fails with
RIVET_STATE_LEASE_KEEPER_UNAVAILABLE, having written nothing to the warehouse and
nothing to the load ledger, on every run until a connection is free. Free a
connection on the state database, raise the open-file limit, or lower--pool. - A load or compact whose lease is no longer its own (not renewed within the TTL, or
taken by another process) stops before the warehouse write with exit 5
RIVET_STATE_LEASE_LOST, recordedrefusedin the load ledger; the next
rivet loadloads the table. When the loss is found only after the write, the
command fails with exit 3RIVET_LOAD_LEASE_LOST_DURING_WRITEand the ledger row
isfailed, notsuccess: another load or compact may have written the table at
the same time, and the nextrivet loadloads the same runs again. - A checkpointed
rivet runtakes its run lease through the same connection and is
refused withRIVET_STATE_LEASE_KEEPER_UNAVAILABLEwhen it cannot be opened. A
PostgreSQL state that cannot be reached for want of file descriptors is anError:
line and exit 1, where 0.31.0 crashed with exit 101.
- If that connection cannot be opened, the table is not loaded: the command fails with
- Connections. A pooled load or compact holds one state connection per worker plus
one, so--pool 16is at most 17 connections to the PostgreSQL state where 0.31.0
held 32 (48 withcleanup_source). Count N + 1 against the role's and the server's
connection limits.
- What could happen in 0.31.0 and earlier. Every lease (one per table being loaded
-
Fix:
rivet loadinto BigQuery opens fewer files per table loaded at once. Each
table of a--pool Nload built its own HTTP client for the BigQuery API (a thread, an
event queue and its connections apiece); one client now serves every table of the
process. Measured on macOS, 16 tables from GCS at--pool 16: the peak of open files
goes from 204 to 139 (event queues 51 to 6), the load takes the same 14 s, and
--pool 1and--pool 4are unchanged at 18 and 30. The rest of the peak is one state
connection and the storage connections of each table, which this change does not touch.
No config, state or file format change. -
Fix: a PostgreSQL export through a transaction-mode pgBouncer that keeps prepared
statements is delivered under its own columns. Applies to every batch mode (full,
chunkedby key or by range,incremental) when the source URL points at pgBouncer in
transaction pooling mode withmax_prepared_statementsabove zero (pgBouncer's
documented default is 200). Reproduced on pgBouncer 1.26.0 with rivet 0.31.0; a direct
connection was never affected, and neither was MySQL through ProxySQL.- What was wrong in 0.31.0 and earlier. rivet read every table through a statement
of one text,FETCH n FROM _rivet. Such a pooler keeps a prepared statement by its text
and reuses it for every client, and PostgreSQL keeps the columns it described when that
text was first parsed. A table read after another one through the same pooled server
connection was decoded as the earlier table:- with the same number of columns the run exited 0 with a success manifest, and the
part carried the earlier table's column names and types (adouble precision1.5
delivered as theBIGINT4609434218613702656); - one table whose column type was altered between two runs was delivered under the old
type, exit 0; - with another number of columns, or after
ALTER TABLE .. ADD COLUMN, the run failed
withDataRow field count does not match the number of columns, on every run until
the pooler dropped its server connection; - after a chunk key was widened the run failed with
cached plan must not change result typeorsent a malformed wire payload, and a chunked run skipped its schema-drift
check withcould not resolve schema for drift check (skipping).
- with the same number of columns the run exited 0 with a success manifest, and the
- Now every statement whose columns follow a table (the cursor
FETCH, the bound and
count probes, the type probe) is prepared under a text of its own, so the pooler never
answers it with a statement another read left. The first run of this release through a
pooler that still holds statements of an older rivet delivers the table as the source
holds it. - If you ran 0.31.0 or earlier through such a pooler: compare each delivered part's
column names and types with the source table (DESCRIBE SELECT * FROM read_parquet('<part>')in DuckDB against\d <table>in psql). A part whose columns
are the table's own was decoded as that table. A part with another table's columns, or
a column under a type the table does not have, holds wrong values: remove it and
extract the export again with this release (rivet state reset -c <config> --export <name>first for anincrementalor checkpointed export, so the pass is a full one).
- What was wrong in 0.31.0 and earlier. rivet read every table through a statement
-
Breaking: a destination or format edited under stored progress is refused, and a run
resumed after a crash delivers the rows that arrived since. Applies toincremental,
keyset_incremental, MongoDBresumeandchunk_checkpointexports on every source
engine.- A moved destination or another format under a stored cursor is refused. Before, an
export that continues from a cursor kept it when itsdestination(bucket,path,
prefixor store) or itsformatwas edited with no reset: exit 0, and the new
location held only the rows past the cursor, or one prefix held parts of two formats.
Now the run is refused with exit 5RIVET_STATE_CURSOR_DESTINATION_MISMATCH, naming
where the cursor was delivered and where the config points, on every run. Restore the
destination and format to continue, or run
rivet state reset -c <config> --export <name>for a full pass delivered where the
config points now. The parts already delivered are not moved or removed: empty that
location first if the full pass lands in it too.rivet state acceptdoes not lift
this refusal. The destination is compared as written, so a{date},{export},
{table}or{run_id}placeholder is one destination on every day, and arivet plan
artifact sealed before this release still applies. A destination that comes from
${VAR}is compared after substitution: a value that moves the path between runs is
refused, so spell a per-run prefix with{date}. - An unfinished checkpointed run is not finished in another format. Before, a
chunk_checkpointrun interrupted while writing parquet and continued after
format: csvexited 0 over one success manifest naming parts of both formats. Now the
run, andrivet run --resume, is refused with exit 5
RIVET_STATE_INTERRUPTED_RUN_OWNER_MISMATCH, naming both formats. Restore the format
and run once to finish it, or abandon it (rivet state reset-chunksfor a range-chunk
run,rivet state resetfor a keyset run) and the next run starts a full pass in the
new format. This holds for a run 0.31 or earlier left unfinished too. - A resumed chunk-checkpoint run reads the source's key span again. Before, a plain
run or--resumethat continued a crashedchunk_column+chunk_checkpointrun
finished the plan the crashed run had stored: rows that reached the source past it
were in no part, under exit 0 and_SUCCESS, and an empty table that filled after the
crash was delivered as 0 rows. Now the resumed run detects the key span again, adds
tasks for the keys outside the stored plan, and plans a run that committed no part
again whole. The windows the crashed run committed are not read again. The units of
rivet apply --pool --split --resumeresume the same way, so rows gained past the top
key land in the last unit (before, the resumed split delivered the keys of the crashed
plan only).rivet applyof a sealed plan artifact replays the ranges its plan
sealed, as before. A resume now runs themin/maxand NULL-key probes a fresh run
does and fails where a fresh run would (a NULL key, a key span past the sparse-plan
limit). - A resumed parallel keyset run reads past its last range. Before, a
chunk_by_key+
parallel+chunk_checkpointrun whose last key range had committed before the crash
read no key above it on resume. Now it reads the keys past the highest one that range
delivered. A range committed by 0.31 or earlier recorded no such key: it is read again
whole, and none of its parts is declared twice. - A chunk run whose task rows are gone is planned again. Before, a resume over a
chunk_runwith nochunk_taskrows wrote a success manifest with 0 rows. - Upgrading: nothing to migrate, and the state schema version does not change. A cursor
row written by 0.31 or earlier holds no destination: the first run of this release
records where the export delivers, logs onepredates destination trackingwarning
and continues, so an edit of the destination or format made in the same step as the
upgrade is not seen; make the upgrade and the edit two separate runs. A run that 0.31
left unfinished is resumed with the behaviour above.
- A moved destination or another format under a stored cursor is refused. Before, an
-
Behaviour change: a table that is empty at its CDC baseline gets an empty warehouse
table. Applies to amode: cdcexport withcdc.backfill:loaded under
load.layout: base_buffer(BigQuery), every source engine.- Before, the load consumed the empty baseline as "nothing to load" and created no
<table>.rivet compactpassed the table by until its first change was buffered, and
from then on refused it on every cycle (the base table does not exist; 0.30 and older
passed BigQuery'sNot found: Tablethrough), so the command exited non-zero although
every other table was merged. The buffered changes stayed in<table>__changes. - Now the load of an empty baseline creates
<table>empty, with the spec's columns,
__is_deleted, partition and clustering, and the first change merges into it. A new
baseline that holds no row empties an existing base the same way, in every load; before,
it did so only when the same load also carried change files. - Upgrading: nothing to do. The first
rivet loadof this version creates the missing
table and printsnote: ... does not exist although its baseline was loaded, and the next
rivet compactmerges every change buffered since the baseline. It does so only when the
state DB beside the config recorded exactly one baseline run for the table, with 0 rows,
and no compaction ever merged a row into it. A table whose base was dropped by hand after
it held rows is still refused byrivet compact, as before, and so is one whose state DB
does not hold the extract's run records (a load that runs on another host). rivet compactover several tables exits 1 when at least one table failed, names each
failed table, and still merges and reports (COMPACT OK) the others.
- Before, the load consumed the empty baseline as "nothing to load" and created no
-
Breaking (file names): a single-runner part is named after its run, with the process id
and a random nonce appended. Applies tomode: full,mode: incrementaland time-window
exports that are not chunked, on every source engine, and to MongoDBparallelexports.- Before:
orders_20261008_174122_776.parquet(the UTC millisecond the read ended). Two
rivet runof one export that ended their read in the same millisecond wrote the same
name; the second file replaced the first, both runs exited 0, and both manifests,
file_logandexport_metricscounted the one file (two runs of a 200-row table:
400 rows recorded over one 200-row file). - Now:
orders_20261008_174117_653_33972_9f3a1c0b5d7e2a41.parquet, that is
<export>_<YYYYMMDD>_<HHMMSS>_<mmm>_<pid>_<nonce>[_partN].<format>: the UTC millisecond
the run started and its process id (both from the run's id), then 16 random hex digits,
so no two runs write one name, two containers with the same process id included. A
MongoDBparallelpart is
<export>_<YYYYMMDD>_<HHMMSS>_<mmm>_<pid>_<nonce>_w<worker>_keyset<page>.<format>. - What to change: a consumer that matches part names with a pattern anchored after the
millisecond field (orders_\d{8}_\d{6}_\d{3}\.parquet) must allow the_<pid>_<nonce>
fields.orders_*.parquetand a date prefix (orders_20261008_*) match as before, and
names still sort by time across the upgrade. Take part names frommanifest.jsonrather
than from a pattern. - Nothing else moves: no state or manifest format change. Parts written by earlier
releases keep their names and are read, validated and loaded as before, and a prefix may
hold both shapes. Chunked, keyset and CDC part names are unchanged.
- Before:
-
Breaking: stored progress belongs to one source, one table and one set of rows, and a run
that reads another is refused. Applies toincremental, keyset (chunk_by_key) and
MongoDBresumeexports on every source engine, and torivet state reset.- An edited
query:filter is refused. Before, an export whosequery:kept itsFROM
table but changed its filter (aWHERE, a join, a CTE) continued from the old filter's
cursor: exit 0 and none of the rows the new filter admits below it. Now the run is refused
with exit 5RIVET_STATE_CURSOR_STREAM_MISMATCH, naming both queries, on every run.
Restore the previous query to continue, or run
rivet state reset -c <config> --export <name>to deliver the new filter's rows in full,
or runrivet state accept -c <config> --export <name>to keep the cursor under the new
query (see below). A changed column list, letter case or layout is not an edit and
continues; any other change of the text afterFROM <table>is one, a renamed alias or
reordered predicates included. - A
${param}value that changes between runs is not an edit. The query is compared
as written, before--paramand environment values are substituted, so
WHERE region = '${region}'run with another region continues from the stored cursor
with no refusal and no warning, as it did before this release. An edit of the text
around the placeholder is refused like any other. Arivet planartifact sealed before
this release still applies, before or after a run of this release: it carries the
substituted query only, and a substituted query that fills the placeholders of the
stored one is the same rows. - New command
rivet state accept --config <PATH> --export <NAME> [--param KEY=VALUE].
It answers the refusal above with "the edit is meant, go on from where the export was":
the stored progress of that one export is recorded as belonging to what the config reads
now, the cursor value is not touched, and the next run continues from it (rows of the
new query below the cursor are not delivered;state resetdelivers them). It is given
per export and per edit; no setting accepts later edits. A cursor written for another
cursor_columnis not accepted (RIVET_STATE_CURSOR_OWNER_MISMATCH). Each acceptance
is listed byrivet journalwith statusacceptedand a line naming both queries. - The same table under another PostgreSQL
search_pathis refused. Before, a source
URL that setoptions=-c search_path=<schema>moved an unqualified table to another
schema under the cursor of the first: exit 0, zero rows. Now it is the same refusal with
the same three remedies. Qualifying the table by hand with the schema it was read through
continues. rivet state resetclears one source. Before,state reset --export Xdeleted the
cursor of every export namedXin the state database, so a same-named export of another
source re-delivered its whole table on its next run. Now it clears the progressXholds
on the source its config names, and the config's source URL must resolve (a missing
url_envvariable stops the reset with nothing removed).- A source URL without its default port is the same source. Before, dropping or adding
:5432/:3306/:1433/:1521/:27017in the URL started a whole-table pass
under a second state row. Now both spellings are one source and the run continues. - A resumed parallel keyset run stores the run's highest key. Before, a
chunk_by_key+parallel+chunk_checkpointrun resumed after a crash stored the
highest key of the ranges it re-ran (or none), so a laterincrementalrun on the key
delivered rows again. Now each committed range records its highest key and the resumed
run stores the highest of all. - Upgrading: nothing to do. State written by 0.31 or earlier is read as follows. A cursor
row stored under a URL without its default port is moved onto the spelled source on the
first run orstate resetof that export and continues (if both spellings hold a row,
the one under the spelled port is kept). A cursor row holds no query and no
search_pathyet: the first run records the ones the export has now, logs one
predates stream trackingorpredates query trackingwarning, and continues; edits
are compared from the second run on, so make the upgrade and a filter edit two separate
runs. A parallel keyset run interrupted before the upgrade and resumed after it stores
the highest key of the ranges that commit after the upgrade, as 0.31 did. The state
schema moves to v35 (keyset_range.max_key), so an older rivet no longer opens the
state database.
- An edited
-
Breaking: a CDC stream that cannot continue exits 5 with a
RIVET_*code, where it exited 1
with none. Applies tomode: cdcruns andrivet cdcon PostgreSQL, MySQL, SQL Server,
MongoDB and Oracle.RIVET_SOURCE_CDC_CHECKPOINT_INVALID: a checkpoint file that is not JSON (every engine),
and a checkpoint file that is gone while the export has a baseline (MySQL, SQL Server,
MongoDB, Oracle).RIVET_SOURCE_CDC_TRUNCATED: a TRUNCATE of a captured table (PostgreSQL, MySQL; Oracle
gave it already), a DROP of one (MySQL), a dropped captured collection or database
(MongoDB).RIVET_SOURCE_CDC_LOG_GAP: a PostgreSQL slot missing under a checkpoint or a baseline,
MySQL binlogs purged past the checkpoint (ERROR 1236), a MongoDB oplog rolled past the
resume token (error 286). The MongoDB refusal used to end with "change streams require a
replica set", which was not the cause; it now names the oplog and the re-baseline.- The corrupt-checkpoint message said "delete it to accept a new anchor from a fresh
snapshot". That was wrong on every engine but PostgreSQL: with a baseline the next run
refused the missing file, and without one it anchored at the current position and skipped
every change since the checkpoint. The message now ends "Restore the file, or:" and the
re-baseline steps, the same ones every other CDC data-loss message prints. - What is refused has not changed, and nothing stored changes: a checkpoint and a state
database written by 0.31.0 resume as before. - Upgrading: a script or scheduler that matched exit 1 for these cases now sees exit 5
(refusal: do not retry, follow the remedy in the message).rivet schema errorslists
the three codes.
-
Breaking: a run that fails before its first write leaves the destination as it was.
Applies to every batch export (full,incremental,chunked, keyset, the snapshot leg
of a CDC export) on every source engine and on every destination that keeps objects at
rest (local, S3, GCS, Azure).- Before, a run that could not reach the source (exit 2), was given a wrong password, or
failed in any other way before it wrote a part, still rewrotemanifest.jsonto
status: failedwith 0 rows, added amanifest-<run id>.jsoncopy and deleted
_SUCCESS. After a network failurerivet validateexited 1 and a loader or sensor
keyed on_SUCCESSstopped seeing an export that was whole. - Now
_SUCCESSand asuccessmanifest mean "this prefix holds a whole export", not
"the last run succeeded". Over a prefix that already holds a manifest such a run writes
nothing to the destination; the failure is told by the exit code and the run journal
(rivet metrics,export_metrics). On a prefix with no manifest yet (the export's
first run) it records its failure as before: afailedmanifest with 0 parts and no
_SUCCESS, sorivet validateexits 1 on a prefix whose only run failed. A run that
put a part at the destination and then failed still deletes_SUCCESSand writes a
failedmanifest naming those parts. - What a scheduler or loader that reads the destination sees differently: after a run
that failed before its first write,_SUCCESSis still there,manifest.jsonstill
sayssuccesswith the previous run'srun_idandfinished_at, and no
manifest-<run id>.jsonof the failed run is added. A consumer that used "the marker is
gone" or "the manifest says failed" as its signal that the last run failed must read
the run's exit code, or compare the manifest'srun_idwith the run it started. --resumereads the run journal beside the marker, so its behaviour after such a
failure is what it was:rivet apply <config> --resumeruns again an export whose last
journaled run failed, even though an earlier run's_SUCCESSis still at its prefix, and
rivet run --resumeis not refused over that marker. Both readexport_metricsin the
state database the config names; with a state database that has no row for the export
the marker alone decides, as before.- Upgrading: nothing to migrate. A prefix that 0.31 or earlier already re-marked as
failed stays so until the next successful run writes its manifest and marker.
- Before, a run that could not reach the source (exit 2), was given a wrong password, or
-
Breaking:
--resumeover a complete prefix is refused by code, and--resume --force
runs on every strategy. Applies torivet run --resumeon every source engine.- Before,
--resumeinto a prefix that holds_SUCCESSexited 1 with no code and said
"Re-running would overwrite a verified dataset. Pass --force to override". On a
chunk_checkpoint: trueexport--resume --forcethen exited 1 with "nothing to
continue" and re-marked the complete prefix as failed; on the other strategies it
exported again. Nothing was ever overwritten. - Now the refusal is
RIVET_DEST_ALREADY_COMPLETE(exit 5, was 1) and says what
--forcedoes: it continues an interrupted run of the export if one is recorded, else
the run goes on as a plain run does (afullexport is exported again, a delta export
continues past its cursor); new parts land beside the old ones andmanifest.jsonthen
describes only the new run.--resume --forcedoes that on a
checkpointed chunked export too.--resumewithout--forceand with no run to
continue is refused as before ("nothing to continue; run without --resume"). - A script that matched exit 1 or the old text of this refusal must match exit 5 or the
code.
- Before,
-
Behaviour change: a CDC run that refuses a prerequisite leaves nothing behind, and
rivet doctoragrees withrivet run. Applies tomode: cdcandrivet cdcon
PostgreSQL and MySQL.- Before, on PostgreSQL a run refused for a table that does not exist had already created
its replication slot (inactive, pinning WAL); a bounded run on a standby was refused only
after the slot, the baseline andsnapshot/_SUCCESSwere written. On MySQL a first run
refused for binlog compression, a replica that does not re-log, or an unroutable table had
already written its checkpoint.rivet doctorreportedAll checks passedfor a
PostgreSQL server withwal_levelbelowlogicaland for a MySQL replica with
log_replica_updates = OFF. - Now every prerequisite is asked before the first durable write, and these refusals carry
RIVET_SOURCE_CDC_PREREQUISITE(exit 1, as before): no slot, checkpoint, baseline or state
row is left. A PostgreSQL table that does not exist andwal_levelbelowlogicalare
named by rivet instead of by the server's error. Doctor asks the same function the run
refuses by and fails aCDC run prerequisitescheck with the run's own message. rivet init --mode cdcagainst a PostgreSQL standby writesuntil_current: false, the
mode a standby runs; it wroteuntil_current: true, which the run refuses there.- Upgrading: nothing to do for a healthy stream. A slot or checkpoint left by a run 0.31
refused is reused as it is once the prerequisite is met; if the export was abandoned
instead, drop the slot (SELECT pg_drop_replication_slot('<slot>')) or delete the
checkpoint file.
- Before, on PostgreSQL a run refused for a table that does not exist had already created
-
Behaviour change:
rivet cdcto stdout emits each change once on PostgreSQL. Before,
a drain to stdout never advanced the slot, so every run printed every change since the slot
was created, with or without--checkpoint, and the slot pinned WAL without bound. Now a
drain acknowledges the last commit it printed when it ends. Upgrading: the first drain with
this version prints the backlog the slot still holds once more, then the slot advances. A
consumer that relied on each run re-printing the history must keep its own copy. -
Fix: PostgreSQL CDC with
rollover: 1(--rollover 1) delivers the pending changes.
A bounded run leaves its drain barrier (a logical message) in the WAL; atrollover: 1the
next run's one-change read window held only that message, and the run exited 0 with a
Success manifest of 0 rows while commits were pending, on every run. Withrollover: Nthe
same happened after N idle bounded runs in a row. Messages outside a transaction are now
stepped over, up to the run's open-time bound. Upgrading: the first run with this version
delivers what such a slot was holding. -
Breaking: SQL Server CDC refuses a log gap from the first run after the baseline.
Applies to amode: cdcSQL Server export withcdc.initial: snapshot.- Before, the checkpoint the baseline wrote (
"pinned": true) could sit below the capture
instance's start, and a run moved such a position up to the start. If the change table lost
rows before the first run that read a change (retention cleanup past them, or the capture
instance disabled and re-enabled, which is also the remedy the ADD COLUMN refusal names),
the next runs ended with exit 0 and those changes were never delivered. - Now the baseline's anchor is always inside the capture instance's lifetime, and any resume
position below the instance'sstart_lsnis refused: exit 5RIVET_SOURCE_CDC_LOG_GAP
with the re-baseline remedy, on every run until the stream is re-baselined. An idle run now
moves a baseline-only checkpoint forward like any other, so a quiet table is not refused
when the cleanup job passes its anchor. - Upgrading: a checkpoint that already read a change needs nothing. A baseline-only
checkpoint written by 0.31 or earlier that is below the instance's start is refused on
the first run of the new version, with the same exit 5RIVET_SOURCE_CDC_LOG_GAP: the
file cannot tell a harmless anchor from a gap that already happened, so rivet does not
guess. This affects a table that had no captured change since its baseline while the
cleanup job ran at least once, or whose capture instance was enabled just before the
baseline. Re-baseline such a stream once, as the refusal says.
- Before, the checkpoint the baseline wrote (
-
Breaking:
rivet run --validateexits non-zero when its validation fails. Applies to
every source engine and every destination that keeps objects at rest.- Before, a run whose end-of-run verification failed printed
validated: FAILand exited 0,
whilerivet validateon the same prefix exited 3. A script or scheduler keyed on the
run's exit code learned nothing. - Now the run exits as
rivet validatedoes on that prefix:RIVET_VALIDATE_FAILED
(exit 3) when a check found the destination wrong (a missing part, a size or checksum
mismatch,verify: contentnot met),RIVET_VALIDATE_UNVERIFIED(exit 1) when the
destination could not be read back. The error line names each failure. The export itself
is not undone: its parts, manifest,_SUCCESS, metrics row and cursor are as before. - Upgrading: a job that ran
rivet run --validate && nextnow stops beforenextwhen
the verdict is FAIL. The case to look for isverify: contenton a destination that
exposes no content checksum, such as a local path: those runs printvalidated: FAIL
today and exit 0. Useverify: sizethere, or drop--validate.
- Before, a run whose end-of-run verification failed printed
-
Breaking: a run over a state database it cannot write is refused, and a cursor that could
not be stored fails the run. Applies torivet runandrivet apply, every batch mode
and source engine.- Before, with a read-only
.rivet_state.db(file permissions, a read-only volume, a
read-only role) a run logged warnings, exported, and exited 0. An incremental run could
not advance its cursor, so every later run delivered the same rows again; no run left a
metrics row. - Now the run is refused before it exports anything (
RIVET_STATE_NOT_WRITABLE, exit 1),
on every run, until the state takes the write. Make the state database writable and run
again: the run then delivers from the stored cursor. - A read-only DIRECTORY around a writable state file (a read-only volume mount) gets the
same refusal, from every command that opens the state. Before, it failed with exit 2 and
could not acquire the migration lock ... Another rivet process is migrating this state database; wait for it to finish and retry, although no other process existed. - If the state fails only at the end, when the rows and the manifest are already written,
the run exits non-zero withRIVET_STATE_CURSOR_NOT_STOREDinstead of logging an error
and exiting 0. The rows are delivered and the next run delivers them again from the
previous cursor, as before; the exit code now says so. - Not changed:
mode: cdcruns, achunk_checkpointrun (it already failed, with
Permission denied), and commands that only read the state (state show,metrics). - Upgrading: nothing to do for a writable state. A deployment that ran on a read-only
state and relied on exit 0 now sees exit 1.
- Before, with a read-only
-
Breaking: a
RIVET_STATE_URLthat is not a PostgreSQL URL is refused. Applies to every
command that opens the state (run,plan,apply,load,state,metrics, ...).- Before, any value that did not start with
postgres(a typo such aspostgre://,
another engine's URL) was ignored without a message: the command kept its state in a
SQLite file beside the config. In a container that file is lost with the container, so
every run started from no cursor. - Now it is refused with
RIVET_STATE_URL_SCHEME_UNSUPPORTED(exit 1); no SQLite file is
created. SetRIVET_STATE_URLto apostgres://orpostgresql://URL, or unset it to
keep the state in.rivet_state.dbbeside the config. An empty value still selects
SQLite, as an unset one does. - Upgrading: a deployment with such a value has been running on SQLite. After correcting
the URL the PostgreSQL state is empty, so each incremental export starts with a full
pass; to keep the stored cursors, unset the variable instead and keep the SQLite file.
- Before, any value that did not start with
-
Breaking:
rivet planfails over a source it cannot read, and writes no plan.
Applies to every engine.- Before, when the preflight failed (no connection, a login the source rejects, a table that
does not exist),rivet planprinted a WARN, exited 0 and wrote a plan whose diagnostics
saidunknown (preflight failed);rivet applythen failed on it. On MongoDB a collection
that does not exist was planned with no warning at all. - Now the command ends with
RIVET_PLAN_SOURCE_UNREADABLEand the source's own error, and
writes no plan file for any export of the config: exit 1, or exit 2 when the cause is
transient (a refused or dropped connection). A script that relied on exit 0 here now sees
a failure; fix the connection, the credentials or the table name and plan again. - Upgrading: nothing to migrate. A plan file written by 0.31 or earlier over an unreadable
source is still accepted byrivet apply, which fails on it as it did before.
- Before, when the preflight failed (no connection, a login the source rejects, a table that
-
Breaking:
rivet validateexits 1 on a bucket or container that does not exist.
Applies tos3,gcsandazuredestinations.- Before, a destination whose bucket does not exist (a typo, another account) was reported
asstatus: legacy_run, exit 0: the same answer as a prefix that was never written. - Now the export's verdict is
NO MANIFESTwithRIVET_VERIFY_MANIFEST_READ_ERRORnaming
the bucket, and the command exits 1 (could not be verified). An empty prefix of a bucket
that exists is stilllegacy_run, exit 0, and so is a local path that does not exist. - The same question is asked wherever rivet looks for an object and the store answers 404
(run --resume,repair,reconcile): a missing bucket is now
RIVET_DEST_CONTAINER_NOT_FOUNDthere instead of "the object is absent". It costs one
extra listing request (one result) per destination per process.
- Before, a destination whose bucket does not exist (a typo, another account) was reported
-
Breaking:
rivet checkdescribes the planrivet runbuilds.
Applies tomode: chunkedon PostgreSQL, MySQL, SQL Server and Oracle, and to MongoDB.- Before, the
Strategy:andMode:lines were derived from the config alone. A
mode: chunkedexport with nochunk_columnon a table that fits one chunk printed
Strategy: chunked(id, size=100000)andMode: chunked (column: ?, ...), while the run
exported it as one unchunked pass. - Now both lines (and
strategy/modeunder--json) come from the plannerrunuses:
Strategy: full-scanandMode: full (mode: chunkedruns as one pass: ...)for that
table, the resolved primary key instead of?when the run does range-chunk, and the
resolvedchunk_sizeunderchunk_size_memory_mb. The verdict, warnings and
recommendations are computed for the same resolved plan.rivet planrecords its
diagnostics for the resolved plan as well. - On MongoDB,
rivet checknow fails on a collection that does not exist
(RIVET_SOURCE_COLLECTION_NOT_FOUND, exit 1), with the refusalrivet rungives; before
it printedLooks goodand exited 0. - Exit codes of
rivet checkare otherwise unchanged; a tool that parses thestrategyor
modefield sees the new values for the exports above.
- Before, the
-
Breaking: a CDC table put back into
tables:is refused until it is re-baselined.
Applies tomode: cdcexports withcdc.initial: snapshotorbackfill:, on every engine.- Before, a table removed from
tables:and added again later kept its old baseline: the run
exited 0 and every insert, update and delete made while the table was out was missing from
the destination. The same happened whentable:was pointed at another table and the
destination already held the first table's baseline: the new table got no baseline at all. - Now each run records the tables the stream captures. A table that the last run did not
capture, whose baseline is already recorded complete, is refused
(RIVET_STATE_CDC_TABLE_REJOINED, exit 5, nothing read or written), on every run, until
it is re-baselined: move every file out of the table's directory in the destination,
delete itscdc_snapshotrow from the state DB, and truncate its__changestable if a
warehouse load consumes the stream. The next run reads the table in full and captures it
from there. A table that was never captured is still baselined when it is added, as before. - Upgrading: nothing to do and no state migration. The first run on the new version records
the current table set without checking it, so a table that was removed and put back
under 0.31 or earlier is not detected: re-baseline it as above if that happened.
- Before, a table removed from
-
Breaking: a keyset run that a gate refused is read again, not adopted, by the next run.
Applies tochunk_by_keywithparallel: N+chunk_checkpoint: true, and to
keyset_incremental: true(sequential or parallel), when aquality:content rule
(unique_columns,null_ratio_max) oron_schema_drift: failrefuses the run after its rows
were written.- Before, the refused run kept its checkpoint. The next run adopted its pages, read nothing,
graded nothing, and ended with exit 0,_SUCCESSand "quality: pass" over the refused
data; the refused run'srun_statusrow was rewritten tosuccess. - Now every re-run reads the same rows again and is graded again: it keeps exiting 3
(quality) or 4 (schema drift) until the source data or the rule is fixed, and the refused
runs stayfailed. Each refused run leaves its parts on the destination under a Failed
manifest, as a refused run without a checkpoint always did. - A sequential
keyset_incrementalrun no longer moves the stored cursor page by page: the
cursor moves once, after the manifest is written, so a refused or interrupted run leaves it
where the run began. A run that breaks off mid-way is still resumed from its last
committed page. - Upgrading: an export whose last run finished needs nothing. An export whose last run
under 0.31 or earlier was refused by such a gate still gets the old behaviour once: the
first run on the new version adopts the pages that run left. The same holds for a
sequentialkeyset_incrementalrun interrupted under 0.31 or earlier (its cursor had
already moved): if the new version resumes it and a gate refuses it, the run warns that
it "cannot be read again", names the command below, and the next run adopts its pages. To
have those rows read and graded again, runrivet state reset -c <config> --export <name>
before the next run.
The export then starts from the beginning: akeyset_incrementalexport reads the whole
table again.
- Before, the refused run kept its checkpoint. The next run adopted its pages, read nothing,
-
Breaking: a run of another mode refuses to continue past an unfinished checkpoint run
(RIVET_STATE_INTERRUPTED_RUN_OWNER_MISMATCH, exit 5, nothing read or written). The stored
record of an export now says whether it is a committed high-water or an interrupted run, and
which mode (keyset,chunked) opened that run.- Before (0.31.0 and earlier): a keyset run with
chunk_checkpoint: truecrashed after its
first page, the export was switched tomode: incrementalwithout a reset, and the
incremental run continued from the crashed run's page cursor: exit 0,_SUCCESS, a manifest
listing 1900 of 2000 rows. - Now any other mode that meets the interrupted run is refused, including
mode: fulland a
switch away from an interrupted range-chunk run (which lost nothing before; it is refused so
the interrupted run is finished or abandoned on purpose). Remedy, either one: restore the
previous mode's settings and run once to finish the run, then switch; or abandon it with
rivet state reset -c <config> --export <name>(keyset run) orrivet state reset-chunks -c <config> --export <name>(range-chunk run), after which the next run is a full pass. - A switch after a run that FINISHED is unchanged: keyset then incremental on the same key
still continues from the high-water.
- Before (0.31.0 and earlier): a keyset run with
-
Breaking: a run without the checkpoint is refused beside an unfinished checkpoint run
(RIVET_STATE_INTERRUPTED_RUN_OWNER_MISMATCH, exit 5, nothing read or written), the same way a
run of another mode is.- Before (0.31.0 and earlier): a range-chunk run with
chunk_checkpoint: truewas killed after
6 of 20 chunks; the export ran once withchunk_checkpointremoved and finished (exit 0); 20
source rows changed, one per chunk; withchunk_checkpoint: trueback, the next run resumed
the killed run: exit 0, manifestsuccess, 120000 rows, and 14 of the 20 changed rows. The
other six chunks were the parts written before the kill (PostgreSQL, MySQL, SQL Server). - Now the run without the checkpoint is refused while the interrupted run exists: range chunks,
keyset, parallel keyset, and a MongoDBpage_sizeexport withresumeremoved. Remedy, either
one: restore the checkpoint setting and run once to finish the run, then remove it; or abandon
it withrivet state reset-chunks -c <config> --export <name>(range-chunk run) orrivet state reset -c <config> --export <name>(keyset run, MongoDB), after which the next run is a
full pass. - Changed for keyset: a keyset run without
chunk_checkpointused to read the whole table
beside an interrupted checkpointed keyset run and release it on success. It is refused now. - An interrupted range-chunk run counts even when the export's stored row no longer points at
it (afterrivet state reset, or a run left by 0.31.0): another mode and a run without the
checkpoint are refused there too.rivet state reset-chunksis the command that abandons it. - Upgrade, state written by 0.31.0 and earlier: an interrupted chunk run that nothing finished
beside is refused without the checkpoint and resumed with it, as above. One that a later run
of the export already finished beside (those versions allowed it) is refused for every config
(RIVET_STATE_CHUNK_RUN_SUPERSEDED, exit 5) untilrivet state reset-chunks -c <config> --export <name>; the next run starts a fresh pass. Both paths were run with the 0.31.0 binary
first and this version second.
- Before (0.31.0 and earlier): a range-chunk run with
-
Fixed: an interrupted range-chunk run is resumed only by the source that opened it. Two
configs with one export name, one state database and different sources: when the first crashed
after a chunk, the second resumed the first one's chunk windows over its own table (exit 0,
1000 of 5000 rows). Each source now has its own in-progress run; state schema v34 records the
source onchunk_runandkeyset_range.- Upgrade: a chunk run left in progress by 0.31.0 recorded no source. It is resumed, with one
warning, by the next run of that export name when the name has progress under no other source;
when it has, the run is refused (RIVET_STATE_CHUNK_RUN_OWNER_UNKNOWN, exit 5) until
rivet state reset-chunks -c <config> --export <name>, and the next run starts a fresh pass.
Finished cursors and interrupted keyset runs carry over unchanged. Older binaries refuse a
v34 state (RIVET_STATE_SCHEMA_NEWER).
- Upgrade: a chunk run left in progress by 0.31.0 recorded no source. It is resumed, with one
-
CDC memory: the events of one transaction share its commit position. Closing a
transaction copied its resume position, a JSON object, into every event (about 475 to 730
bytes each), which was the largest single item on the heap of a large transaction. The
position is now one shared value. Measured on MySQL, one transaction: 500 000 changes
940 MB -> 427 MB peak and 1.90 s -> 1.39 s; 20 000 changes 77 MB -> 51 MB. Peak memory still
grows with the largest transaction. The transaction-buffer and rollover byte budgets charge a
position once per run of events that share it, so they keep tracking real memory. Output,
checkpoint files and the NDJSON stream are unchanged. -
Breaking: an incremental or CDC load refuses a view or a
<table>__changesit has no record
of loading, on BigQuery, Snowflake and ClickHouse. Before, only a foreign TABLE at<table>
was refused: a view there was replaced by rivet's current-state view, a foreign
<table>__changeswas appended to, and the load exited 0. A base-and-buffer load appended its
buffer beside a base it had never loaded, and that success made the base rivet's own for
rivet compact.- The load now stops before any warehouse write, with exit 5 and
[RIVET_LOAD_TARGET_NOT_RIVETS], and names the object. The whole-table refusal
(refusing to overwrite ...) carries the same code and exits 5; it exited 1. - What "no record" means: the state DB this
rivet loadopened has no load of that target.
Tables, views and logs loaded under the same state DB are unaffected, and so is a load
without a state DB. A load pointed at a NEW state DB is now refused where it used to
append: a lost state DB, another host, or a container whose state DB does not outlive the
run (refused from its second cycle). Keep the state DB between runs (a volume, or a
PostgreSQL state), restore it, or drop or rename the view and the log and load again. - A state DB that is configured and cannot be read refuses the same objects, as it already
did for a table. - A stop before the write exits as what stopped it.
RIVET_LOAD_ADOPTION_COLUMN_MISMATCH
now reaches the error line and exits 5 (it exited 1 with no code).
- The load now stops before any warehouse write, with exit 5 and
-
A load that stops before writing never becomes the reason the next load overwrites. The
stale-buffer stop of a base-and-buffer load, and a staged part that vanished between the
ledger read and the write, were recordedfailed; the nextrivet loadthen treated a table
it had never loaded as its own and replaced it. They are recordedrefused. The ownership
question is also asked before thewritingledger row, so a load killed while refusing, or
one whose state DB failed its closing write, leaves no row that claims the table. -
PostgreSQL CDC: a failed run no longer acknowledges the slot past what it wrote. When a
part rolled in the middle of a read window (most often throughrollover_memory_mb) and a
later part of the same run then failed to write, the run still advanced the slot's
confirmed_flush_lsnto the last transaction it had READ. The