Repository navigation
v0.25.0
-
New:
rivet state runsandrivet state finish-run— the frozen-prefix
escape hatch.runslists the run-status ledger (gc/cleanup's activity
signal);finish-runterminal-stamps a row you KNOW is dead (hard crash with
no successful successor — nothing else can release it now that supersession
is success-only). The cleanup refusal names the commands. -
CDC re-baseline loads now REFUSE instead of appending silently-wrong
data. A snapshot-carryingrivet loadinto a<table>__changesthat
already holds change rows hard-fails with a per-warehouseTRUNCATE
prescription: a re-snapshot row cannot win the dedup against already-loaded
changes (and no snapshot can express a PK deleted during the gap), so the
view would silently keep serving pre-gap values. Stateless loads degrade to
a note; an errored state store refuses (fail-safe). -
Snapshot legs are anchor-stamped. On checkpointed MySQL/SQL
Server/MongoDB flows theinitial: snapshotparquet now carries constant
__pos(the CDC anchor) and__seq = -1columns, so a re-baseline orders
correctly in the warehouse dedup. PostgreSQL-without-a-checkpoint and
pre-existing snapshots keep the legacy NULL shape. Anyone querying
__changes WHERE __pos IS NULLto find baselines sees the new shape. -
rivet validategot stricter — previously-green runs can now exit 3.
CDC prefixes presence-check EVERY cycle's committed parts (the dataset is
the union of run-unique manifests; deleting a historical part used to
validate PASSED), and a declared part that is present at its size but does
not decode as Parquet is VERIFIED-WRONG even when the manifest records no
checksums. Budget accordingly:--depth fullon a long CDC stream now
honestly re-reads the accumulated history. -
Resume verifies the destination. A chunked resume re-exports committed
chunks whose parts were deleted between attempts (destination probe + the
M8 rewrite path); a keyset resume REFUSES over a deleted committed page
(namingrivet state reset) instead of finalizing Success over a hole. -
gc/census supersession requires a newer SUCCESS. A failed or
merely-running successor no longer retires a live run's marker; a failed
run's parts are spared until a Success supersedes them (checkpoint resumes
may still adopt them); deadrunningmarkers are retired via the
terminal-ledger sweep oncefinish-run/the split stamp closes their rows. -
New refusals of previously-accepted configs (each was a silent
misbehaviour):format: csvunder aload:block;load:on a non-GCS
destination (it silently listed a same-named GCS bucket); destination
prefixes with a..segment or an embedded URI scheme;--splitwith
{date}/{run_id}placeholders; equivalent checkpoint-path spellings; a
MySQL CDC open whose server-identity query fails (no more unverifiable
anchors); a MongoDB resume token whose_datais not a string. -
rivet initgenerates case-exact configs. Mixed-case tables route
through a quotedquery:(aCaseTwin/casetwintwin used to silently
export the WRONG table's rows), SELECT columns are quoted per engine (the
same twin bug at column level), Mongo always emitstable:(hyphenated
collections get an honest SKIPPED comment), and-orefuses to overwrite
an existing file. -
CSV: NULL and the empty string are distinct at last. A present empty
string renders as""; NULL stays the bare field (the PostgreSQL COPY
convention). Existing consumers see different bytes for empty strings;
DuckDB readers needallow_quoted_nulls=falseto recover the distinction. -
Secrets scrubbed from more places. URL query parameters carrying
token/password/api-key/signature families are redacted in errors and logs;
a Slack webhook delivery failure no longer prints the webhook URL (its
path IS the credential); load-path errors route through the redactor. -
Smaller: BigQuery DDL backticks every column (reserved-word columns
survive); a set-but-missingGOOGLE_APPLICATION_CREDENTIALSis a loud
error naming the path;RIVET_BQ_ACCESS_TOKENnow takes precedence over
ADC resolution;rivet checkpreflight understands Mongo keyset
(page_size) and stops sparse-alarming configs that took its own
chunk_countadvice; the doctor recognises MySQL's "Unknown database". -
CDC memory budgets measure memory now, with an operator-visible surface.
rollover_memory_mbgained a default (256 MiB) so absence no longer means "no
byte budget" — wide-row tables roll parts by resident size where they previously
buffered to the row cap; part counts can change for such configs. The
RIVET_CDC_MAX_TX_*caps are measured in RESIDENT cost (what the buffer actually
holds) instead of a payload estimate that under-counted ~12.7x, so a transaction
that slipped under the old 2 GiB figure may now refuse — the refusal was always
the intent, the old number was not a guard. Zero is rejected for both rollover
knobs at validation (0meant "roll a part on every transaction", not
"disable").bytes_readmetrics return to approximately the released DECODED-payload
unit (the old +32/event constant is gone — percent-level on realistic rows)
and stay comparable with batch exports. -
Opt-in transaction spilling (
RIVET_CDC_SPILL_DIR). A transaction past the
cap can spill its tail to disk instead of failing (PostgreSQL/MySQL/SQL Server;
MongoDB needs none by construction). Off by default: it moves the adapter's copy
only (~11% of peak RSS measured on PostgreSQL), so the caps stay the honest
guard. On the config-driven path every resolved location is config-anchored (the ad-hocrivet cdcCLI anchors to the invoking shell's directory — the directory the operator chose). -
A service-account key file is now minted IN PROCESS —
gcloudis off the
BigQuery path for it. WithGOOGLE_APPLICATION_CREDENTIALSpointing at a
service_accountJSON, rivet's shared Google auth seam
(destination::gcs_auth) returned "not my shape" and the BigQuery loader
shelled out togcloudfor a token. Rivet now builds the RFC 7523 claim set,
signs it RS256 with the file's ownprivate_key, and exchanges it for an
access token itself — the same seam, so the GCS destination and the BigQuery
loader mint from ONE implementation and a run's two legs act as ONE identity.
Verified against a real service account:INFORMATION_SCHEMA.JOBS_BY_PROJECT
recordsuser_email = rivet-export@….iam.gserviceaccount.comfor the load's
jobs, with thegcloudfallback compiled to abort so a green run could not
come from the subprocess. Two behaviour notes: a service account is billable
on its own, so rivet no longer sendsx-goog-user-projectfor it (the header
would demandserviceusage.services.use, which the roles that grant a load do
not include); andexternal_account/ workload identity plus GCE metadata
still use thegcloudfallback — an STS exchange rivet does not model, and a
half-implemented subset would be worse than the documented subprocess. The
crypto isjsonwebtoken, already in the tree viareqsign: zero new crates,
zero added build time. Extension seam (ADR-0026): a WIDENING, not a break —
AdcUserTokenLoader/try_authorized_user_loader()keep their names and
signatures and gaincredential_kind()/principal(). -
A multiplex CDC export (
mode: cdcwithtables:) can now be loaded.
The configrivet init --mode cdcemits for a whole schema carries no
table:orquery:, sorivet loadbailed on it outright ("must specify
exactly one of 'query', 'query_file', or 'table'") andrivet check --json
silently dropped itscolumnsblock. Both halves now fan out per table: the
type report emits one document per table (with a newtable:field and a
cdc/ordersdisplay name), and the loader builds one plan per table — its
own warehouse table, its own<base>/<table>/staging prefix. That prefix is
the load-critical half: the manifest listing is recursive, so N plans sharing
one base prefix would each sweep in every sibling table's parts and merge N
source tables into one warehouse table with every count agreeing. -
BigQuery loads no longer need the
bqCLI (or the Google Cloud SDK) on
PATH.rivet load --target bigquerytalks to the BigQuery JSON API
directly over HTTPS, using the same Application Default Credentials rivet
already uses for GCS — one Google auth path, not two. A container or CI image
that installed the Cloud SDK only for rivet can drop it. Job labels, the
4,000-partition batch split,PARSE_JSON, and the partition-quota error hint
are unchanged; error messages now name the BigQuery job id. If ADC is a user
credential with no usable refresh grant, rivet still asksgcloudfor a token
when it is present, and honoursRIVET_BQ_ACCESS_TOKENwhen it is not. -
rivet loadno longer shells out to itself. The load planner obtained
every column's warehouse type by SPAWNINGrivet check --target X --jsonand
parsing the child's stdout; it now calls the same resolver in process
(preflight::collect_type_reports, the function that command renders from).
Version skew between the loading binary and the type-resolving one is gone by
construction, so the--rivet-binflag that existed only to mitigate it is
removed (passing it is now an error — it had no other use), and the config
is parsed ONCE instead of twice with two different${VAR}resolutions. The
plan also keeps the resolver's wholeTargetColumnSpec—autoload_type, the
note and the L5cast_sqlwere silently dropped by the four-field JSON mirror.
rivet load's planning step is now reachable from offline tests for the first
time. -
ADR-0028: one finalize seam for the export tail. The post-write tail
(manifest schema-fingerprint pin,on_schema_driftgate, Form-B checksum
harvest, shape-drift warn) is now applied exactly once, at
finalize::finalize_export— called by the dispatcher between the runner
returning andfinalize_manifest— from aCommitLedgerthe runners feed as
they commit. Previously every export runner re-assembled that tail by hand,
which is how a feature wired into one runner ended up silently absent on the
others (the recurring runner-bypass class: the keyset/mongo drift-gate miss,
Form-B computed-then-discarded on the large-table runners). Runners now only
feed the ledger; the ordering constraints live in one place. Behavior is
unchanged except keyset/mongo manifests now also carry the dest-schema
fingerprint single mode always recorded (uniform pin at the seam). -
rivet planis now READ-ONLY without--annotate-waves. It computes and
prints the advisory schedule (and the reviewable plan artifact) but no longer
touches your config file unless you pass--annotate-waves. Previously, plan
gated only the overwrite of an existingwave:/parallel_safe:behind the
flag but still SILENTLY ADDED those fields to any export that lacked them — so
a read-only-lookingrivet planmutated a fresh config (the ADD-arm of the
0.24.4 config-clobber incident, whose overwrite half--annotate-waveswas
introduced to require consent for). Now config mutation is a single explicit
opt-in: no flag ⇒ no write, present field or absent.--annotate-waves
(over)writes the whole schedule as before.rivet applyis unaffected —
exports with nowave:run last, as one implicit final wave. -
Keyset part filenames no longer repeat the export name. Parts landed as
exports/<export>/<export>_<export>_<stamp>_pk_w3_2.parquetbecause the
keyset name format prepends the export name AND derives its run-unique middle
segment from the run id, which is itself<export>_<stamp>; its three sibling
runners use a fresh stamp and never doubled. Run-uniqueness (the millisecond
stamp) and retry-overwrite (the tag is stable within a run) are unchanged, and
nothing parses the prefix — the_pk_w/_keyset_suffix readers key off is
untouched. Existing files are not renamed; new runs write the shorter name. -
Internal, no behaviour change: the two orchestrator entry points now share one
post-plan execution script, the ~800-line load orchestrator moved out of the
CLI dispatch module, and the state / manifest / preflight seams were split
along the same lines. Test-harness and CI-gate hardening rides along.