Skip to content

v0.25.0

Choose a tag to compare

@github-actions github-actions released this 30 Aug 13:41
· 524 commits to main since this release
a5ca538
  • New: rivet state runs and rivet state finish-run — the frozen-prefix
    escape hatch. runs lists the run-status ledger (gc/cleanup's activity
    signal); finish-run terminal-stamps a row you KNOW is dead (hard crash with
    no successful successor — nothing else can release it now that supersession
    is success-only). The cleanup refusal names the commands.

  • CDC re-baseline loads now REFUSE instead of appending silently-wrong
    data.
    A snapshot-carrying rivet load into a <table>__changes that
    already holds change rows hard-fails with a per-warehouse TRUNCATE
    prescription: a re-snapshot row cannot win the dedup against already-loaded
    changes (and no snapshot can express a PK deleted during the gap), so the
    view would silently keep serving pre-gap values. Stateless loads degrade to
    a note; an errored state store refuses (fail-safe).

  • Snapshot legs are anchor-stamped. On checkpointed MySQL/SQL
    Server/MongoDB flows the initial: snapshot parquet now carries constant
    __pos (the CDC anchor) and __seq = -1 columns, so a re-baseline orders
    correctly in the warehouse dedup. PostgreSQL-without-a-checkpoint and
    pre-existing snapshots keep the legacy NULL shape. Anyone querying
    __changes WHERE __pos IS NULL to find baselines sees the new shape.

  • rivet validate got stricter — previously-green runs can now exit 3.
    CDC prefixes presence-check EVERY cycle's committed parts (the dataset is
    the union of run-unique manifests; deleting a historical part used to
    validate PASSED), and a declared part that is present at its size but does
    not decode as Parquet is VERIFIED-WRONG even when the manifest records no
    checksums. Budget accordingly: --depth full on a long CDC stream now
    honestly re-reads the accumulated history.

  • Resume verifies the destination. A chunked resume re-exports committed
    chunks whose parts were deleted between attempts (destination probe + the
    M8 rewrite path); a keyset resume REFUSES over a deleted committed page
    (naming rivet state reset) instead of finalizing Success over a hole.

  • gc/census supersession requires a newer SUCCESS. A failed or
    merely-running successor no longer retires a live run's marker; a failed
    run's parts are spared until a Success supersedes them (checkpoint resumes
    may still adopt them); dead running markers are retired via the
    terminal-ledger sweep once finish-run/the split stamp closes their rows.

  • New refusals of previously-accepted configs (each was a silent
    misbehaviour): format: csv under a load: block; load: on a non-GCS
    destination (it silently listed a same-named GCS bucket); destination
    prefixes with a .. segment or an embedded URI scheme; --split with
    {date}/{run_id} placeholders; equivalent checkpoint-path spellings; a
    MySQL CDC open whose server-identity query fails (no more unverifiable
    anchors); a MongoDB resume token whose _data is not a string.

  • rivet init generates case-exact configs. Mixed-case tables route
    through a quoted query: (a CaseTwin/casetwin twin used to silently
    export the WRONG table's rows), SELECT columns are quoted per engine (the
    same twin bug at column level), Mongo always emits table: (hyphenated
    collections get an honest SKIPPED comment), and -o refuses to overwrite
    an existing file.

  • CSV: NULL and the empty string are distinct at last. A present empty
    string renders as ""; NULL stays the bare field (the PostgreSQL COPY
    convention). Existing consumers see different bytes for empty strings;
    DuckDB readers need allow_quoted_nulls=false to recover the distinction.

  • Secrets scrubbed from more places. URL query parameters carrying
    token/password/api-key/signature families are redacted in errors and logs;
    a Slack webhook delivery failure no longer prints the webhook URL (its
    path IS the credential); load-path errors route through the redactor.

  • Smaller: BigQuery DDL backticks every column (reserved-word columns
    survive); a set-but-missing GOOGLE_APPLICATION_CREDENTIALS is a loud
    error naming the path; RIVET_BQ_ACCESS_TOKEN now takes precedence over
    ADC resolution; rivet check preflight understands Mongo keyset
    (page_size) and stops sparse-alarming configs that took its own
    chunk_count advice; the doctor recognises MySQL's "Unknown database".

  • CDC memory budgets measure memory now, with an operator-visible surface.
    rollover_memory_mb gained a default (256 MiB) so absence no longer means "no
    byte budget" — wide-row tables roll parts by resident size where they previously
    buffered to the row cap; part counts can change for such configs. The
    RIVET_CDC_MAX_TX_* caps are measured in RESIDENT cost (what the buffer actually
    holds) instead of a payload estimate that under-counted ~12.7x, so a transaction
    that slipped under the old 2 GiB figure may now refuse — the refusal was always
    the intent, the old number was not a guard. Zero is rejected for both rollover
    knobs at validation (0 meant "roll a part on every transaction", not
    "disable"). bytes_read metrics return to approximately the released DECODED-payload
    unit (the old +32/event constant is gone — percent-level on realistic rows)
    and stay comparable with batch exports.

  • Opt-in transaction spilling (RIVET_CDC_SPILL_DIR). A transaction past the
    cap can spill its tail to disk instead of failing (PostgreSQL/MySQL/SQL Server;
    MongoDB needs none by construction). Off by default: it moves the adapter's copy
    only (~11% of peak RSS measured on PostgreSQL), so the caps stay the honest
    guard. On the config-driven path every resolved location is config-anchored (the ad-hoc rivet cdc CLI anchors to the invoking shell's directory — the directory the operator chose).

  • A service-account key file is now minted IN PROCESS — gcloud is off the
    BigQuery path for it.
    With GOOGLE_APPLICATION_CREDENTIALS pointing at a
    service_account JSON, rivet's shared Google auth seam
    (destination::gcs_auth) returned "not my shape" and the BigQuery loader
    shelled out to gcloud for a token. Rivet now builds the RFC 7523 claim set,
    signs it RS256 with the file's own private_key, and exchanges it for an
    access token itself — the same seam, so the GCS destination and the BigQuery
    loader mint from ONE implementation and a run's two legs act as ONE identity.
    Verified against a real service account: INFORMATION_SCHEMA.JOBS_BY_PROJECT
    records user_email = rivet-export@….iam.gserviceaccount.com for the load's
    jobs, with the gcloud fallback compiled to abort so a green run could not
    come from the subprocess. Two behaviour notes: a service account is billable
    on its own, so rivet no longer sends x-goog-user-project for it (the header
    would demand serviceusage.services.use, which the roles that grant a load do
    not include); and external_account / workload identity plus GCE metadata
    still use the gcloud fallback — an STS exchange rivet does not model, and a
    half-implemented subset would be worse than the documented subprocess. The
    crypto is jsonwebtoken, already in the tree via reqsign: zero new crates,
    zero added build time. Extension seam (ADR-0026): a WIDENING, not a break —
    AdcUserTokenLoader / try_authorized_user_loader() keep their names and
    signatures and gain credential_kind() / principal().

  • A multiplex CDC export (mode: cdc with tables:) can now be loaded.
    The config rivet init --mode cdc emits for a whole schema carries no
    table: or query:, so rivet load bailed on it outright ("must specify
    exactly one of 'query', 'query_file', or 'table'") and rivet check --json
    silently dropped its columns block. Both halves now fan out per table: the
    type report emits one document per table (with a new table: field and a
    cdc/orders display name), and the loader builds one plan per table — its
    own warehouse table, its own <base>/<table>/ staging prefix. That prefix is
    the load-critical half: the manifest listing is recursive, so N plans sharing
    one base prefix would each sweep in every sibling table's parts and merge N
    source tables into one warehouse table with every count agreeing.

  • BigQuery loads no longer need the bq CLI (or the Google Cloud SDK) on
    PATH.
    rivet load --target bigquery talks to the BigQuery JSON API
    directly over HTTPS, using the same Application Default Credentials rivet
    already uses for GCS — one Google auth path, not two. A container or CI image
    that installed the Cloud SDK only for rivet can drop it. Job labels, the
    4,000-partition batch split, PARSE_JSON, and the partition-quota error hint
    are unchanged; error messages now name the BigQuery job id. If ADC is a user
    credential with no usable refresh grant, rivet still asks gcloud for a token
    when it is present, and honours RIVET_BQ_ACCESS_TOKEN when it is not.

  • rivet load no longer shells out to itself. The load planner obtained
    every column's warehouse type by SPAWNING rivet check --target X --json and
    parsing the child's stdout; it now calls the same resolver in process
    (preflight::collect_type_reports, the function that command renders from).
    Version skew between the loading binary and the type-resolving one is gone by
    construction, so the --rivet-bin flag that existed only to mitigate it is
    removed (passing it is now an error — it had no other use), and the config
    is parsed ONCE instead of twice with two different ${VAR} resolutions. The
    plan also keeps the resolver's whole TargetColumnSpec — autoload_type, the
    note and the L5 cast_sql were silently dropped by the four-field JSON mirror.
    rivet load's planning step is now reachable from offline tests for the first
    time.

  • ADR-0028: one finalize seam for the export tail. The post-write tail
    (manifest schema-fingerprint pin, on_schema_drift gate, Form-B checksum
    harvest, shape-drift warn) is now applied exactly once, at
    finalize::finalize_export — called by the dispatcher between the runner
    returning and finalize_manifest — from a CommitLedger the runners feed as
    they commit. Previously every export runner re-assembled that tail by hand,
    which is how a feature wired into one runner ended up silently absent on the
    others (the recurring runner-bypass class: the keyset/mongo drift-gate miss,
    Form-B computed-then-discarded on the large-table runners). Runners now only
    feed the ledger; the ordering constraints live in one place. Behavior is
    unchanged except keyset/mongo manifests now also carry the dest-schema
    fingerprint single mode always recorded (uniform pin at the seam).

  • rivet plan is now READ-ONLY without --annotate-waves. It computes and
    prints the advisory schedule (and the reviewable plan artifact) but no longer
    touches your config file unless you pass --annotate-waves. Previously, plan
    gated only the overwrite of an existing wave:/parallel_safe: behind the
    flag but still SILENTLY ADDED those fields to any export that lacked them — so
    a read-only-looking rivet plan mutated a fresh config (the ADD-arm of the
    0.24.4 config-clobber incident, whose overwrite half --annotate-waves was
    introduced to require consent for). Now config mutation is a single explicit
    opt-in: no flag ⇒ no write, present field or absent. --annotate-waves
    (over)writes the whole schedule as before. rivet apply is unaffected —
    exports with no wave: run last, as one implicit final wave.

  • Keyset part filenames no longer repeat the export name. Parts landed as
    exports/<export>/<export>_<export>_<stamp>_pk_w3_2.parquet because the
    keyset name format prepends the export name AND derives its run-unique middle
    segment from the run id, which is itself <export>_<stamp>; its three sibling
    runners use a fresh stamp and never doubled. Run-uniqueness (the millisecond
    stamp) and retry-overwrite (the tag is stable within a run) are unchanged, and
    nothing parses the prefix — the _pk_w / _keyset_ suffix readers key off is
    untouched. Existing files are not renamed; new runs write the shorter name.

  • Internal, no behaviour change: the two orchestrator entry points now share one
    post-plan execution script, the ~800-line load orchestrator moved out of the
    CLI dispatch module, and the state / manifest / preflight seams were split
    along the same lines. Test-harness and CI-gate hardening rides along.