Skip to content

sluice v0.99.291

Choose a tag to compare

@github-actions github-actions released this 23 Jul 21:29
· 1857 commits to main since this release

HIGH silent-loss fixes for filtered sync — the client-side --where evaluator now follows the SOURCE engine's own comparison semantics for temporal literals and for NaN/±Infinity rows, closing two stale-row/orphan classes present since filtered sync shipped — plus sluice sync decommission, the closing waves of the 2026-07-23 blind-audit remediation, and a grpc security bump. The temporal defect affects every filtered-sync release (v0.99.276–v0.99.290): a --where literal finer-grained than its column (a time-of-day against a DATE column, or >6 fractional-second digits) was resolved one way by the source engine on the snapshot leg and another way by sluice's full-precision client evaluator on the CDC leg, so boundary rows went stale at exit 0. The non-finite defect (float: v0.99.282–v0.99.290; numeric: v0.99.276–v0.99.290) dropped every UPDATE and DELETE for a NaN row the snapshot had copied. Drop-in upgrade, no breaking changes — if you run sync --where, see "Who needs this" below.

Features

sluice sync decommission — retire a finished stream's durable footprint in one gated command. A stopped wave leaves its replication slot and per-stream publication on the source and its control row on the target; the slot pins WAL for the rest of a multi-week staged migration and, since v0.99.289's existence-semantics scope guard, blocks every later differently-scoped cold start — the only remedy was raw SQL in the staged-wave guide. The new cross-DSN command verifies the stream exists and is not running (an active stream refuses with the new coded SLUICE-E-DECOMMISSION-STREAM-ACTIVE, naming sync stop --wait), drops the slot (with the lingering-walsender 55006 retry idiom), drops the stream's RECORDED per-stream publication — never the shared sluice_pub, and a legacy/empty recorded name drops nothing and says so — then clears the control row LAST, so a partial failure keeps the record an idempotent re-run needs to finish. Gated on --yes with --dry-run as the frictionless preview; MySQL-family sources get control-row-only cleanup (the binlog is the stream) and trigger-CDC sources keep their shared change-log/triggers (trigger prune/teardown own those). Pinned by orchestration-order/partial-failure/idempotency unit tests plus real-PG and real-MySQL lifecycle gates; the staged-wave guide's step 5 is now this command.

Fixed

HIGH (silent loss): filtered replicas now follow the source engine's own comparison semantics for temporal literals (affected: v0.99.276–v0.99.290, every release with filtered sync). A temporal literal finer-grained than the column is resolved three different ways by the three engines — Postgres casts the literal to the column's type (DATE truncates time-of-day; fractional seconds round to µs by PG's own DOUBLE-MEDIATED rint(strtod(fraction)·10⁶) rule, now reproduced bit-for-bit); MySQL PROMOTES the DATE column to datetime and rounds extra digits half-up; MariaDB promotes but truncates — while sluice's client CDC evaluator compared at full precision, disagreeing with the engine-evaluated snapshot leg: on a PG source, --where "d = '2024-01-01 08:30'" on a date column snapshot-copied the row and the client evaluator then dropped all of its CDC changes — stale target rows at exit 0, with or without the push-down. rowpredicate.Compile now normalizes each temporal literal to the source engine's coercion (ir.TemporalLiteralSemantics, threaded engine-neutrally on the resolver seam; unknown engines fail closed to a loud refusal), ground-truthed live on PG 16.14 / MySQL 8.0.46 / MariaDB 11.8.8 and pinned by a three-engine real-server matrix plus a RANDOMIZED real-PG fraction sweep (300 fractions/run, half forced onto the exact-half boundary) after the value-fidelity review falsified the first cut's exact half-even rounding (~0.1% of 7-digit fractions silently diverged on live PG). Two edges close the class: VStream sources route engine-coerced temporal terms through the existing A0 client-side fallback instead of the server push (vtgate evalengine's coercion is an unverified surface), and v0.99.290's interim push-down exclusions move back INTO the PG envelope with granularity equivalence cells in the oracle.

HIGH (silent loss): NaN/±Infinity rows now order with PG's total order in the client --where evaluator (affected: float v0.99.282–v0.99.290; numeric v0.99.276–v0.99.290). PG orders NaN above every other value, so an ordering predicate's snapshot leg copied a NaN row — and the client CDC evaluator, which mapped NaN to UNKNOWN→drop, then dropped its every UPDATE (stale target row) and swallowed its DELETE (permanent orphan), at exit 0. The numeric sibling was worse: NUMERIC stores NaN (and, PG 14+, ±Infinity), the values travel as strings, and the exact big.Rat parse mapped them to UNKNOWN→drop while numeric sits INSIDE the push-down envelope — the server delivered the row's changes and the client belt itself destroyed them. compareFloat and compareNumeric now implement PG's total order (NaN greater than every non-NaN; −Infinity below everything), safe engine-neutrally because PG is the only supported source that can deliver a float NaN. Pinned by two real-PG gates streaming NaN/±Inf through real logical decoding, asserting server-verdict == client-verdict per op plus UPDATE/DELETE routing on the NaN row.

Post-slot publication re-verification closes the concurrent-first-boot race (affected: v0.99.287–v0.99.290; requires two streams cold-starting simultaneously under one publication name). Cold start ensures the publication BEFORE its own slot exists, so two simultaneously cold-starting streams sharing a name could swap definitions past the ADR-0175 existence guard — the loser silently streamed through the peer's row filter or member set forever. After the snapshot open creates the slot, the pipeline now re-verifies the publication against the exact definition it ensured (a read-only transactional probe with PG's own pg_get_expr normalization as the judge), refusing with SLUICE-E-CDC-PUBLICATION-SCOPE-CONFLICT before any data moves; the racer whose definition the catalog holds proceeds. The refusal also no longer quotes a PEER stream's row-filter predicate (row filters routinely carry data values — a co-located tenant's predicate was disclosed into another operator's terminal); the operator's own incoming predicate stays visible.

--publication-name is validated at resolve time (affected: v0.99.287–v0.99.290). sluice's quoted CREATE PUBLICATION preserves case while START_REPLICATION downcases publication_names (and CREATE silently truncates past 63 bytes) — so a mixed-case or over-length name created one publication and streamed from another: 42704 at the first change on a busy source, a permanently idle GREEN stream on a quiet one, and the ratchet then RECORDED the broken name so every resume repeated it. Now refused up front with the new coded SLUICE-E-CDC-PUBLICATION-NAME-INVALID (lowercase [a-z0-9_], ≤63 bytes), before anything touches the source.

One shared transient network-shape matcher replaces four hand-mirrored classifier lists — fixing the pgtrigger-on-Windows terminal exit. The transient transport vocabulary lived in four places (trigger-CDC poll, pipeline connect-phase retry, both engine applier text legs) and had already drifted one release after Bug 199: the Windows dial wordings never reached the trigger-CDC list, so a postgres-trigger-source sync on Windows still exited terminally on a routine managed-PG restart — always loud and zero-loss, an availability gap. All four sites now delegate to the new internal/nettransient corpus (structured sentinels first, text shapes second; no-such-host, auth, DSN-parse, and coded refusals stay terminal-by-design), each site keeps its genuinely local wordings, and corpus-parity change-detectors at every consumer make one-sided drift a CI failure. The D0-3/D0-8 terminal-code shield is untouched — the matcher runs only when no structured server error is present.

Control-table preamble and publication-ratchet reads now ride the connect-phase retry (affected: v0.99.288–v0.99.290; loud). Sync phases 2–2.8 (ensure control table, clear stop signal, fingerprint ListStreams, recorded-publication read) run on every retry re-establish but carried the bare schema-apply hint instead of the connect-phase marker, so a network blip during a retry attempt's re-establish was terminal at exactly the step class Bugs 199/200 fixed one phase earlier. Refusals (stream-id collision, drift, permission faults) stay terminal; injected-fault unit pins per site, RED pre-fix.

PG ListStreams gains the legacy-column fallback MySQL has had since v0.32.2. An upgraded binary's read-only paths (sync status/health, the live panel, --dry-run) query the control table without running its ALTER migrations, so against a table an older release created the widened SELECT errored with SQLSTATE 42703 and took down the observability path. PG now retries with the original column set, degrading to empty extras exactly like MySQL; a previous-release-DDL pin on BOTH engines auto-covers every future column addition.

Two small filtered-sync corrections. A filtered --reset-target-data restart now derives its per-stream publication instead of landing row filters on the shared sluice_pub (the ratchet read the control row before ClearStream deleted it, so the reset looked like a legacy stream — affected v0.99.290 only), and the before-image preflight now runs AFTER --where key canonicalization, so a case-mismatched key gets the correct unknown-table refusal instead of a spurious SLUICE-E-WHERE-CDC-BEFORE-IMAGE. The push-down classifier also gains fail-closed belts: an unrecognized AST node now refuses instead of contributing zero terms, and catalog-marked non-deterministic text collations refuse in the envelope arm.

Also in this release

The gate-ratchet batch converts seven audit finding classes into permanent deterministic checks: an exhaustive push-down-envelope pin over the full ir type universe with envelope↔oracle coupling gates, check-shard-coverage.sh now cross-greps the real ci.yml matrix, an expiry-token gate in prerelease-triggers.sh (its first run flagged 10 real stale markers, all fixed), an ADR index↔file status-parity test (first run caught ADR-0135/0136 still saying Proposed for engines shipped in v0.99.148/149), a reflective fleet-spec↔CLI flag-parity gate — whose first run caught that the documented --notify-smtp-tls flag never actually existed (kong derived --notify-smtptls); the documented spelling is now primary with the old derived one kept as a compatibility alias — cell-level error-codes content equality, and a fail-on-skip belt over all five CI integration shards. A new chaos-lite gate restarts the REAL target and source containers mid-CDC on both cross-engine legs, pinning the whole v0.99.286–290 retry arc (ride-out, warm resume, outage-window DELETE replay, byte-identical convergence) per PR. The Postgres publication lifecycle moved out of cdc_reader.go into publication.go (mechanical, byte-verbatim — no behavior change); the vacuum-health alerter gained an injectable tick and a -race lifecycle test; the weekly vstream leg gained a vacuous-green guard; MariaDB joined both operator migration guides with a doc-sync gate asserting every CDC-capable engine is covered; and google.golang.org/grpc is bumped 1.82.0→1.82.1 (GHSA xDS RBAC / HTTP-2 HIGH; govulncheck says the vulnerable path is not reachable from sluice code — bumped anyway).

Compatibility

No breaking changes; drop-in upgrade. migrate is unaffected by the evaluator fixes (its --where renders into the engine-evaluated snapshot SELECT), and unfiltered syncs are untouched. Filtered-sync boundary rows now match the source engine's verdicts — a behavior correction, not a change of contract: the docs already promised a faithful filtered replica. Three refusals are new or newly reachable, all deliberate: a mixed-case/over-length --publication-name now refuses up front (it never actually worked — see Fixed), a temporal literal finer-grained than its column refuses when no engine semantics are available (unreachable through the normal CLI, which always threads the source engine), and the new sync decommission refuses on an active stream and without --yes. --notify-smtp-tls is now the real flag spelling; the previously-derived --notify-smtptls keeps working as an alias, so no invocation breaks. VStream sources with engine-coerced temporal --where terms now keep those tables on client-side filtering (the A0 fallback) instead of the server push — delivered rows are identical; only the wire-traffic reduction narrows. MySQL, PlanetScale, and Vitess sources are untouched by the publication work.

Who needs this — action required

  • Anyone running filtered sync (sync --where) on v0.99.276–v0.99.290 whose predicate compares a temporal column against a finer-grained literal — a time-of-day (or any time component) against a DATE column, or a literal with more than 6 fractional-second digits — upgrade, then re-verify the filtered tables (sluice verify): rows on the predicate boundary may be stale on the target (their CDC changes were dropped at exit 0). --restart-from-scratch or a targeted backfill repairs any divergence. Predicates whose literals match the column's granularity were not exposed.
  • Anyone running filtered PG sync whose filtered float or numeric column can hold NaN/±Infinity (float since v0.99.282, numeric since v0.99.276) — same action: re-verify the filtered tables; NaN rows' updates and deletes were silently dropped after the snapshot copied them.
  • Anyone who used a mixed-case or >63-byte --publication-name on v0.99.287–v0.99.290 — the stream may be idle-green (no changes ever delivered). Check sync status lag against source activity; decommission the stream and restart it under a valid lowercase name, then re-verify.
  • Anyone who cold-started two PG-source streams simultaneously under one shared publication name on v0.99.287–v0.99.290 — narrow race, but if it describes your fleet bring-up, re-verify each stream's tables; the loser may have streamed through the peer's scope.
  • Windows operators with postgres-trigger sources — upgrade; a managed-PG restart now reconnects instead of exiting. The exits were loud with clean warm resumes, so no data action.
  • Operators finishing staged waves — no action required, but sluice sync decommission now replaces the raw-SQL slot/publication cleanup in the staged-wave guide.
  • Everyone else: no action needed beyond upgrading.

Install

  • Binaries: https://github.com/sluicesync/sluice/releases/tag/v0.99.291
  • Homebrew: brew install sluicesync/tap/sluice
  • Scoop: scoop bucket add sluicesync https://github.com/sluicesync/scoop-bucket; scoop install sluice
  • winget: winget install sluicesync.sluice
  • Docker: docker pull ghcr.io/sluicesync/sluice:0.99.291

Full changelog: https://github.com/sluicesync/sluice/blob/main/CHANGELOG.md