Skip to content

v0.163.10

Choose a tag to compare

View all tags
@github-actions github-actions released this 24 Oct 06:03
· 29 commits to main since this release
v0.163.10
7d66ef5

CRITICAL fix, silent: sluice schema add-table --no-drain from a Postgres source lost or undid changes made to the new table during the add (NO-DRAIN-POSTGRES-SOURCE, sluice-testing Bug 332). On a stream started with --include-table (the staged-wave flow) every delete, update and insert made between the add's snapshot and the stream's next live-added poll was lost; on an unfiltered stream a delete or update that reached a row before the copy wrote it was undone by the copy. Exit 0, no WARN. Affected: every Postgres source, every target, v0.24.0 through v0.163.9. --no-drain is now refused on every source, before anything is read or written; use the drained flow (sluice sync stop --wait, sluice schema add-table, sluice sync start), now measured exact from a Postgres source. Not quite drop-in: a script that passes --no-drain now exits 1 (see Compatibility). Tables already added this way need a check (see Who needs this).

Fixed

  • --no-drain refuses on a Postgres source (NO-DRAIN-POSTGRES-SOURCE, Bug 332, CRITICAL, silent; v0.24.0 through v0.163.9). This was ADR-0030's publication-add live path, introduced in v0.24.0 (27d94608), and the one live path v0.163.8 kept when it refused --no-drain on binlog and VStream sources. ADR-0036 had called it strict zero-loss since v0.32.0. That was false: its pins wrote inserts only, and inserts are the one change kind that commutes with the add's idempotent copy. The running pgoutput stream does not apply the added table's changes in order after the copy. On a filtered stream its dispatch filter drops them until the live-added poll admits the table; on an unfiltered stream it applies them while the copy runs, so a delete or update reaching a row the copy has not written yet is a no-op and the copy then writes the snapshot's version. Measured with a 20,000-row table added to a running stream (TestAddTableNoDrainWindow_PG_*, at v0.163.8 and again on the v0.163.9 tree), deletes kept / updates stale / inserts missing: Postgres to Postgres 50/20/20 filtered and 40/15/0 unfiltered; Postgres to MySQL 50/20/20 filtered (50/20/10 on the v0.163.9 re-run) and 50/10/0 unfiltered. The add exited 0 with no WARN, and its success line said the running stream had applied the table's changes. The v0.163.8 regression cycle measured the same shape on Postgres to Postgres (25 deletes kept, 24 inserts missing and 25 rows stale filtered; 12 deletes kept unfiltered) and the same loss on v0.150.0, v0.162.0 and v0.163.7. Now noDrainRefusalForStream never returns nil: a Postgres source refuses with NO-DRAIN-POSTGRES-SOURCE, a binlog source keeps NO-DRAIN-BINLOG-SOURCE, a VStream source keeps NO-DRAIN-VSTREAM-SOURCE (kept separate so each stays grep-stable), and any other source gets a generic "withdrawn on every source" refusal; every message names the drained flow (drainedAddTableFlow). The door moved earlier: it fires at the top of preflightStream, right after the stream row is found, before publication scoping, the target-schema check, the source schema read, the dry-run report, the pending record and any write, so --dry-run refuses too. Pinned by the four TestAddTableNoDrainWindow_PG_{PGToPG,PGToMySQL}_{Filtered,Unfiltered}Live cells, which now require the refusal with no snapshot, no target table, no live-added or pending record, the table in no publication and the running stream still applying its other tables; by TestAddTable_LiveMode_PublicationSourceRefusesBeforeAnyWrite (stop-in-flight × dry-run: no publication add, snapshot, live-add record or writer); by TestNoDrainRefusal_EngineMatrix (no engine may take a live path); and through the real binary by TestIncludeTableScopeDrift_PgoutputLiveAddOfAnExcludedTableStops and its binlog twin, which assert schema add-table --no-drain exits non-zero carrying the marker. Removed with the path: the live preflight, the snapshot-LSN ≥ slot-LSN invariant (it held throughout and could not see this loss), the Postgres engine's ReadSlotPosition, and the in-flight-stop hint that offered --no-drain.
  • The drained flow from a Postgres source is measured exact, including the windows around the publication add. The drained controls run filtered Postgres to Postgres, unfiltered Postgres to Postgres, and Postgres to MySQL across four phases (just after the add's snapshot opens, mid-copy, after the add returns, after the live-added poll interval), plus two phases the earlier rig did not measure: changes committed after sync stop and before the ALTER PUBLICATION, and one transaction opened before the ALTER PUBLICATION and committed after it, so its WAL straddles the catalog change the restarted stream decodes through. All exact, every phase 0/0/0, with rig guards that both phases and the publication add ran in order, and requirePGDrainedExact now failing on any per-phase verdict, not only the whole-table diff. The add's completion message now also names the fleet-leg restart, as the drained-flow text does.

Not closed, stated

  • A drained add against a stream that was never stopped (DRAINED-ADD-TABLE-RUNNING-STREAM-UNCHECKED, OPEN, HIGH, silent). schema add-table without --no-drain refuses a stop still in progress but does not detect a running stream. On a Postgres source a drained add runs the same publication-add flow --no-drain ran (they differed only in the preflight), so against a running stream it takes the Bug 332 loss above (by the code; not measured as a drained add). It is to be closed by an upcoming release. Until then the stop is not optional: run sluice sync stop --wait and keep the stream stopped, with no fleet reload and no supervisor or systemd restart, until schema add-table returns.
  • A Postgres fast-path door for that check is filed, not built (DRAINED-ADD-PG-SLOT-ACTIVE-DOOR). Refusing a drained add while the stream's replication slot reads active would narrow the gap on Postgres; it is a new guard and ships on its own, with its own review.
  • Test coverage on this release. This release's integration tests run in the per-PR CI shards. The psverify (PlanetScale) and d1verify suites were not run for it; every Bug 332 measurement is on local Postgres and MySQL containers.

Compatibility

  • Breaking for --no-drain: sluice schema add-table --no-drain now exits 1 on every source, including a dry run, having read and written nothing. The flag is kept so a script that passes it gets the refusal rather than an unknown-flag error. Drop --no-drain and run the drained flow: sluice sync stop --wait, sluice schema add-table without --no-drain, then sluice sync start with the same --stream-id (or the fleet restart for a sluice sync run leg).
  • Upgrade the binary that runs schema add-table. The refusal lives in add-table, not the stream: an add run by v0.163.9 or older against a stream a newer release runs still takes the lossy path.
  • State left by an older add: a Postgres --no-drain add run by v0.163.0 through v0.163.9 wrote exactly the records a drained add writes, so a stream on this release reads it as a drained add and keeps the table in scope. Its lost changes are not detectable from that state.
  • No other change: drained schema add-table, sync, migrate, backups and BackupFormatVersion are unchanged.

Who needs this — action required

  • Anyone who ran sluice schema add-table --no-drain from a Postgres source (v0.24.0 through v0.163.9) while the source took writes to the added table (deletes or updates on any stream; any writes, inserts included, on a stream started with --include-table). Assume the added table is diverged. How to tell: no sluice check can rule it out. sluice verify --depth count does not see stale values, and kept deletes offset missing inserts in a count. Only a key-by-key, value-by-value comparison against a quiesced source can clear the table. Remedy, measured exact on Postgres to Postgres and Postgres to MySQL (TestAddTableNoDrainWindow_PG_RecopyRemedy, TestAddTableNoDrainWindow_PG_RecopyRemedyMySQLTarget): stop the stream with sluice sync stop --wait; empty the target table without firing the foreign keys that reference it: sluice creates the source's ON DELETE CASCADE keys on the target, a plain DELETE cascades into the child tables, and the restarted stream never puts those rows back (measured: 0 of 200 child rows left). On a Postgres target run the DELETE in a transaction that first does SET LOCAL session_replication_role = replica; on a MySQL-family target SET FOREIGN_KEY_CHECKS = 0 in that session. Then run sluice schema add-table for the table without --no-drain, and start the stream again with the same --stream-id. Alternatively re-copy the whole stream with --restart-from-scratch.
  • Anyone with scripts or runbooks that pass --no-drain: they now fail; switch them to the drained flow above.
  • Anyone running a drained schema add-table: stop the stream first and keep it stopped until add-table returns; nothing checks it yet.
  • Everyone else: nothing to do.

Install: brew install sluicesync/tap/sluice · go install sluicesync.dev/sluice/cmd/sluice@v0.163.10 · Container: ghcr.io/sluicesync/sluice:0.163.10

Full changelog: https://github.com/sluicesync/sluice/blob/main/CHANGELOG.md

Assets 15
Loading