v0.163.9

CRITICAL fix, silent: a PlanetScale/Vitess cold start or backup full of a keyspace with more than one shard kept rows the source deleted (VSTREAM-SHARD-SKEW-COPY-CHANGES-DROPPED, Bug 331). This was the default copy, with no flag and no fault: shards finish copying a table at different times, and the changes a finished shard made while another shard was still copying were treated as copied rows, so deletes were dropped and a primary-key change left its old row, at exit 0. Affected: sync from v0.2.0 through v0.163.8, backup full from v0.44.0 through v0.163.8. A one-shard keyspace is not affected by this mechanism. Drop-in: no flag, default or backup format change; a replayed change the copy cannot apply now fails its table loudly (VSTREAM-COPY-ROW-CHANGE-REFUSED) instead of being dropped. Streams and backups copied by an earlier release need a check (see Who needs this).

Fixed

• A finished shard's in-copy changes are replayed from its copy position (VSTREAM-SHARD-SKEW-COPY-CHANGES-DROPPED, Bug 331, CRITICAL, silent; sync cold start since v0.2.0, backup full since v0.44.0). Every table's copy is one VStream across all shards. When a shard finishes its copy, vttablet streams that shard's binlog from its snapshot GTID at once and vtgate forwards it before the stream-wide COPY_COMPLETED. sluice's copy dispatchers (bufferCopyRow, and copyStream.bufferRow for the concurrent copy) decoded after images only and skipped a change with none as "malformed": every DELETE was dropped, an INSERT or UPDATE was written early, and a key-changing UPDATE wrote the new row and left the old one. The table's resume position was the VGTID at the stream-wide COPY_COMPLETED, already past those changes, so the change stream never replayed them. The v0.163.7 regression cycle measured 52 of 52 deletes kept on the early shard at exit 0 (the later shard's 48 applied), the same on v0.163.6, v0.150.0 and v0.100.0, and a copy count of 700,483 for a 700,456-row snapshot; backup full archived the early shard's changed rows a second time, so its restore failed with SQLSTATE 23505. Now shardCopyBoundary records each shard's GTID at its own per-scope COPY_COMPLETED, holds (counts, never buffers) that shard's later row events, and snapshotPosition keeps that shard at its copy-finish GTID, so the change stream, or the backup's incremental, replays every held change through the ordinary decoder with both row images: deletes delete and a key change moves the row. It refuses loudly if a recorded GTID is not contained in the stream's final position, or is missing while changes were held. All three copy paths are fixed (the default per-table copy, the one-table copy, the concurrent vstream_copy_table_parallelism copy), as is backup full, which rides the same stream. A run that did this logs, once per finished shard, mysql/vstream: snapshot: shard finished its copy before the stream did; its later changes are replayed from its copy position with keyspace, shard, tables, changes and resume_gtid. Pinned by TestStreamer_VStreamShardSkewCopy, which deliberately skews a vttestserver keyspace (a one-shard control, two shards, four shards with two early finishers) and makes deletes, updates, key changes, inserts and insert-then-delete pairs on the finished shards while the last shard copies: with the hold and rewind taken back out, every finished-shard arm left 23 rows on the target at exit 0 (all 15 deletes, the old key of all 5 key changes, all 3 insert-then-delete rows) in the one-table, two-table and concurrent cells, and the backup's chain restore failed on a duplicate key; with the fix every cell matches the source by key and value and the held count equals each finished shard's 46 row changes. Unit pins include TestShardCopyBoundary_SnapshotPosition, TestVStreamSnapshot_FinishedShardChangesAreHeldAndRewound, TestVStreamSnapshot_OneShardPositionUnchanged and TestVStreamConcurrentCopy_FinishedShardChangesAreHeld.
• A copy row that is not a copied row now refuses its table instead of being dropped (VSTREAM-COPY-ROW-CHANGE-REFUSED). On a shard still copying, Vitess sends only after images except when it replays changes inside the copy, which it does for two reasons. (1) vtgate restarts a shard's tablet stream on a retriable error (a reparent, a vttablet restart, PlanetScale maintenance; up to three consecutive times), resuming from the shard's copy cursor, and vttablet first replays every change to the in-progress table since its snapshot. That reaches every copy path, the default one included, and before this release the replayed deletes were dropped and a key change left its old row: a previously unfiled silent class, now filed as VSTREAM-VTGATE-SHARD-RETRY-CATCHUP-DELETES and covered. It is derived from the Vitess code and not reproduced end to end: vttestserver runs vtgate and every tablet in one process, so no tablet stream can fault on its own. (2) One VStream copying several tables replays the tables it already copied before each later one; the one such path left is sluice schema add-table, whose copy streams the whole keyspace. Now, on a shard still copying, an after-image row and an UPDATE that leaves every primary-key cell byte-identical are applied, and a DELETE, a key-changing UPDATE, a change with no image, an UPDATE of a table with no key, or an UPDATE whose after image is partial (a self-hosted Vitess on binlog_row_image=NOBLOB or PARTIAL_JSON, which can omit a column or carry a JSON diff — the change stream already refused those) refuses that table (copyRowChangeRefusal). The refusal is classified by stream scope: on a stream that copies one table (every sync cold start path and backup full) the only cause is vtgate's retry, a transient fault, so it is retriable and takes the same route as a mid-copy connection fault (VSTREAM-COPY-RECONNECT-STALE-ROWS): a first cold start clears the in-scope target tables and copies them again in process, a re-copy of an established stream refuses COLD-START-INCOMPLETE (remedy --restart-from-scratch), under --schema-already-applied it refuses RESTART-FROM-SCRATCH-TARGET-NOT-CLEARED, and backup full fails and can be re-run. On a keyspace-wide or multi-table stream it is terminal, since a retry would meet the same deterministic replay; the only production caller of that shape that does not defer (below) is the cold start's empty-table-list fallback in streamer_coldstart.go, whose change stream starts at the copy's own position. Pinned by TestCopyRowChangeRefusal_Shapes (every shape, single- and multi-table, classification read through errors.As on ir.RetriableError), TestVStreamSnapshot_CopyPhaseRefusalClassifiedByStreamScope, TestVStreamSnapshot_CopyPhaseDeleteRefusesItsTable and TestVStreamConcurrentCopy_CopyPhaseDeleteRefusesItsTable.
• A drained schema add-table leaves in-copy replays to its stopped stream instead of refusing (ir.CopyReplayDeferrer). v0.163.8 measured that a drained add from a PlanetScale/Vitess source is exact although its keyspace-wide copy drops the added table's catch-up deletes, because the stopped stream resumes from its pre-add position and replays every change made during the copy; with --no-drain refused on VStream since v0.163.8, drained is the only mode there, and the terminal refusal above would have turned it into a failed add on any delete during the copy. openSnapshotForAdd now calls DeferCopyReplaysToStream right after opening the snapshot (drained mode only), and the VStream snapshot counts each unappliable replayed change and leaves it unapplied, logging once per table mysql/vstream: snapshot: the copy received changes Vitess replayed inside it and left them to the change stream, which resumes from before the copy and replays them in order with keyspace, table and changes. A key change is now left to the stream, which moves the row, instead of the copy writing its new row beside the old one. Measured on vttestserver (TestStreamer_VStreamShardSkewCopy/AddTable): 40 deletes and 5 key changes during the add's copy, exactly 45 changes logged as left to the stream, and the target exact after the restart, both for a table that existed before the stream was established (1,001 and 159,961 rows) and for one created and filled while the stream was stopped, where the restart also replayed its CREATE TABLE and its whole 40,000-row seed onto the table the add had filled (1,001 and 39,961 rows). v0.163.8's TestStreamer_VStreamAddTableKeyspaceWideCopy passes in every cell. TestCopyReplayDeferralRoster walks every production caller of the table-less OpenSnapshotStream in internal/pipeline and cmd and requires it to defer or carry a reasoned exemption in copyReplayDeferralExempt, failing on a stale exemption, if openSnapshotForAdd stops deferring, and below a floor of two callers. The deferral relies on the stream really being stopped, and nothing checks that yet (DRAINED-ADD-TABLE-RUNNING-STREAM-UNCHECKED, below).

Not closed, stated

• A drained add against a stream that was never stopped (DRAINED-ADD-TABLE-RUNNING-STREAM-UNCHECKED, OPEN, HIGH, silent). Unchanged from v0.163.8: an add run beside a running stream loses the changes made during its copy, now including the replayed changes the copy leaves to the stream, plus every change between the copy and the stream's next poll, at exit 0. It is to be closed in the next release. Until then stop the stream with sluice sync stop --wait and keep it stopped, with no fleet reload and no supervisor or systemd restart, until schema add-table returns.
• A vtgate shard retry before the shard's first copy cursor stalls the copy (VSTREAM-VTGATE-SHARD-RETRY-BEFORE-FIRST-LASTPK, OPEN, MEDIUM). By the Vitess code, a retry with a position and no cursor makes vttablet replicate without copying and never send COPY_COMPLETED, so the copy hangs while heartbeats keep re-arming the watchdog. Loud only as a stall; no silent loss, since the copy never hands off. Not reproduced.
• --no-drain from a Postgres source is not safe either (sluice-testing Bug 332, CRITICAL, silent). Also note the correction now at the top of the v0.163.8 notes and CHANGELOG entry: schema add-table --no-drain from a Postgres source loses changes made to the new table during the add. Do not use --no-drain on any source; use the drained flow. This release does not change that path; the refusal of --no-drain on every source ships in its own release.
• schema add-table from a PlanetScale/Vitess source still copies the whole keyspace (VSTREAM-ADD-TABLE-KEYSPACE-WIDE-COPY, OPEN, loud), as described in v0.163.8.
• Test coverage on this release's branch. Real-cluster suites passed: the filtered move-OUT crux family, the vitesscluster suite (12 of 12), the Vitess chaos suite (6 pass, 1 quarantined skip) and the reshard suite (4 pass, including ShardedColdCopy and StreamerFollowsReshardEndToEnd); local vttestserver pins at one, two and four shards. Every Bug 331 measurement is on a local vttestserver, not on PlanetScale itself. The dispatch-only Extended suites ran on this branch (4c4aac13): the pipeline vstream suite (32 tests, including TestStreamer_VStreamShardSkewCopy), reshard, NOBLOB, cross-version backup, corpus, DDL-fixture and KMS legs passed; the CI chaos leg failed on TestVitessChaos_ErrantGTID_PositionAndLineage ("no position was carried by any change"), which failed the same way on main on 2026-10-18 and on v0.163.8's run, and passed on the local cluster run above.

Compatibility

• Drop-in. No flag added or removed, no default changed. BackupFormatVersion stays 15.
• A new loud failure on a VStream copy: a DELETE, key-changing UPDATE or image-less change arriving inside a still-copying shard's copy now fails that table with VSTREAM-COPY-ROW-CHANGE-REFUSED instead of being dropped. On sync and backup full it is retriable (a first cold start restarts its copy in process; an established stream's re-copy refuses COLD-START-INCOMPLETE; backup full is re-run). A drained schema add-table defers instead of refusing.
• The copy's reported row count no longer includes the finished shards' changes; it is the snapshot's own count (the cycle's 700,483 would now read 700,456).
• On a one-table copy of a multi-shard keyspace where a shard finished early, the change stream after the copy is a new VStream opened at the rewound position rather than the copy's stream continued in place, as the per-table copy already did. One-shard keyspaces, and copies where no shard finished early, continue in place as before.
• Not affected: the standalone VStream CDC reader and its copy-mode start (it decodes both images), the post-copy CDC pump and reshard reopen, migrate (SQL row reader), MySQL/MariaDB binlog sources, Postgres sources, and every non-VStream engine.

Who needs this — action required

• Anyone who cold-started sync (v0.2.0 through v0.163.8) or took a backup full (v0.44.0 through v0.163.8) from a PlanetScale or Vitess keyspace with more than one shard while the source took deletes or primary-key changes. The target may hold rows the source deleted, the old row of a key change, or a row inserted and deleted during the copy. How to tell: this defect only ever adds rows to the target, so let the stream catch up, pause writes on the source, and run sluice verify --depth count against the same source and target, passing the stream's --where values if any: a table with more rows on the target than on the source is affected. A count taken while the source is writing is not reliable, and an unrelated shortfall in the same table can hide the surplus, so a matching count is evidence, not proof. Remedy: re-run sluice sync start with --restart-from-scratch, which clears the in-scope target tables and copies them again (measured on a two-shard vttestserver: an established stream holding 20 rows the source had deleted, re-copied under the same in-copy changes, matched the source exactly). Under --schema-already-applied it refuses while the tables hold rows (RESTART-FROM-SCRATCH-TARGET-NOT-CLEARED): empty them yourself first, minding any foreign keys that reference them from outside the stream's scope, then re-run.
• Anyone with a backup full of such a keyspace: a restore that fails on a duplicate key (SQLSTATE 23505) is this defect, but a restore that succeeds can still hold rows the source deleted, because a dropped delete leaves no duplicate behind (check a restored copy the same way as above). Re-take the full backup on this release.
• Copies of any VStream keyspace, one shard included, that ran through a reparent, vttablet restart or PlanetScale maintenance: by the Vitess code, vtgate's in-copy shard retry replayed deletes that sluice dropped (VSTREAM-VTGATE-SHARD-RETRY-CATCHUP-DELETES). This is derived from the code, not reproduced, and the affected releases are not pinned beyond "any release whose copy skipped image-less rows", which is every VStream snapshot through v0.163.8. If you know such an event overlapped a copy, apply the same check and remedy.
• Anyone using drained schema add-table on a PlanetScale/Vitess source: keep the stream stopped until add-table returns; nothing checks it yet.
• Everyone else: nothing to do.

---

Install: brew install sluicesync/tap/sluice · go install sluicesync.dev/sluice/cmd/sluice@v0.163.9 · Container: ghcr.io/sluicesync/sluice:0.163.9

Full changelog: https://github.com/sluicesync/sluice/blob/main/CHANGELOG.md