Skip to content

v0.99.29

Choose a tag to compare

View all tags
@github-actions github-actions released this 09 Jun 18:14
· 2966 commits to main since this release
v0.99.29
b68b49d

New: PG→PG bulk copy is now fast at scale — and so is the sync start cold-start. This release lands the pgcopydb-competitive throughput arc: sluice migrate copies many-table Postgres schemas concurrently across tables, builds each table's indexes as its copy lands (instead of a serial post-copy index phase), and — for same-engine, no-transform PG→PG — byte-pipes the raw COPY stream past the typed IR. The headline new capability is that sluice sync start's PG-source cold-start now reuses all of that machinery, so the "copy fast and keep following" workflow (pgcopydb's --follow equivalent) finally gets the fast initial copy instead of the old serial one. On a 110 GB / 43-table PG→PG benchmark the combined wins close sluice's end-to-end gap to pgcopydb from 1.75× → 1.42× (1564 s → 1269 s), all zero-loss. Drop-in from v0.99.28 — every speedup is automatic / capability-detected; no config change required.

Features

  • Cross-table copy worker pool (--table-parallelism, ADR-0076) + adaptive --bulk-parallel-min-rows. sluice migrate now copies multiple tables concurrently across the schema (the cross-table axis, pgcopydb's --table-jobs), composed with the existing within-table --bulk-parallelism PK-range splitting — closing the many-medium-table gap where each table sat below the within-table-split threshold and the old serial table loop left cores idle. The two axes multiply, and the product (tableP × withinP) is bounded by the target's connection budget (and --max-target-connections) at a single chokepoint, so concurrency can't exhaust the target. --table-parallelism 0 (default) = auto 4; 1 disables. Separately, --bulk-parallel-min-rows became a 0=auto sentinel that dials the within-table-split threshold down on many-table schemas (base/table-count, floored at 10000), so a many-medium-table migrate engages within-table parallelism instead of single-streaming each table; explicit values are honoured verbatim, single/few-table behaviour is unchanged. ~2.76× on the 30×50k benchmark shape, every run zero-loss. Pinned by the combined-budget invariant matrix + resume-under-concurrency clone-under-lock discipline + the CI -race Integration gate.

  • Index-build overlap (ADR-0077, PG target). migrate now builds each table's secondary indexes as soon as that table's copy lands, concurrently with the still-copying tables, instead of running the whole copy phase and then a separate whole-schema index phase. On the 110 GB / 43-table benchmark the deferred index phase had been a sequential ~457 s tail (29% of total); pgcopydb hides it the same way. Mechanism: the cross-table copy pool fires an onTableCopied callback into a consumer goroutine that drains tables into the engine's incremental index builder, both under one errgroup (a copy error cancels the index pool and vice versa); copy and index connections are open simultaneously, so the connection budget reserves a clamped ~25% slice (capped at 8) for the index pool at the same single chokepoint (indexBudget + copyBudget ≤ CopyBudget, no runtime semaphore). Constraints/FKs/identity-sync still run after the combined phase. Resume is additive: ir.TableProgress.IndexesBuilt (omitempty — old resume tokens decode false and re-feed, a no-op under CREATE INDEX IF NOT EXISTS). PG-only; a MySQL target falls back to the pre-existing post-copy CreateIndexes. Pinned by the budget-split invariant matrix, the overlap orchestrator (fed-once / channel-closed / copy-error-cancels-index / index-error-cancels-copy), the IndexesBuilt wire round-trip, and a PG integration test that asserts the overlap actually happened (min(indexBuildStart) < max(copyComplete), observed ~30 s) + zero-loss.

  • PG→PG identity passthrough (ADR-0078). For a same-engine, no-transform PG→PG copy, migrate now byte-pipes the raw COPY stream — COPY (SELECT …) TO STDOUT → io.Pipe → COPY … FROM STDIN via pgx's pgconn — bypassing the typed IR entirely, which removes the per-value decode/re-encode and closes the per-stream copy-rate gap vs pgcopydb. The lane is auto-engaged behind a single auditable value-fidelity gate: it takes the raw path only when the copy is same-engine and carries no transform (no --redact, no --type-override, no --expr-override, no --inject-shard-column) and the per-table projection excludes OID/wire-format-sensitive types (extension / verbatim / bit / geometry / array — the Bug-74 element-family classes) in v1; any transform present falls back to the IR copy path. --raw-copy-format=text|binary|auto (text default — cross-major safe, pgcopydb's default; binary is used only when source and target server majors match, downgrading to text loudly on a mismatch). The win is eliminating the per-value CPU, not text-vs-binary. Both raw-copy sessions are forced to client_encoding=UTF8 so the byte-pipe is self-consistent regardless of either DSN (an asymmetric client_encoding would otherwise silently corrupt non-ASCII text since the byte-pipe skips the IR per-value re-encode — caught by the value-fidelity review). Slotted into the cold-start fast-loader branch (migrate cold-start only; resume and (until this release) sync stay on the IR path). Pinned by the gate negative matrix (one fallback assertion per transform, asserting the raw lane is NOT taken AND the transform is applied), the per-type-family identity projection, the binary→text negotiate downgrade, and 9 PG integration tests (positive zero-loss text + binary with a raw-lane-taken seam; chunked; generated-column recomputed; resume falls back).

  • Fast parallel cold-start for the sync path (ADR-0079, PG source) — the headline new capability. sluice sync start's PG-source cold-start now reuses the same fast machinery — cross-table pool + index-build overlap + raw passthrough — so the one-command "copy then continuously follow" workflow (the pgcopydb---follow equivalent) gets the fast initial copy instead of the old serial one. All N parallel readers are pinned to the one exported snapshot via SET TRANSACTION SNAPSHOT, so the parallel cold-start reads are snapshot-consistent (gap-free) — strictly better than migrate's per-connection-snapshot window. The capability gate is interface-/field-presence-driven (a shareable exported-snapshot surface on the source), never an engine-name check (IR-first): MySQL and VStream/PlanetScale sources stay on today's serial cold-start with a loud INFO log. The connection budget reserves one slot for the CDC connection that goes live after the bulk copy (tableP × withinP + indexBudget + 1 ≤ CopyBudget). Resume and --schema-already-applied stay serial by design (the durable-watermark + idempotent-COPY coupling lives only on the serial path); multi-database / multi-schema cold-start also stays serial in v1, and the fast lane is documented to never be reached by a spanning/multi-schema stream (holds by construction today). New sync start flags mirror migrate's: --table-parallelism, --bulk-parallelism, --bulk-parallel-min-rows, --bulk-batch-size, --raw-copy-format (all inert on MySQL/VStream sources). The raw-copy value-fidelity gate was refactored off the migrator onto an engine-neutral config so the no-transform guarantee is byte-identical on both migrate and sync paths (a Streamer parity matrix pins it).

Performance

  • 110 GB / 43-table PG→PG benchmark (docs/comparison-pgcopydb.md): index-overlap + passthrough closed sluice's end-to-end gap to pgcopydb from 1.75× → 1.42× (1564 s → 1269 s), all zero-loss. Honest caveat: this run is disk-bound at scale, so the passthrough was the bigger lever (−235 s, removing the per-value IR CPU) and index-overlap the smaller (−60 s — little of the 457 s index tail can hide under a saturated disk). More parallelism (--table-parallelism 8 --bulk-parallelism 8) does nothing here for the same reason. Both wins grow on faster storage, where the structure (overlap) and the per-stream rate (passthrough) stop being masked by I/O.

Compatibility

  • No breaking changes. Drop-in from v0.99.28. All speedups are automatic / capability-detected — no config change is required, and the new flags only let you pin or tune what auto already chooses.
  • Affected directions/engines: the new fast paths are PG→PG / PG-source only. The cross-table pool and adaptive min-rows apply to any migrate target (PG and MySQL); index-overlap is PG-target only (MySQL target falls back to the post-copy whole-schema index build); the raw passthrough is same-engine PG→PG only (any cross-engine or transformed copy uses the unchanged IR path). The fast sync cold-start is PG-source only; MySQL/VStream/PlanetScale cold-starts are unchanged (serial, with a loud INFO log).
  • Unaffected: migrate resume, sync warm-resume, multi-database / multi-schema cold-start, and all cross-engine copies stay on the existing serial / IR path. Single/few-table migrate behaviour and the --bulk-parallel-min-rows semantics for explicit values are byte-compatible.

Who needs this — action required

  • Anyone doing PG→PG migrations or continuous sync at scale — especially the "copy fast and keep syncing" workflow that previously forced a choice between migrate (fast, one-shot) and sync start (follows, but copied serially). No action required: the speedups engage automatically when the source/target support them; you do not need to set any flag. To pin or experiment, the new --table-parallelism / --bulk-parallelism / --bulk-parallel-min-rows / --raw-copy-format flags are available on both migrate and sync start.
  • No re-verification of prior migrations is needed. This release contains no silent-loss fix — there is nothing in a previously published version that this corrects on the data path. (The one encoding fix in this range, client_encoding=UTF8 on the raw-copy sessions, hardens the PG→PG passthrough that is itself first introduced in this same release; no published binary ever shipped that path, so no prior version is affected.)
  • MySQL / cross-engine users: nothing changes for you — those paths are untouched.

Install: brew install sluicesync/tap/sluice · go install sluicesync.dev/sluice/cmd/sluice@v0.99.29 · Container: ghcr.io/sluicesync/sluice:0.99.29
Full changelog: https://github.com/sluicesync/sluice/blob/main/CHANGELOG.md

Assets 15
Loading