Skip to content

v0.20.0

Latest

Choose a tag to compare

@morgo morgo released this 10 Nov 21:15
· 29 commits to main since this release
758e772

Highlights

  • move detects new target hosts and copies into them faster. A move into many shards on new servers was limited by write throughput: every secondary index was maintained during the copy, and the client write budget divided across 128 targets pinned each target at one writer. move now measures each target server. A server is a new host when its base tables, outside the system schemas and the move's own tables, hold under 100 MiB. On new hosts the move:

    Before On new hosts
    Regular secondary indexes Maintained during the copy, unless --defer-secondary-indexes Deferred, restored before the checksum
    Pre-created (empty) target tables Kept their indexes Deferrable indexes dropped, restored before the checksum
    Write workers per target Client budget ÷ targets (1 on a 128-target move) At least 2

    --defer-secondary-indexes is removed from move; see Upgrade Notes. An index a foreign key can use is never deferred. A resume keeps the indexes its targets already have (#1497). check.TargetSchemaDiff accepts the same missing indexes the checks accept (#1489). By @morgo in #1487

    Two related autoscaling fixes for move and sync:

    • The copier's read threads are no longer divided by the shards on one server. That division is correct for the checksum, which reads every target, but copy readers read only the sources. A move to 4 shards on each of 32 × 8-vCPU servers ran its whole copy with 1 reader and a ceiling of 1 (about 296k rows/s, a 76-hour ETA). The copier now gets 2 readers with a ceiling of 4 there (#1482).
    • Read workers scale on the applier queue, not on target load, when no source shares a server with a target. Target load used to shed readers that put no load on the targets. A source that is also a target server keeps the previous behaviour. migrate is unchanged (#1496).

    The new-host defaults have not been benchmarked end to end.

  • move cutovers are bounded, timed and retried. A move cutover holds LOCK TABLES WRITE from the first lock until the rename, and the work under the locks grows with the number of targets (#1478):

    • New --cutover-lock-timeout (default 1m) bounds the flush, the checks and the pre-switch work under the locks. An attempt that runs past it releases the locks without switching traffic and is retried. The traffic-switch callback itself is not bounded.
    • Sources (forward) and targets (reverse) are locked in parallel.
    • Each attempt logs its lock wait, lock hold, how long the tables were blocked, and each step's duration.
    • A reverse cutover attempt that fails before ownership moves now releases the locks and is retried on the next poll (backoff 2s doubling to 10s, up to MaxRetries). Before, it ended the move.
    • The AUTO_INCREMENT carry and the stale _revert cleanup run in parallel. On 17 tables × 128 targets they were about 4,350 and 2,176 serial round trips.
    • A statement under LOCK TABLES that runs past 5s logs the server's metadata locks, without statement text (#1479).

    By @morgo

  • move resumes faster on many targets. Each step below ran serially across targets and logged nothing while it ran:

    Step Change Observed before (128 targets)
    Delete the recopy range All servers at once ~7 min — #1485
    Column check per table per target Removed; the resume_state check already covers it ~3 min 50 s — #1493
    ANALYZE TABLE (also on a fresh run) All servers at once ~5 min — #1501
    Recopy after the copy finished Skipped when the checkpoint has a checksum watermark ~1 hour of deletes, plus the recopy — #1502 #1505

    Targets that share a server still run one after another, so no server gets more than one of these statements at a time. A checkpoint written after the copy but before the checksum's first watermark still takes the old path. By @morgo

  • The checksum reports an ETA. The lockless checker's status line now starts with checksum: instead of lockless: and includes eta=, and Progress().ETA is set during the checksum. The ETA covers the current pass, so a pass that ends with repairs is followed by another and can run past DUE. --legacy-checksum reports no ETA. By @morgo in #1507

  • Correctness fixes:

    • spirit diff restored CHAR() as CHAR_FUNC(). A generated column using CHAR() produced a CREATE TABLE MySQL rejects (error 3763) — by @christianladam in #1509
    • A deferred secondary index could be skipped at restore when a CHECK or FOREIGN KEY constraint had the same name, so the move cut over without the index. This affected v0.19.0 moves with --defer-secondary-indexes — #1489

Upgrade Notes

  • Checkpoint compatibility: complete any in-progress schema changes before upgrading, as the checkpoint format is not guaranteed to be stable between versions.

  • Removed and added flags. Passing a removed flag is a parse error.

    v0.19.0 v0.20.0
    spirit move --defer-secondary-indexes Removed. move defers secondary indexes exactly when it detects new hosts (#1487). spirit sync --defer-secondary-indexes is unchanged.
    — spirit move --cutover-lock-timeout (default 1m) (#1478)
  • move cutover timeout. A forward cutover with many targets can exceed the 1-minute default. It then fails closed (nothing switched, locks released) and is retried. If every attempt times out, raise --cutover-lock-timeout. The per-target cost of the steps under the locks has not been measured (#1478).

  • New-host detection caveats (#1487):

    • information_schema lists only tables the user has a privilege on, so data in schemas the user cannot see is not counted.
    • Any measurement error makes the move treat every target as serving.
    • Moving a few tables into an existing keyspace counts that keyspace's other tables.
  • Relaxed check: move no longer requires visibility of events and stored routines. A missing EVENT or SHOW_ROUTINE grant now logs a warning instead of refusing the move. Views and triggers are still enforced. A hidden event in the source schema can write to the retired _old tables during the reverse window, so a rollback could restore data that differs from the target. This requirement will be re-enabled in a later release. sync is unchanged (#1476).

  • Other behaviour changes:

    • The deferred-cutover sentinel wait limit is 720 hours (30 days), up from 48, for migrate and move (#1477).
    • The checksum status line starts with checksum:, not lockless:, and includes eta=. Update anything that parses it (#1507).
    • move stops its source table statistics updates once the copy completes. In the reverse window they logged Table ... doesn't exist every 5 minutes. A failed update no longer also logs "table statistics updated" (#1480).
  • Go 1.26.10 is required to build from source. It fixes GO-2026-6809, reachable from the GTID change feed's TLS handshake (#1475). github.com/klauspost/compress is bumped to v1.20.2 for GO-2026-6865, reachable when decoding compressed binlog transactions (#1488).

  • Library API changes (for programs embedding Spirit):

    • table.Chunker gains MarkRead() error. Implementations outside Spirit must add it (#1505).
    • Move.DeferSecondaryIndexes is removed (#1487).
    • check.TargetSchemaDiff now accepts a target that lacks deferrable secondary indexes, matching what target_state and resume_state accept (#1489).
    • Privilege refusals now match check.ErrRefused, so a caller can stop retrying without matching the message. A failed force-kill probe stays unmarked (by @aparajon in #1481).
    • table.MockChunker.Reset no longer clears the feedback history (#1506).
    • sentinel.WaitLimit is now 720 * time.Hour (#1477).
    • Added:
      • dbconn.LowerCaseTableNames (by @Kiran01bm in #1500)
      • dbconn.UserDataBytes, dbconn.NewHostDataLimit, autoscale.NewHostMinWriteThreads, statement.DropDeferrableSecondaryIndexes (#1487)
      • concurrency.Plan.CopyReadStart and MaxCopyReadThreads (#1482)
      • copier.AutoscaleConfig.IndependentReads and host.Disjoint (#1496)
      • host.Each (#1501)
      • status.Rate and utils.SubFloor (#1507)
      • table.NewMockChunkerFromChunks, MockChunker.Chunks and MockChunker.ResetCount (#1506)

Additional Changes

  • Refactors. lint and check read lower_case_table_names through dbconn.LowerCaseTableNames (by @Kiran01bm in #1500). The checksum tests use table.MockChunker instead of a private test chunker (#1506). By @morgo

  • Tests, CI and dependencies.

New Contributors

Full Changelog: v0.19.0...v0.20.0