Skip to content

v0.19.0

Latest
Latest

Choose a tag to compare

View all tags
@morgo morgo released this 22 Oct 16:59
· 15 commits to main since this release
v0.19.0
4e35849
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • Autoscaling is on by default. On Aurora, Spirit now sets its thread counts from the instance size in migrate, move and sync:

    Target Thread counts
    Aurora, 4 or more vCPUs Autoscaling: read, write, checksum and flush pools are sized from the instance and scaled on throttler feedback
    Aurora, fewer than 4 vCPUs Small-instance mode: 1 read thread, 1 write thread, 1 MiB chunks, flush width 1, nothing scales
    Not Aurora, or --skip-autoscaling --threads / --write-threads as configured (unchanged)

    --enable-experimental-autoscaling is replaced by --skip-autoscaling (#1436). Small-instance mode replaces v0.18.0's low-memory mode. It no longer looks at the buffer pool, so 2-vCPU instances with ample memory (db.r6g.large, db.r7g.large, db.t4g.large) are included (#1438). The Aurora throttlers run either way. By @morgo

  • migrate uses the lockless checksum by default. It takes no LOCK TABLES at the start of each pass and holds no long-lived REPEATABLE READ snapshot, so a long checksum no longer grows the InnoDB history list. move and sync already used only this checker. --legacy-checksum selects the previous snapshot checker (#1437). For callers that decide whether to retry a failed checksum, checksum.IsReproducible is now the single answer, and SingleChecker no longer reports ErrDifferencesExhausted for a run whose repair was never re-read (by @Kiran01bm in #1448).

  • Experimental: migrate on tables with foreign keys, on MySQL 9.7+. MySQL 9.6 moved foreign key enforcement into the SQL layer, so cascaded changes are now written to the binary log. --enable-experimental-foreign-keys (default off) allows altering a child table: any copy-path ALTER, DROP FOREIGN KEY, and renaming a column a foreign key uses. It requires MySQL 9.7+ with innodb_native_foreign_keys=OFF (the default). These are still refused: tables referenced by another table's foreign key (including self-references), ADD FOREIGN KEY or inline REFERENCES, and DROP CONSTRAINT of a foreign key. _new has no foreign keys during the copy. They are added under the cutover lock with foreign_key_checks=0, after a probe on empty copies confirms MySQL will not build an index. Force-kill covers sessions blocking on the parent tables. move and sync are unchanged. See docs/migrate.md. By @morgo in #1431

  • Correctness fixes:

    • The final change-feed flush under the cutover lock skipped MySQL warning checks. A truncating or out-of-range value that arrived only in that last batch could be written to _new right before it became live. Writes under the lock now use the same unsafe-warning policy as every other write, and a warning stops the cutover — #1470
    • The privileges check could pass when MySQL would not grant the privileges. MySQL applies one database-level grant row to a schema, so an exact-name grant can shadow a pattern grant. The base set is now required on every matching database-level grant — #1442
    • Copies with composite or non-auto-increment keys ignored --target-chunk-size and sized chunks by elapsed time. When writes queued, chunks could shrink to the 10-row minimum — #1469
    • A valid migration of a table with CHECK constraints was refused when CREATE TABLE ... LIKE rewrote a charset introducer in the copied expression. The source definitions are now restored on the empty copy before the ALTER — by @budzeg in #1461
    • spirit diff / DeclarativeToImperative: a schema that defines the same table twice was silently planned from the second definition, and a case-only rename on a lower_case_table_names=1|2 server produced CREATE + DROP (by @Kiran01bm in #1449). A repeated column, index or constraint name could panic Diff, return "no change", or emit an ALTER MySQL rejects (by @aparajon in #1466). Both cases now return an error.

Upgrade Notes

  • Checkpoint compatibility: complete any in-progress schema changes before upgrading, as the checkpoint format is not guaranteed to be stable between versions.

  • Removed and renamed flags. None has an alias. Passing an old name is a parse error.

    v0.18.0 v0.19.0
    --enable-experimental-autoscaling Removed; autoscaling is the default. --skip-autoscaling turns it off.
    --enable-experimental-lockless-checksum Removed; lockless is the default. --legacy-checksum selects the snapshot checker.
    --checksum-yield-timeout --legacy-checksum-yield-timeout (only applies with --legacy-checksum)

  • Behaviour changes on Aurora (#1436 #1438):

    • --threads and --write-threads are ignored on Aurora unless you pass --skip-autoscaling. A non-default value logs a warning when it is replaced.
    • 2-vCPU instances that ran at the configured counts in v0.18.0 (default 4/4, 16 MiB chunks, flush width 8) now run at 1/1, 1 MiB chunks, flush width 1. We expect lower copy throughput on those instance sizes. This has not been measured. --skip-autoscaling restores the old behaviour.
    • Non-Aurora servers are unchanged.

  • Lockless checksum behaviour (#1437): a lossy ALTER (for example, a UNIQUE index added to non-unique data) now fails after up to 10 lockless passes instead of 3 snapshot attempts. Each pass re-walks the table, so this failure takes longer on large tables. The error still wraps ErrDifferencesExhausted. The checksum status row reads lockless: instead of experimental lockless:.

  • Stricter checks:

    • Privileges: a pattern grant (app\_%.*) without the full base set, next to an exact-name grant with it, is now refused (#1442).
    • move now runs the version check at preflight against every source and target (#1458).
    • sync checks the source's binary log settings (binlog_format=ROW, binlog_row_image=FULL, and others), and now requires TRIGGER and EVENT on the target schema (#1458).
    • Cutover: an unsafe warning during the final flush under the lock now stops the cutover (#1470).

  • Other behaviour changes:

    • move cutover retries back off (100ms, doubling, capped at 10s), as migrate already did. The final error lists every attempt as attempt N: …. The log key max-retries is now max_retries (#1457).
    • sync: the copy's write ceiling is no longer capped to --max-connections, and repairs run at --write-threads (#1463).
    • spirit lint, spirit diff and spirit fmt connect the same way as the rest of Spirit (#1456):
      • TLS defaults to PREFERRED. Add tls=false to the DSN for the old behaviour.
      • The session time_zone is +00:00. On a non-UTC server, literal TIMESTAMP defaults can render differently.
      • spirit fmt runs under Spirit's sql_mode, so DEFAULT '0000-00-00' is now accepted.
      • A --database value containing ? or % is no longer misread as driver parameters.
    • spirit diff reads lower_case_table_names from --source-dsn and matches table names case-insensitively when it is 1 or 2 (#1449).

  • Go 1.26.9 is required to build from source. It fixes 7 standard-library vulnerabilities that govulncheck reports as reachable (#1440).

  • Library API changes (for programs embedding Spirit):

    • flags.Common.EnableExperimentalAutoscaling is now SkipAutoscaling. The zero value now means "autoscale", so a caller that set nothing gets autoscaling on Aurora. Set SkipAutoscaling: true to keep the old behaviour (#1436).
    • Migration.EnableExperimentalLocklessChecksum is now LegacyChecksum, with the opposite meaning. A caller that set EnableExperimentalLocklessChecksum: false must set LegacyChecksum: true to keep the snapshot checker. ChecksumYieldTimeout is now LegacyChecksumYieldTimeout (#1437).
    • New flags.Common.ThreadsAreBaseline (programmatic only). It marks your Threads/WriteThreads as a baseline for non-Aurora servers, so autoscaling replaces them without logging a warning. A caller whose counts are a real load cap should set SkipAutoscaling instead (#1439).
    • pkg/migration/check and pkg/move/check are merged into pkg/check (#1458):
      • ScopeX → PhaseX.
      • check.ChecksInScope(check.ScopeStatement) → check.Checks(check.Migration, check.PhaseStatement).
      • TargetSchemaObjectsError(ctx, targets, tables) → TargetSchemaObjectsError(ctx, resources).
      • StatementRefusal is unchanged.
    • Small-instance mode renames: Plan.LowMemory → Plan.SmallInstance, and the LowMemory* constants → SmallInstance*. LowMemoryMaxBufferPoolBytes, LowMemoryMaxVCPUs and dbconn.BufferPoolSize are removed (#1438).
    • Added:
      • checksum.IsReproducible (#1448)
      • statement.DiffOptions.LowerCaseTableNames (#1449)
      • throttler.AuroraResult.Close, throttler.BuildAurora, dbconn.MonitorOpener (#1454)
      • runtime.RetryCutover (#1457) and runtime.Lifecycle.Teardown (#1463)
      • dbconn.NewTableLockReferencing and TableLock.ExecUnderLockWithoutForeignKeyChecks (#1431), and TableLock.ExecUnderLockWithWarnings (#1470)
      • utils.DBLevelGrantsCoverSchema (#1442)
      • CompareMySQLVersions moves from pkg/testutils to pkg/utils (#1431)
    • Removed:
      • ApplierConfig.ChunkletMaxRows and ChunkletMaxSize. They were never read; batches are fixed at 1,000 rows / 1 MiB (#1468).
      • CopierConfig.DBConfig, CheckerConfig.TargetChunkTime, Copier.StartTime, Copier.GetProgress, NewTrxPool's config parameter, utils.ParseRoleNames and dbconn.KillTransaction (#1450).
      • copier.StartWriteAutoscaler, concurrency.Request.Fit and Plan.Cap (#1463).

Additional Changes

  • move fixes. A sharded move whose later source failed to connect leaked the pools it had already opened (#1453). The reverse window is now the only writer of the checkpoint row once it starts. This is defense in depth for #1239 (#1447). By @morgo

  • Runner unification. Code that migrate, move and sync each had their own copy of is now shared:

    • one check registry (#1458)
    • the cutover retry loop (#1457)
    • Aurora throttler setup and teardown (#1454)
    • Close teardown order and read-bound fitting (#1463)

    AGENTS.md no longer has a "not yet unified" list. The remaining differences are recorded as deliberate. By @morgo

  • Refactors. Feed code shared by the binlog and GTID clients now lives on feedCore (#1455). Dead fields, constants and test-only helpers are removed (#1450). Goroutine, timer, warning and TLS code paths are simplified (#1451). By @morgo

  • Tests, docs and dependencies.

    • Test de-flaking (#1444 #1446).
    • Package READMEs checked against current behaviour (#1467).
    • The docs now say the copier writes INSERT IGNORE, not REPLACE (#1452).
    • The non-strict sql_mode is documented: zero dates copy, and out-of-range values (1264) still stop the copy (by @aparajon in #1465).
    • Dependency bumps (#1464 #1473 #1474).

New Contributors

Full Changelog: v0.18.0...v0.19.0

Assets 6
Loading