Skip to content

v0.18.0

Choose a tag to compare

View all tags
@morgo morgo released this 07 Oct 12:18
· 53 commits to main since this release
v0.18.0
267aaa5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Highlights

  • The lockless checksum verifies move, and is available for migrate behind a flag. It verifies a table without a setup lock or long-lived REPEATABLE READ snapshots. A row that is written continuously is compared against its next binlog after-image instead of a SELECT. move always uses it, across N sources and M targets, and DistributedChecker is removed (#1281). migrate --enable-experimental-lockless-checksum (default off) uses it as the pre-cutover gate, with the same behaviour as the default checker (#1250 #1270). See pkg/checksum/README.md for how it works and how the two checkers compare. Related changes: hot-row settling (#1273 #1274 #1358), hot-range splitting (#1237 #1249 #1243), retry pacing (#1279 #1403), one checker factory and repair policy (#1251 #1271 #1253 #1405), progress reporting (#1230 #1247), cancellation (by @aparajon in #1258) and docs (#1287 #1288). By @morgo

  • Type and charset correctness. A sweep through type handling, much of it from comparing Spirit with gh-ost on the same types. Most of these were caught by the checksum, which repaired the affected rows or refused the cutover; the ones it could not see are listed first:

    • FLOAT → FLOAT copies lost precision past 6 significant digits. MySQL's text protocol renders a FLOAT with 6 digits (0.12345679 → 0.123457), and the checksum's char cast rendered the same 6, so it compared equal. Conversions out of FLOAT (to DECIMAL, DOUBLE, VARCHAR) were caught by the checksum. Copy and checksum now read the exact DOUBLE value, and FLOAT primary keys are refused — #1311
    • A STORED generated column made regular by the ALTER was not copied. The column was outside the column mapping, so the copy, replay and checksum all skipped it and it came out NULL (the same class of bug is open in gh-ost as github/gh-ost#808) — #1312
    • The AUTO_INCREMENT counter could go backwards at cutover, in both migrate and move, when rows above the copy position were inserted and then deleted during the run (e.g. queue-style tables). The counter is not row data, so the checksum does not cover it. It is now read live and carried over at cutover — #1317
    • A trigger, or an inbound foreign key, created during a migration was lost at cutover. Both were only checked at preflight. They are now detected from the binlog and re-checked under the cutover lock — #1342
    • Binlog replay of strings in non-utf8 charsets (latin1, gbk, utf16, ucs2, …) wrote the wrong bytes. The copy path was unaffected and the checksum repaired replayed rows it saw; the exposure was changes applied between the checksum and the cutover. Strings are now emitted in their column's charset (also github/gh-ost#1571) — #1320
    • DDL text restored from the parser. An explicit CHARACTER SET on an ENUM/SET column was dropped from the executed ALTER (#1352), and a backslash in a DEFAULT or COMMENT literal was restored unescaped, storing a different default (by @Kiran01bm in #1269).
    • Caught by the checksum, now correct the first time: YEAR 0000 copied as 2000 (the XOR checksum only saw an odd count of them per chunk) (#1311); ENUM/SET members containing a 4-byte character, which MySQL reports as ? (#1368), and backslash escapes in COLUMN_TYPE (#1370); a key written to the target ahead of the copier and then discarded above the high watermark (#1395); and row changes dropped past a 4GiB LogPos wrap with gtid_mode off, or with an unknown rows-event subtype such as PARTIAL_UPDATE_ROWS — both now fatal (#1039).
    • Refused instead of failing late. A primary key collation change makes the checksum's range predicate select different rows on each side, so the run copied the table and then aborted before cutover (#1284; predicted at statement scope by @aparajon in #1286). XA transactions write row events at PREPARE, before the outcome is known, so any XA activity on the source now stops the run (#1079).
    • By @morgo unless noted.

  • Force-kill only kills what is actually blocking. Force-kill has been mandatory since v0.17.0, so its mistakes cost more:

    • It killed application transactions holding a granted metadata lock even when the ALTER was not waiting on them, e.g. during an INPLACE rebuild. It now kills only while the statement is waiting for its lock — by @aparajon in #1343
    • A cancel could return a LOCK TABLES session to the pool with the lock still held, so later borrowers failed with error 1100. Lock sessions are now kept until cleanup — #1277 #1278
    • Each retry re-arms the kill timer (by @Kiran01bm in #1280), and retries stop when a kill cannot help (#1331 #1361). #1361 also checks PROCESS and CONNECTION_ADMIN at preflight, and handles MySQL 9.7 error 3854 — by @aparajon
    • A KILL denied with error 1095 is retried as CALL mysql.rds_kill(pid), AWS's supported way to kill another user's session. Before, a user whose rds_superuser_role lacked CONNECTION_ADMIN passed preflight, and every kill at cutover was denied — by @Kiran01bm in #1434
    • The new --force-kill-after flag sets the kill delay separately from --lock-wait-timeout — by @deanbaker in #1263
    • A cutover RENAME that has started now finishes when the run is cancelled (#1367). Related cancel and connect-timeout fixes: #1363 #1365 #1369.

  • spirit diff converges on what MySQL stores, and emits DDL MySQL accepts. A schema file now diffs clean against its own live table in many more cases. Before, some emitted ALTERs would have changed the table if applied: for example, NVARCHAR columns were converted to utf8mb4 (#1301). The emitted DDL was also tested against MySQL on 100 source/target pairs. Statements that failed, changed the table unasked, or never converged are fixed for columns, indexes and foreign keys (#1420) and for partitions (#1417). A MODIFY that changes a column's collation now names it (by @aparajon in #1418). See docs/diff.md. The 21 normalization rules: #1224 #1225 #1268 #1304 #1315 #1316 #1337 #1339 #1341 #1345 #1346 #1347 #1349 #1350 #1353 #1354 #1355 #1356 #1357 #1375, plus #1301; spirit fmt and the normalization pipeline are documented in #1297. By @morgo and @aparajon.

  • move and sync refuse the schema objects they do not copy. Both copy base tables only. A moved table with a trigger was cut over with the trigger left on _old and none on the target, exit 0; a view failed discovery with no primary key found. move now refuses source triggers, views, routines and events (#1364) and target triggers and events (#1393); sync skips source views and refuses target triggers and events (#1371). By @morgo

  • Aurora throttling and autoscaling, across all three commands. migrate has always throttled on Aurora; move and sync only did so with autoscaling engaged, so a move or sync onto Aurora ran with no load throttling at all. They now build the same throttlers for every Aurora target and gain --max-commit-latency (#1289). --enable-experimental-autoscaling is added to move, scaling all shards together on the busiest target host (#1216), and to sync, through the copy, continuous verification and checksum repairs (#1235). The three commands now share one set of flags and one engage rule (#1404). Small Aurora instances (≤2 vCPUs and ≤1.5 GiB buffer pool, e.g. db.t4g.medium) ran out of memory on the default 4 read threads, 4 write threads, 16 MiB chunks and 8 concurrent flushes; with --enable-experimental-autoscaling set they now run in a low-memory mode: 1 read thread, 1 write thread, 1 MiB chunks and 1 concurrent flush (#1426). By @morgo

Upgrade Notes

  • Checkpoint compatibility: complete any in-progress schema changes before upgrading, as the checkpoint format is not guaranteed to be stable between versions.

  • MySQL versions. CI now tests 8.0.28 (Aurora 3.04), 8.0.42 (Aurora 3.10 LTS), 8.0.46, 8.4, 9.7, and MySQL 26.7 for the first time. Aurora 3.04 standard support ends 2026-10-31. After v0.18.0, the 8.0.28 job moves to the next Aurora LTS, so this is the last release tested against MySQL 8.0.28 (#1425).

  • The sentinel only holds a run that set --defer-cutover (#1429). Before, any _spirit_sentinel in the schema held every migrate and move cutover — including one left behind by an earlier deferred run that was cancelled or failed, which could hold an unrelated run for up to 48 hours. Now:

    • Creating _spirit_sentinel by hand during a run started without --defer-cutover no longer holds it. To hold such a run: cancel it, create the table, and resume with --defer-cutover.
    • A deferred run resumed without --defer-cutover cuts over even if the sentinel still exists.
    • A run that finds a sentinel it is not waiting on logs a warning.
    • The hidden --respect-sentinel flag is removed and is now a parse error.

  • Flags are shared across migrate, move and sync (#1404).

    • move --threads defaults to 4 (was 2).
    • move gains --force-kill-after, --lock-wait-timeout, --tls-mode and --tls-ca; sync gains --tls-mode, --tls-ca and --checkpoint-max-age.
    • sync enforces --checkpoint-max-age (default 168h) on resume: an older checkpoint fails with status.ErrCheckpointTooOld rather than re-copying into a non-empty target; --force re-copies.
    • migrate --enable-experimental-autoscaling on a non-Aurora target no longer engages, matching move and sync.
    • Negative --lock-wait-timeout and --max-commit-latency are rejected. Before, they silently became the default and "disabled", respectively.

  • New privileges are required.

    • move on source schemas: EVENT, plus SHOW_ROUTINE on *.* (or SELECT on *.*, or EXECUTE/ALTER ROUTINE/CREATE ROUTINE on the schema). information_schema hides events and routines from a user without them, so the new object check would otherwise miss them (#1364).
    • move on target schemas: TRIGGER and EVENT (#1393).
    • migrate and move: PROCESS, and either CONNECTION_ADMIN/SUPER or EXECUTE on mysql.rds_kill, are now enforced at preflight. They were already needed by force-kill; a user without them now fails before the copy instead of at cutover. rds_superuser_role is no longer accepted by its name; an active role still passes through the privileges it actually grants (#1361 #1434).
    • CREATE TEMPORARY TABLES for tables whose ENUM/SET members contain a 4-byte character (#1368).

  • New refusals. Each of these previously ran and either failed late or produced wrong data:

    • partial_revokes=ON — preflight passed and the run failed at cutover on LOCK TABLES (#1376). Turning it off is a server-wide security change.
    • Changing a primary key column's collation, including utf8mb3_bin → utf8mb4_bin (#1284 #1286).
    • FLOAT and BIT primary keys, in migrate, move and sync (#1311 #1321 #1366).
    • . or a backtick in a schema or table name (#1340).
    • Any XA activity on the source server, including on unrelated tables (#1079); a binlog LogPos wrap; unknown rows-event subtypes (#1039).
    • Inline column REFERENCES in a migration (#1366).
    • move: triggers, views, routines and events (see Highlights).
    • A temporal precision narrowing (e.g. TIMESTAMP(6) → TIMESTAMP) that would truncate a stored fraction now fails the checksum. MySQL accepts the change, but Spirit documents that it does not support lossy changes, and the checksum previously rounded both sides the same way and let it through; this makes it consistent with that rule. If no row holds a fraction, the migration still succeeds (#1294).

  • Behaviour changes.

    • move and sync throttle on Aurora by default, with --max-commit-latency defaulting to 100ms (0 disables it). The flag --enable-experimental-autoscaling now only adds thread-count scaling (#1289).
    • wait_timeout=600 is set on every connection, so a frozen Spirit process can no longer hold LOCK TABLES ... WRITE for the server's default 8 hours (#1321).
    • Periodic flush errors stop the run instead of being logged (#1321), and a fatal abort — including a failed checkpoint write — returns its cause instead of context.Canceled (#1321 #1366).
    • The checksum during the sentinel wait reports, and does not repair (#1253 #1405). A divergence found while --defer-cutover is waiting aborts migrate and move with ErrPermanentDivergence; the checksum watermark is cleared before the wait, so the resumed run's initial checksum is a full pass and repairs it. In move this checksum now runs at --threads (autoscaled when enabled) instead of 1 thread, and a move restarted during the wait re-runs the full initial checksum.
    • sync stops on a divergence found after its first clean pass, instead of repairing it. Its initial verification can return ErrVerificationUnresolved on a range that never converges (#1405).
    • A cancel during cutover can take up to lock_wait_timeout + 30s to return, because a started RENAME is allowed to finish (#1367).
    • spirit diff emits different statements for NCHAR/NVARCHAR, utf8mb3 and other fixed-default charsets, width-less ZEROFILL, and utf8mb4 tables or columns without a COLLATE (#1301 #1304 #1345 #1349 #1357), for the column, index, foreign-key and partition cases in Highlights (#1417 #1420), and a collation-changing MODIFY now names the collation (#1418). These are the cases where the old output did not converge or was not accepted. Some changes are now emitted as more than one statement, in order (e.g. a second FULLTEXT index, a spatial SRID change, VIRTUAL → regular).
    • Lint can report new errors. A new linter, spirit_compatible, reports an error for a new table that Spirit could never alter later: no primary key, a FLOAT or BIT primary key, a foreign key (in either direction), an inline REFERENCES, or a ./backtick in the name. Existing tables are not reported; it can be disabled through Enabled (#1423). The reserved-words list is now the union of what 8.0.46, 8.4, 9.7 and 26.7 reserve, adding INTERSECT, QUALIFY, TABLESAMPLE, EXTERNAL and LIBRARY (#1425).
    • Other lint changes: rename_column reports a case-only change as its own warning, not a rename error (#1232); allow_charset treats utf8 and utf8mb3 as one charset (#1308); type_pedantic recommends the collation that replaces the others (e.g. utf8mb4_0900_ai_ci over utf8mb4_general_ci) even when it is in the minority, instead of the most common one (#1421), and its messages quote and count the example tables (#1387) — by @aparajon.
    • Copy progress in Summary counts rows, not auto-increment keyspace distance (#1220), and the copy ETA averages the last 2 minutes of rate samples instead of the last 10s, so it no longer alternates between its true value and double it under throttling (by @aparajon in #1400).

  • Library API changes (for programs embedding Spirit):

    • The MySQL driver is github.com/block/mysql (#1219). applier.Target.Config, check.SourceResource.Config and dbconn.UnsafeWarningError.Warning carry its types, so a caller passing go-sql-driver/mysql types no longer compiles. Linking both drivers also breaks silently: a tls=rds DSN name only resolves in the driver that registered it, and error classification stops recognising the other driver's *mysql.MySQLError. dbconn.DriverName is the registered name; depguard enforces one driver in-tree (#1222). The driver maps a signed tinyint(1) to a Go bool by default (tinyInt1IsBool), which collapses every non-zero value to true; Spirit disables it in the DSNs it builds, so every Spirit connection reads tinyint(1) as an integer (by @aparajon in #1226).
    • RDS TLS comes from the driver (#1221): the embedded bundle is removed, --tls-mode=DISABLED now sends tls=false explicitly, GovCloud and China endpoints are no longer auto-detected as RDS, the minimum is TLS 1.2, and RejectReadOnly is removed.
    • applier.New(targets []Target, cfg) returns a *MySQLApplier; NewSingleTargetApplier and NewShardedApplier are removed, and under-lock flushes match locks to targets by connection identity in every topology (#1292 #1306).
    • Checksum: checksum.NewChecker is the only constructor, CheckerConfig.Lockless replaces the Algorithm enum, the lockless types are renamed LocklessChecker*, and DistributedChecker is removed (#1250 #1251 #1271 #1281). CheckerConfig.FixDifferences, CheckerConfig.MinPassInterval, ErrRepairUnverified and LocklessChecker.ConfirmedDifferences() are removed, and CheckerConfig.Applier is required (#1405).
    • CLI structs embed flags.Common and flags.Cutover (new pkg/flags), so composite literals must name them: migration.Migration{Common: flags.Common{Threads: 8}, Cutover: flags.Cutover{DeferCutOver: true}}. Field access (m.Threads) is unchanged. Migration.RespectSentinel is removed (#1404 #1429).
    • charset.Collation reports how a collation compares strings (case, accent and kana sensitivity, Binary, UCAVersion, DeprecatedByCollationID); GetCollationByName is renamed FindCollationByName and FindCollationByID is added. 25 utf8mb3_* collations in the registry were wrongly marked NO PAD and are corrected to PAD SPACE — by @aparajon in #1419
    • change.Source gains VerifyRowAtNextChange; RowParker, ParkedRow and FeedStats.MergeSubscriptions are exported for out-of-tree implementations (#1273 #1274).
    • status.Progress.Copy (structured row-copy progress) (#1220); Column.DefaultKind (#1225); a schema load reports the tables its filters excluded (#1241); dbconn.NewContext (#1369); dbconn.ErrLockHeld, so a caller can tell "another run holds this table's advisory lock" from a failed run with errors.Is (by @aparajon in #1430); DBConfig.ForceKillAfter (#1263).
    • dbconn.AnalyzeTable (#1410).
    • Removed: dbconn.BeginStandardTrx (#1278); LazyFindP90, QuoteColumns and other duplicated helpers (#1300); and exported functions nothing in Spirit calls, including lint.Enable/Disable/List/Get/Reset, lint.FilterByLinter, sqlescape.FormatSQL, dbconn.GetLockingTransactions, AdvisoryLock.CloseDBConnection and move.ReverseFeed.Run; KillLockingTransactions drops its unused config parameter (#1412).

Additional Changes

  • move's reverse window checkpoints its feed positions. Before, a restart during --reverse-window re-read the target binlog from cutover, and failed if those binlogs had been purged. By @morgo in #1394

  • One applier. SingleTargetApplier and ShardedApplier are merged into MySQLApplier, with one target as the N=1 case. By @morgo in #1292 #1306

  • Fixes found by comparing the three runners. migrate ignored ANALYZE TABLE error rows (#1410); Cancel before or during Run could panic or be lost (#1408); move and sync now report applier metrics (#1409); and move's status shows its sentinel-wait checksum (#1411). By @morgo

  • Progress accounting. Rows copied and the copy aggregate stay correct across a resume (by @aparajon in #1254 #1255); copier cancellation and phase timing are fixed (#1324 #1325 #1336). An AUTO_INCREMENT key whose key space is more than 5× wider than its rows (e.g. after one very large explicit id) reports progress and ETA in rows instead of key space (by @aparajon in #1433). By @morgo unless noted.

  • Small fixes. Checksum casts for DECIMAL UNSIGNED and BIT(N) (patches by @aparajon in #1294); the index that AUTO_INCREMENT needs is kept when secondary indexes are deferred (#1245); negative replica lag (#1332); a binlog ALTER USER form (#1229); StripAutoIncrement formatting (by @aparajon in #1240); identifier escaping (#1321); and stale tables after native DDL (#1366). By @morgo unless noted.

  • Refactors. Code the three runners duplicated is shared (#1404 #1411 #1422 #1424), as are the binlog and GTID client methods (#1414), along with helper consolidation (#1300 #1307 #1309 #1318) and dead-code removal (#1412). By @morgo

  • Tests, docs and CI. CI stability fixes (#1261 #1265 #1295 #1296 #1305 #1323 #1329 #1348 #1351 #1374 #1382 #1383 #1386 #1389 #1390 #1391 #1392 #1399 #1415 #1428) and docs (#1379 #1406 #1407). By @morgo. Dependency bumps by @dependabot[bot] in #1228 #1259

Full Changelog: v0.17.0...v0.18.0

Assets 6
Loading