Skip to content

Conversation

@morgo
Copy link
Collaborator

@morgo morgo commented Aug 27, 2025

A Pull Request should be associated with an Issue.

We wish to have discussions in Issues. A single issue may be targeted by multiple PRs.
If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues,
and potentially we'll be able to point development in a particular direction.
Further notes in https://github.com/block/spirit/blob/main/.github/CONTRIBUTING.md

Fixes #394
Fixes #393
Fixes #435

The multi path does not yet support checkpoints, checksum, resuming, sentinel tables. There are no tests for multi-path yet: it is working end to end though:

mtocker@BLKD5234QXY40 spirit % go run . --statement="ALTER TABLE bigtable ENGINE=innodb; ALTER TABLE bigtable2 ENGINE=innodb;" --multi --threads=8 --target-chunk-time=5s
INFO[0000] Starting spirit migration: concurrency=8 target-chunk-size=5s
INFO[0000] could not resume from checkpoint: reason=resume-from-checkpoint is not yet supported in multi-statement migrations
INFO[0000] create BinlogSyncer with config {ServerID:1149 Flavor:mysql Host:127.0.0.1 Port:3306 User:spirit Password: Localhost: Charset: SemiSyncEnabled:false RawModeEnabled:false TLSConfig:<nil> ParseTime:false TimestampStringLocation:UTC UseDecimal:false RecvBufferSize:0 HeartbeatPeriod:0s ReadTimeout:0s MaxReconnectAttempts:0 DisableRetrySync:false VerifyChecksum:false DumpCommandFlag:0 Option:<nil> Logger:0x1400028b580 Dialer:0x1007de8f0 RowsEventDecodeFunc:<nil> TableMapOptionalMetaDecodeFunc:<nil> DiscardGTIDSet:false EventCacheCount:10240}
INFO[0000] begin to sync binlog from position (BLKD5234QXY40-bin.015432, 682188214)
INFO[0030] migration status: state=copyRows copy-progress=7545795/40094411 18.82% binlog-deltas=0 total-time=30s copier-time=30s copier-remaining-time=TBD copier-is-throttled=false conns-in-use=9
INFO[0030] finished periodic flush of binary log: total-duration=1.527833ms batch-size=1000

INFO[0060] finished periodic flush of binary log: total-duration=1.158083ms batch-size=1000
INFO[0060] migration status: state=copyRows copy-progress=14320591/40094411 35.72% binlog-deltas=0 total-time=1m0s copier-time=1m0s copier-remaining-time=2m43s copier-is-throttled=false conns-in-use=8
WARN[0067] dynamic chunking is not working as expected: target-time=5s p90-time=1.924584ms new-target-rows=259796402 max-dynamic-row-size=100000
WARN[0067] switching to prefetch algorithm
WARN[0067] disabling chunk prefetching: min-val=13731117 max-val=13732618 max-dynamic-row-size=100000
INFO[0090] migration status: state=copyRows copy-progress=21786956/40094411 54.34% binlog-deltas=0 total-time=1m30s copier-time=1m30s copier-remaining-time=2m21s (±0m from 0s ago) copier-is-throttled=false conns-in-use=8
INFO[0090] finished periodic flush of binary log: total-duration=1.569583ms batch-size=1000
INFO[0120] migration status: state=copyRows copy-progress=26471270/40094411 66.02% binlog-deltas=0 total-time=2m0s copier-time=2m0s copier-remaining-time=1m20s (±0m from 1m ago) copier-is-throttled=false conns-in-use=8
INFO[0120] finished periodic flush of binary log: total-duration=1.10875ms batch-size=1000
INFO[0150] migration status: state=copyRows copy-progress=30598135/40094411 76.32% binlog-deltas=0 total-time=2m30s copier-time=2m30s copier-remaining-time=1m2s (±0m from 1m ago) copier-is-throttled=false conns-in-use=9
INFO[0150] finished periodic flush of binary log: total-duration=1.240709ms batch-size=1000
INFO[0180] approaching the end of the table, synchronously updating statistics
INFO[0180] finished periodic flush of binary log: total-duration=1.372917ms batch-size=1000
INFO[0180] migration status: state=copyRows copy-progress=35555319/40094411 88.68% binlog-deltas=0 total-time=3m0s copier-time=3m0s copier-remaining-time=24s (±0m from 2m ago) copier-is-throttled=false conns-in-use=8
INFO[0192] copy rows complete
INFO[0192] waiting to catch up to source position: (BLKD5234QXY40-bin.015458, 300798619), current position is: (BLKD5234QXY40-bin.015458, 243865612)
INFO[0192] Running ANALYZE TABLE
INFO[0192] waiting to catch up to source position: (BLKD5234QXY40-bin.015460, 997), current position is: (BLKD5234QXY40-bin.015460, 997)
WARN[0192] Attempting final cut over operation (attempt 1/5)
WARN[0192] trying to acquire table locks, timeout: 30
WARN[0192] table lock(s) acquired
INFO[0192] waiting to catch up to source position: (BLKD5234QXY40-bin.015461, 197), current position is: (BLKD5234QXY40-bin.015461, 197)
WARN[0192] table lock released
WARN[0192] final cut over operation complete
INFO[0194] successfully dropped old table: _bigtable_old
INFO[0195] successfully dropped old table: _bigtable2_old
INFO[0195] apply complete: instant-ddl=false inplace-ddl=false total-chunks=860 copy-rows-time=3m12s checksum-time=0s total-time=3m15s conns-in-use=0
INFO[0195] syncer is closing...
INFO[0195] kill last connection id 34863
INFO[0195] syncer is closed

This commit adds support for atomic multi-table changes to Spirit:

- Refactor checksum to support multi-chunker architecture
- Add new change abstraction to encapsulate table operations
- Update migration runner to handle multiple table changes
- Add support for multi-statement migrations with proper rollback
- Update tests to work with new multi-table structure

The changes maintain backward compatibility while enabling
atomic operations across multiple tables.
@morgo morgo changed the title Add atomic multi-table changes support Refactor migration to support multi chunker Aug 27, 2025
@morgo morgo force-pushed the refactor-for-multi-3 branch from c60883d to c56ec0f Compare August 27, 2025 15:31
@morgo morgo requested a review from kolbe September 2, 2025 13:33
@morgo morgo enabled auto-merge September 15, 2025 13:57
@morgo morgo merged commit b4803f5 into block:main Sep 15, 2025
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement multi-chunker Refactor runner to support multiple statements Have statement.New() support returning multiple statements

2 participants