Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migration over 100% and staying in migrating state. Never in postpone state. #764

Open
njfix6 opened this issue Jul 20, 2019 · 3 comments
Open

Comments

@njfix6
Copy link

njfix6 commented Jul 20, 2019

We are unable to run the unpostpone command.

The migration says over 100% and when we do the unpostpone command, if says You may only invoke this when gh-ost is actively postponing migration. At this time it is not.. The state is also saying migrating when it should say postpone
Any idea how to resolve this?

Command to run:

gh-ost
--max-load=Threads_running=3000,threads_connected=20000
--critical-load=Threads_running=3000
--chunk-size=1000
--max-lag-millis=1500
--allow-on-master
--verbose
--assume-rbr
--allow-master-master
--cut-over=default
--exact-rowcount
--concurrent-rowcount
--default-retries=20
--panic-flag-file=/tmp/ghost.panic.flag
--postpone-cut-over-flag-file=/tmp/ghost.postpone.flag
--user=<user>
--password=<password>
--host=<host>
--database=<database>
--table=orders
--dml-batch-size=100
--alter='<ALTER>;'
--execute

Here are the logs from gh-ost:

2019-07-20 03:46:47 INFO rotate to next log from mysql-bin-changelog.411212:0 to mysql-bin-changelog.411212
Copy: 33559300/30109847 111.5%; Applied: 442632; Backlog: 0/1000; Time: 6h51m30s(total), 6h51m28s(copy); streamer: mysql-bin-changelog.411212:7928536; State: migrating; ETA: due
Copy: 33584000/30109953 111.5%; Applied: 443032; Backlog: 15/1000; Time: 6h52m0s(total), 6h51m58s(copy); streamer: mysql-bin-changelog.411212:113314793; State: migrating; ETA: due
[2019/07/20 03:47:24] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411213, 4)
[2019/07/20 03:47:24] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411213, 4)
2019-07-20 03:47:24 INFO rotate to next log from mysql-bin-changelog.411213:134278535 to mysql-bin-changelog.411213
2019-07-20 03:47:24 INFO rotate to next log from mysql-bin-changelog.411213:0 to mysql-bin-changelog.411213
Copy: 33614500/30110056 111.6%; Applied: 443431; Backlog: 0/1000; Time: 6h52m30s(total), 6h52m28s(copy); streamer: mysql-bin-changelog.411213:95973538; State: migrating; ETA: due
[2019/07/20 03:47:58] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411214, 4)
[2019/07/20 03:47:58] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411214, 4)
2019-07-20 03:47:58 INFO rotate to next log from mysql-bin-changelog.411214:134292681 to mysql-bin-changelog.411214
2019-07-20 03:47:58 INFO rotate to next log from mysql-bin-changelog.411214:0 to mysql-bin-changelog.411214
Copy: 33641700/30110158 111.7%; Applied: 443810; Backlog: 0/1000; Time: 6h53m0s(total), 6h52m58s(copy); streamer: mysql-bin-changelog.411214:71714977; State: migrating; ETA: due
[2019/07/20 03:48:36] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411215, 4)
[2019/07/20 03:48:36] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411215, 4)
2019-07-20 03:48:36 INFO rotate to next log from mysql-bin-changelog.411215:134317693 to mysql-bin-changelog.411215
2019-07-20 03:48:36 INFO rotate to next log from mysql-bin-changelog.411215:0 to mysql-bin-changelog.411215
Copy: 33671900/30110274 111.8%; Applied: 444268; Backlog: 0/1000; Time: 6h53m30s(total), 6h53m28s(copy); streamer: mysql-bin-changelog.411215:50180359; State: migrating; ETA: due
[2019/07/20 03:49:14] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411216, 4)
[2019/07/20 03:49:14] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411216, 4)
2019-07-20 03:49:14 INFO rotate to next log from mysql-bin-changelog.411216:134339835 to mysql-bin-changelog.411216
2019-07-20 03:49:14 INFO rotate to next log from mysql-bin-changelog.411216:0 to mysql-bin-changelog.411216
Copy: 33697000/30110371 111.9%; Applied: 444643; Backlog: 0/1000; Time: 6h54m0s(total), 6h53m58s(copy); streamer: mysql-bin-changelog.411216:16571288; State: migrating; ETA: due
Copy: 33724200/30110463 112.0%; Applied: 444986; Backlog: 0/1000; Time: 6h54m30s(total), 6h54m28s(copy); streamer: mysql-bin-changelog.411216:126988305; State: migrating; ETA: due
[2019/07/20 03:49:51] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411217, 4)
[2019/07/20 03:49:51] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411217, 4)
2019-07-20 03:49:51 INFO rotate to next log from mysql-bin-changelog.411217:134320044 to mysql-bin-changelog.411217
2019-07-20 03:49:51 INFO rotate to next log from mysql-bin-changelog.411217:0 to mysql-bin-changelog.411217
[2019/07/20 03:50:00] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411218, 4)
[2019/07/20 03:50:00] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411218, 4)
2019-07-20 03:50:00 INFO rotate to next log from mysql-bin-changelog.411218:31034577 to mysql-bin-changelog.411218
2019-07-20 03:50:00 INFO rotate to next log from mysql-bin-changelog.411218:0 to mysql-bin-changelog.411218
Copy: 33754100/30110555 112.1%; Applied: 445335; Backlog: 0/1000; Time: 6h55m0s(total), 6h54m58s(copy); streamer: mysql-bin-changelog.411218:71285399; State: migrating; ETA: due
[2019/07/20 03:50:39] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411219, 4)
[2019/07/20 03:50:39] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411219, 4)
2019-07-20 03:50:39 INFO rotate to next log from mysql-bin-changelog.411219:134264206 to mysql-bin-changelog.411219
2019-07-20 03:50:39 INFO rotate to next log from mysql-bin-changelog.411219:0 to mysql-bin-changelog.411219
Copy: 33780700/30110659 112.2%; Applied: 445738; Backlog: 0/1000; Time: 6h55m30s(total), 6h55m28s(copy); streamer: mysql-bin-changelog.411219:32264928; State: migrating; ETA: due
Copy: 33809100/30110793 112.3%; Applied: 446249; Backlog: 0/1000; Time: 6h56m0s(total), 6h55m58s(copy); streamer: mysql-bin-changelog.411219:131014331; State: migrating; ETA: due
[2019/07/20 03:51:20] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411220, 4)
[2019/07/20 03:51:20] [info] binlogsyncer.go:723 rotate to (mysql-bin-changelog.411220, 4)
2019-07-20 03:51:20 INFO rotate to next log from mysql-bin-changelog.411220:134265429 to mysql-bin-changelog.411220
2019-07-20 03:51:20 INFO rotate to next log from mysql-bin-changelog.411220:0 to mysql-bin-changelog.411220````



@njfix6
Copy link
Author

njfix6 commented Jul 22, 2019

It seems this issue may be related: #767

@njfix6
Copy link
Author

njfix6 commented Jul 22, 2019

Confirmed the size of the table is: 62087808

@cool-firer
Copy link

any updates?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants