How to recover from "ETA: postponing cut-over" ?

[linwaterbin]

fisrt, this tool is very wonderful,being full of imagination,thank you a lot;
I test my environment, my table PlayerInventory,which rows have 3 million,size is 30G
when gh-ost running, I execute a UPDATA statement, the gh-ost output as following:

Copy: 11241000/299550386 3.8%; Applied: 0; Backlog: 0/100; Time: 5m0s(total), 3m41s(copy); streamer: binlog20000.000052:21082625; ETA: 1h34m28s
2016-08-11 15:49:30 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
2016-08-11 15:49:30 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction
2016-08-11 15:49:30 INFO Row copy complete
Copy: 11241000/299550386 3.8%; Applied: 0; Backlog: 0/100; Time: 5m26s(total), 4m7s(copy); streamer: binlog20000.000052:21091001; ETA: 1h45m49s
Copy: 11241000/299550386 3.8%; Applied: 0; Backlog: 0/100; Time: 5m30s(total), 4m7s(copy); streamer: binlog20000.000052:21092614; ETA: postponing cut-over
Copy: 11241000/299550386 3.8%; Applied: 0; Backlog: 0/100; Time: 5m35s(total), 4m7s(copy); streamer: binlog20000.000052:21094542; ETA: postponing cut-over

next, has been in the state, that is, " ETA: postponing cut-over ", never heals,but update already over
how do I let gh-ost to continue to perform rather than terminate
my command as follows:

./gh-ost \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--throttle-control-replicas="replip" \
--max-lag-millis=1500 \
--user="gh" \
--password="gh123" \
--host="replip" \
--port=20000 \
--database="test" \
--table="PlayerInventory" \
--verbose \
--alter="change orig_source orig_source bigint" \
--switch-to-rbr \
--allow-master-master \
--cut-over=default \
--exact-rowcount \
--default-retries=120 \
--postpone-cut-over-flag-file=/data/dbbak/gh-ost/ghost.postpone.flag \
--execute

anyone can help me? thank you before : )

[shlomi-noach]

You have explicitly set --postpone-cut-over-flag-file=/data/dbbak/gh-ost/ghost.postpone.flag

To cut-over, either:

• rm /data/dbbak/gh-ost/ghost.postpone.flag
• echo unpostpone | socat - /path/to/your/gh-ost.sock -- check the gh-ost output to find out what that file is.

[shlomi-noach]

@linwaterbin confirmed? I'd like to close this

[linwaterbin]

Sorry,shlomi, I'm sorry message to see late,
When rm directly the postpone flag file,which resulting in data inconsistency,
My UPDATA command run over 5 minutes, I get ride of explicitly set --postpone-cut-over-flag-file=/data/dbbak/gh-ost/ghost.postpone.flag, the master will struct in "Waiting for table metadata lock",

[shlomi-noach]

I apologize, I'm not sure I can understand this last statement.

[linwaterbin]

Before My UPDATE statement did not add index, a full table scan,which resulting in MDL lock;
Now,added index, I don't have this problem, gh-ost can run very well 👍
thank you shlomi : )

[shlomi-noach]

Oh good! Glad to hear, thank you for sharing!

[hhl419]

@linwaterbin,You should execute update,then execute gh-ost.Because update hold MDL lock，and Haven't release.

[linwaterbin]

@hhl419
No, my order is that gh-ost and then update