New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sync-diff-inspector: support checkpoint #224

Merged

WangXiangUSTC merged 65 commits into pingcap:master from WangXiangUSTC:xiang/checkpoint

Apr 9, 2019

Contributor

WangXiangUSTC commented Mar 26, 2019

What problem does this PR solve?

save sync-diff-inspector's checkpoint, and then can stop it and restart to continue check.

What is changed and how it works?

update chunk's struct
save chunk's info to downstream when check
when start sync-diff-inspector, will load checkpoint information
refine some code

Check List

Tests

Unit test
Manual test (stop and then restart)

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to be included in the release note

WangXiangUSTC added 15 commits

March 24, 2019 23:38


          update chunk struct

52cf7c5


          save chunk information after generate

34d1730


          add a column

510ae13


          add ExecSQLWithRetry function

3f4d938


          add function to load chunk info

b3c7ac9


          add function to get config hash


          remove struct checkjob

c455da0


          add ignore state

4a1a8fa


          move some code into function checkChunkDataEqual

17ed0a3


          update chunk info after check

4f98312


          update summary info every 10 second

1b52129


          modify update chunk

975bf1d


          refine code

bbc4529


          add unit test

8b32443


          remove useless code

67b9b1f

WangXiangUSTC added priority/important status/PTAL type/feature labels

WangXiangUSTC added 2 commits

March 26, 2019 20:34


          minor update

f087f6c


          add comment

6a73fdb

Contributor Author

WangXiangUSTC commented Mar 26, 2019

/run-all-tests

WangXiangUSTC requested review from IANTHEREAL and csuzhangxc

March 26, 2019 13:01


          minor fix

b4fa0c2

csuzhangxc reviewed

View reviewed changes

pkg/dbutil/common.go Outdated Show resolved Hide resolved

csuzhangxc reviewed

View reviewed changes

pkg/dbutil/common.go Show resolved Hide resolved

csuzhangxc reviewed

View reviewed changes

pkg/diff/chunk.go Outdated Show resolved Hide resolved

pkg/diff/chunk.go Outdated Show resolved Hide resolved

WangXiangUSTC added 3 commits

March 28, 2019 13:53


          use json marshal in chunk.String


          add ignore err and retryable err function

f9d2cd3


          add config use-checkpoint

1ded757


          update table name

18ce1e1

csuzhangxc reviewed

View reviewed changes

pkg/dbutil/common.go Show resolved Hide resolved

pkg/dbutil/common.go Show resolved Hide resolved

WangXiangUSTC added 2 commits

April 3, 2019 15:07


          print slow log although exec sql failed

06ee87d


          format code

b28fdb2

csuzhangxc reviewed

View reviewed changes

pkg/dbutil/common.go Outdated Show resolved Hide resolved

csuzhangxc reviewed

View reviewed changes

pkg/diff/checkpoint.go Outdated Show resolved Hide resolved

pkg/diff/checkpoint.go Outdated Show resolved Hide resolved

pkg/diff/checkpoint.go Outdated Show resolved Hide resolved

pkg/diff/checkpoint.go Show resolved Hide resolved

csuzhangxc reviewed

View reviewed changes

pkg/diff/chunk.go Outdated Show resolved Hide resolved

csuzhangxc reviewed

View reviewed changes

pkg/diff/diff.go Show resolved Hide resolved

pkg/diff/diff.go Outdated Show resolved Hide resolved

csuzhangxc reviewed

View reviewed changes

pkg/diff/diff.go Outdated Show resolved Hide resolved

csuzhangxc reviewed

View reviewed changes

pkg/diff/diff.go Show resolved Hide resolved

WangXiangUSTC added 5 commits

April 4, 2019 10:38


          don't sleep at the last retry time && judge ctx.Done

0b340af


          return err when value is invalid && remove defer in loop

fda47a9


          address comment

d3e2c3e


          return not found error when no chunk found

a4f5261


          minor update

4272d43

Contributor Author

WangXiangUSTC commented Apr 4, 2019

@GregoryIan @csuzhangxc PTAL again


          remove prepare function

dd73053

Collaborator

IANTHEREAL commented Apr 4, 2019

LGTM

IANTHEREAL added status/LGT1 and removed status/PTAL labels


          close channel after no chunk send to channel

1c68b9a

csuzhangxc reviewed

View reviewed changes

pkg/diff/diff.go Show resolved Hide resolved

pkg/diff/diff.go Outdated Show resolved Hide resolved

pkg/diff/diff.go Show resolved Hide resolved

csuzhangxc reviewed

View reviewed changes

sync_diff_inspector/config.toml Show resolved Hide resolved

sync_diff_inspector/config.go Show resolved Hide resolved


          address comment

5f20455

csuzhangxc approved these changes

View reviewed changes

Member

csuzhangxc left a comment

LGTM

csuzhangxc added status/LGT2 and removed status/LGT1 labels

WangXiangUSTC merged commit d9985c6 into pingcap:master

WangXiangUSTC deleted the xiang/checkpoint branch

April 9, 2019 08:36

WangXiangUSTC mentioned this pull request

sync-diff-inspector: support checkpoint #227

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority/important status/LGT2 type/feature