Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ddl: dynamic adjust add index worker number. #8295

Merged
merged 28 commits into from Dec 24, 2018

Conversation

Projects
None yet
5 participants
@crazycs520
Copy link
Contributor

crazycs520 commented Nov 13, 2018

What problem does this PR solve?

Currently, if tidb is doing a add index ddl job, then the client change the tidb_ddl_reorg_worker_cnt want to speed up add index, but this change will take effect next add index, won't affect current add index job.

What is changed and how it works?

Check List

Tests

  • Unit test

Code changes

Side effects

Related changes


This change is Reviewable

crazycs520 added some commits Nov 14, 2018

@crazycs520 crazycs520 force-pushed the crazycs520:adjust-add-index-worker branch from 604526b to f936c59 Nov 14, 2018

crazycs520 added some commits Nov 14, 2018

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Nov 15, 2018

/run-all-tests

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Nov 15, 2018

@crazycs520 crazycs520 changed the title ddl: dynamic adjust add index worker number init ddl: dynamic adjust add index worker number. Nov 19, 2018

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Nov 19, 2018

/run-all-tests

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Nov 19, 2018

/run-integration-ddl-test

1 similar comment
@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Nov 19, 2018

/run-integration-ddl-test

@winkyao

This comment has been minimized.

Copy link
Member

winkyao commented Nov 20, 2018

@crazycs520 Please find out why integration-ddl-test failed, https://github.com/pingcap/tidb-test/tree/master/ddl_test

}
if startReorganization {
workerCnt := rand.Intn(8) + 8
s.mustExec(c, fmt.Sprintf("set @@tidb_ddl_reorg_worker_cnt=%d", workerCnt))

This comment has been minimized.

Copy link
@winkyao

winkyao Nov 20, 2018

Member

How can we ensure the worker count is changed to workerCnt?

This comment has been minimized.

Copy link
@crazycs520

crazycs520 Nov 20, 2018

Author Contributor

em... I check the log when do manual test to know the worker count is changed... 😂

Show resolved Hide resolved ddl/index.go
Show resolved Hide resolved ddl/index.go
@winkyao
Copy link
Member

winkyao left a comment

main changes LGTM

}

reorgInfo.d.mu.Lock()
reorgInfo.d.mu.hook.OnIndexWorkerReorgBefore(len(idxWorkers), len(kvRanges))

This comment has been minimized.

Copy link
@zimulala

zimulala Dec 19, 2018

Member

Could we use gofail instead of this logic?

@@ -630,8 +680,16 @@ LOOP:
s.mustExec(c, sql)
}
num += step
case <-ticker2.C:
if changeWorkerNumEnable {

This comment has been minimized.

Copy link
@zimulala

zimulala Dec 19, 2018

Member

It will have data race.

@crazycs520 crazycs520 force-pushed the crazycs520:adjust-add-index-worker branch from 0befa35 to d86e5b4 Dec 21, 2018

@crazycs520 crazycs520 force-pushed the crazycs520:adjust-add-index-worker branch from d86e5b4 to 77f49bd Dec 21, 2018

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Dec 21, 2018

/run-all-tests

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Dec 21, 2018

/run-all-tests

@zimulala
Copy link
Member

zimulala left a comment

LGTM

s.tk.MustExec("drop table test_add_index")
}

func (s *testDBSuite) TestAddIndexWorkerNum(c *C) {

This comment has been minimized.

Copy link
@zimulala

zimulala Dec 21, 2018

Member

Please put it to failtest/fail_db_test.go

crazycs520 added some commits Dec 21, 2018

@crazycs520 crazycs520 force-pushed the crazycs520:adjust-add-index-worker branch from fcd092d to 50470ad Dec 21, 2018

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Dec 21, 2018

/run-all-tests

1 similar comment
@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Dec 21, 2018

/run-all-tests

@crazycs520 crazycs520 force-pushed the crazycs520:adjust-add-index-worker branch from a2d1db9 to fc6674e Dec 21, 2018

@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Dec 21, 2018

/run-all-tests

Show resolved Hide resolved ddl/testutil/testutil.go

// gofail: var checkIndexWorkerNum bool
//if checkIndexWorkerNum {
// num := int(atomic.LoadInt32(&TestCheckWorkerNumber))

This comment has been minimized.

Copy link
@zimulala

zimulala Dec 22, 2018

Member

Maybe we don't need atomic.LoadInt3.

This comment has been minimized.

Copy link
@crazycs520

crazycs520 Dec 22, 2018

Author Contributor

Use to avoid data race.

Show resolved Hide resolved ddl/index.go Outdated
@crazycs520

This comment has been minimized.

Copy link
Contributor Author

crazycs520 commented Dec 24, 2018

/run-all-tests

@crazycs520 crazycs520 merged commit d20bf28 into pingcap:master Dec 24, 2018

12 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/common-test job succeeded
Details
idc-jenkins-ci-tidb/integration-common-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-compatibility-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-ddl-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/mybatis-test job succeeded
Details
idc-jenkins-ci-tidb/sqllogic-test-1 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/sqllogic-test-2 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/unit-test Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details

@crazycs520 crazycs520 deleted the crazycs520:adjust-add-index-worker branch Dec 24, 2018

crazycs520 added a commit to crazycs520/tidb that referenced this pull request Dec 24, 2018

yu34po added a commit to yu34po/tidb that referenced this pull request Jan 2, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.