New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: Speed up the add index operation #4579

Merged
merged 16 commits into from Sep 22, 2017

Conversation

Projects
None yet
6 participants
@zimulala
Member

zimulala commented Sep 19, 2017

Using 4 TiKV, 1 PD, 2 TiDB , the servers of TiKV and TiDB are not on a computer.
The number of records in this table is 100,000,000, and the table's structure is as follows:
CREATE TABLE tt (
a int(11) DEFAULT NULL,
b varchar(256) DEFAULT NULL,
c varchar(256) DEFAULT NULL,
d int(11) DEFAULT NULL,
e int(11) DEFAULT NULL,
f int(11) DEFAULT NULL)ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin

tidb-before: adding an index e takes time 44m33.015285749s
tidb-current: adding an index f takes time 35m33.008533773s

@zimulala

This comment has been minimized.

Show comment
Hide comment
@zimulala

zimulala Sep 19, 2017

Member

/run-all-tests

Member

zimulala commented Sep 19, 2017

/run-all-tests

@zimulala

This comment has been minimized.

Show comment
Hide comment
@zimulala
Member

zimulala commented Sep 20, 2017

Show outdated Hide outdated ddl/index.go
@zimulala

This comment has been minimized.

Show comment
Hide comment
@zimulala
Member

zimulala commented Sep 21, 2017

PTAL @coocood

Show outdated Hide outdated ddl/index.go
Show outdated Hide outdated ddl/index.go
Show outdated Hide outdated ddl/index.go
}
idxRecord.vals = idxVal
return nil
}
const (
defaultBatchCnt = 1024
defaultSmallBatchCnt = 128
defaultTaskHandleCnt = 128

This comment has been minimized.

@coocood

coocood Sep 21, 2017

Member

We use Seek to fetch handles and Seek is implement by Scan in TiKV, which has 256 batch size, change it to 256 may improve performance.

@coocood

coocood Sep 21, 2017

Member

We use Seek to fetch handles and Seek is implement by Scan in TiKV, which has 256 batch size, change it to 256 may improve performance.

@@ -383,16 +385,18 @@ LOOP:
for i := num; i < num+step; i++ {
n := rand.Intn(num)
deletedKeys[n] = struct{}{}
s.mustExec(c, "delete from t1 where c1 = ?", n)
s.mustExec(c, "insert into t1 values (?, ?, ?)", i, i, i)
sql := fmt.Sprintf("delete from t1 where c1 = %d", n)

This comment has been minimized.

@coocood

coocood Sep 21, 2017

Member

Why change the test?

@coocood

coocood Sep 21, 2017

Member

Why change the test?

This comment has been minimized.

@zimulala

zimulala Sep 22, 2017

Member

If the test is failed, we can have more detail logs.

@zimulala

zimulala Sep 22, 2017

Member

If the test is failed, we can have more detail logs.

Show outdated Hide outdated tablecodec/tablecodec.go
@coocood

This comment has been minimized.

Show comment
Hide comment
@coocood

coocood Sep 22, 2017

Member

LGTM

Member

coocood commented Sep 22, 2017

LGTM

Show outdated Hide outdated ddl/index.go

zimulala added some commits Sep 22, 2017

@zimulala zimulala added status/LGT2 and removed status/LGT1 labels Sep 22, 2017

@zimulala

This comment has been minimized.

Show comment
Hide comment
@zimulala

zimulala Sep 22, 2017

Member

/run-all-tests

Member

zimulala commented Sep 22, 2017

/run-all-tests

@zimulala

This comment has been minimized.

Show comment
Hide comment
@zimulala

zimulala Sep 22, 2017

Member

/run-unit-test

Member

zimulala commented Sep 22, 2017

/run-unit-test

@hanfei1991 hanfei1991 merged commit a764561 into master Sep 22, 2017

5 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls First build on master at 72.497%
Details
jenkins-ci-tidb/build Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details

@hanfei1991 hanfei1991 deleted the zimuxia/speedup-idx-1 branch Sep 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment