Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: add split index region syntax #10203

Merged
merged 30 commits into from May 6, 2019

Conversation

Projects
None yet
4 participants
@crazycs520
Copy link
Contributor

commented Apr 19, 2019

What problem does this PR solve?

PR: #10138 support pre-split table region. But it will only split 1 region for per index. Then the index region may still be the hot region and the write bottleneck.

What is changed and how it works?

This PR support split index region by SQL. Syntax like below:

SPLIT TABLE table_name INDEX index_name BY (val1...), (val2...),(val3...) ...

example

split table t1 idx1 (100), (1000), (10000);   
//range is : 0~100, 100 ~1000, 1000~ 1000010000 ~ maxInt64

split table t1 idx2 ("a", "2000-01-01 00:00:01"),  ("b", "2019-04-17 14:26:19"),  ("c", "");  
// range is: minIndexValue ~ ("a", "2000-01-01 00:00:01"),
("a", "2000-01-01 00:00:01") ~ ("b", "2019-04-17 14:26:19"),
("b", "2019-04-17 14:26:19") ~ ("c", ""),
("c", "") ~ maxIndexValue.

Please merge #10138 firstly.
Related parser PR: pingcap/parser#297

Check List

Tests

  • Adding test.

Code changes

  • Has exported function/method change

Side effects

Related changes

crazycs520 added some commits Apr 10, 2019

@crazycs520 crazycs520 requested a review from nolouch Apr 19, 2019

crazycs520 added some commits Apr 25, 2019

crazycs520 added some commits Apr 25, 2019

@nolouch
Copy link
Member

left a comment

LGTM

regionIDs := make([]uint64, 0, len(e.valueLists))
index := tables.NewIndex(e.table.Meta().ID, e.table.Meta(), e.indexInfo)
for _, values := range e.valueLists {
idxKey, _, err := index.GenIndexKey(e.ctx.GetSessionVars().StmtCtx, values, 1, nil)

This comment has been minimized.

Copy link
@nolouch

nolouch Apr 25, 2019

Member

1 means min handle id?

This comment has been minimized.

Copy link
@crazycs520

crazycs520 Apr 26, 2019

Author Contributor

yes, Maybe 0 is better?

This comment has been minimized.

Copy link
@winkyao

winkyao Apr 29, 2019

Member

you can use a constant, better than hard code!

This comment has been minimized.

Copy link
@zimulala

zimulala Apr 29, 2019

Member

@winkyao
I think 1 isn't always the min handle. If a table has a PKIsHandle column, then the handle may be a negative value.
So I think we need to use math.MinInt64.

This comment has been minimized.

Copy link
@winkyao

winkyao May 6, 2019

Member

@zimulala The key is encoding in binary format, so negative value is always greater than a positive value. @nolouch Right?

This comment has been minimized.

Copy link
@crazycs520

crazycs520 May 6, 2019

Author Contributor

Here is a test:
0 encode to byte is: [128 0 0 0 0 0 0 0]
math.MinInt64 encode to byte is: [0 0 0 0 0 0 0 0]

so, math.MinInt64 is better.

Show resolved Hide resolved store/tikv/split_region.go Outdated
Show resolved Hide resolved planner/core/planbuilder.go Outdated
Show resolved Hide resolved planner/core/planbuilder.go Outdated

crazycs520 added some commits Apr 26, 2019

@codecov

This comment has been minimized.

Copy link

commented Apr 28, 2019

Codecov Report

Merging #10203 into master will decrease coverage by 0.0206%.
The diff coverage is 63.5135%.

@@               Coverage Diff                @@
##             master     #10203        +/-   ##
================================================
- Coverage   77.6772%   77.6566%   -0.0206%     
================================================
  Files           411        412         +1     
  Lines         85460      85511        +51     
================================================
+ Hits          66383      66405        +22     
- Misses        14119      14141        +22     
- Partials       4958       4965         +7
for j, valueItem := range valuesItem {
x, ok := valueItem.(*driver.ValueExpr)
if !ok {
return nil, errors.Trace(errors.New("expect constant values"))

This comment has been minimized.

Copy link
@zimulala

zimulala Apr 29, 2019

Member

Could we remove this trace? Other places don't use it.
And we can print the value to the error message.

This comment has been minimized.

Copy link
@crazycs520

crazycs520 May 6, 2019

Author Contributor

done.

regionIDs := make([]uint64, 0, len(e.valueLists))
index := tables.NewIndex(e.table.Meta().ID, e.table.Meta(), e.indexInfo)
for _, values := range e.valueLists {
idxKey, _, err := index.GenIndexKey(e.ctx.GetSessionVars().StmtCtx, values, 1, nil)

This comment has been minimized.

Copy link
@zimulala

zimulala Apr 29, 2019

Member

@winkyao
I think 1 isn't always the min handle. If a table has a PKIsHandle column, then the handle may be a negative value.
So I think we need to use math.MinInt64.

crazycs520 added some commits May 6, 2019

Show resolved Hide resolved planner/core/planbuilder.go Outdated
Show resolved Hide resolved planner/core/planbuilder.go Outdated
@winkyao
Copy link
Member

left a comment

LGTM

Show resolved Hide resolved executor/split.go
err := s.WaitScatterRegionFinish(regionID)
if err != nil {
logutil.Logger(context.Background()).Warn("wait scatter region failed",
zap.Uint64("regionID", regionID),

This comment has been minimized.

Copy link
@zimulala

zimulala May 6, 2019

Member

Do we need to add the table and index information to this log?

This comment has been minimized.

Copy link
@crazycs520

crazycs520 May 6, 2019

Author Contributor

done.

@zimulala zimulala added the status/LGT1 label May 6, 2019

Show resolved Hide resolved go.mod Outdated
@crazycs520

This comment has been minimized.

Copy link
Contributor Author

commented May 6, 2019

/run-all-tests

@zimulala
Copy link
Member

left a comment

LGTM

@zimulala zimulala added status/LGT2 and removed status/LGT1 labels May 6, 2019

@crazycs520 crazycs520 merged commit 7ecb315 into pingcap:master May 6, 2019

16 checks passed

ci/circleci Your tests passed on CircleCI!
Details
codecov/patch 63.5135% of diff hit (target 0%)
Details
codecov/project 77.6566% (-0.0207%) compared to 0106361
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/build_check_race Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/check_dev Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/check_dev_2 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/common-test job succeeded
Details
idc-jenkins-ci-tidb/integration-common-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-compatibility-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-ddl-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/mybatis-test job succeeded
Details
idc-jenkins-ci-tidb/sqllogic-test-1 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/sqllogic-test-2 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/unit-test Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details

benpigchu added a commit to benpigchu/tidb that referenced this pull request May 7, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.