Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: add InitChunkSize variable as chunk init capacity #8480

Merged
merged 4 commits into from Jan 7, 2019

Conversation

@lysu
Copy link
Member

commented Nov 27, 2018

What problem does this PR solve?

  • enable chunk grow

What is changed and how it works?

  • Add init_chunk_size = 32
  • change max_chunk_size to 1024
  • global and session cross-fields validation

Check List

Tests

Code changes

  • Add init_chunk_size = 32
  • change max_chunk_size to 1024

Side effects

  • Possible performance regression(distsql/join/agg/sort will use 1024 because need other pr to change them)

Related changes

  • Need to be included in the release note

This change is Reviewable

@lysu

This comment has been minimized.

Copy link
Member Author

commented Nov 27, 2018

/run-all-tests

@shenli

This comment has been minimized.

Copy link
Member

commented Nov 27, 2018

What's the benefit of enable chunk grow?

@lysu lysu changed the title distsql,executor: enable chunk-grow & refine distsql batch distsql,executor: enable chunk-grow & refine distsql batch ctrl Nov 28, 2018
@lysu

This comment has been minimized.

Copy link
Member Author

commented Nov 28, 2018

@shenli enable chunk grow will real enable chunk grow(previous doesn't), the main benefit is it will alloc small chunk for small query, and accelerate the first row return time for big query result.

@lysu

This comment has been minimized.

Copy link
Member Author

commented Nov 28, 2018

/run-all-tests

@zz-jason

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

@lysu any update?

@lysu

This comment has been minimized.

Copy link
Member Author

commented Dec 10, 2018

@zz-jason I will take time in this week, there are other high p thing under hands

@lysu

This comment has been minimized.

Copy link
Member Author

commented Dec 10, 2018

@zz-jason and there are much test case rely on order that need address

@zhouqiang-cl

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

/run-all-tests

executor/export_test.go Outdated Show resolved Hide resolved
@zhouqiang-cl

This comment has been minimized.

Copy link
Member

commented Dec 10, 2018

/run-all-tests

@lysu lysu force-pushed the lysu:dev-chunk-grow-distsql branch 3 times, most recently from e6754e2 to fd10bcf Dec 24, 2018
@lysu

This comment has been minimized.

Copy link
Member Author

commented Dec 29, 2018

@lysu lysu force-pushed the lysu:dev-chunk-grow-distsql branch from fd10bcf to 559e767 Dec 29, 2018
sessionctx/variable/session.go Outdated Show resolved Hide resolved
util/mock/context.go Outdated Show resolved Hide resolved
@lysu lysu force-pushed the lysu:dev-chunk-grow-distsql branch from 559e767 to 66ec92c Jan 2, 2019
Copy link
Member

left a comment

LGTM

executor/distsql.go Outdated Show resolved Hide resolved
distsql/select_result.go Outdated Show resolved Hide resolved
distsql/select_result.go Outdated Show resolved Hide resolved
distsql/stream.go Outdated Show resolved Hide resolved
@zz-jason zz-jason requested review from zz-jason, XuHuaiyu and winoros Jan 2, 2019
@lysu

This comment has been minimized.

Copy link
Member Author

commented Jan 3, 2019

/run-unit-test tidb-test=pr/711

@lysu

This comment has been minimized.

Copy link
Member Author

commented Jan 3, 2019

/run-unit-test tidb-test=pr/711 gofail=etcd-io

executor/aggregate_test.go Outdated Show resolved Hide resolved
sessionctx/variable/session.go Outdated Show resolved Hide resolved
sessionctx/variable/session.go Outdated Show resolved Hide resolved
session/session.go Outdated Show resolved Hide resolved
session/session.go Outdated Show resolved Hide resolved
session/session.go Outdated Show resolved Hide resolved
session/session.go Outdated Show resolved Hide resolved
@lysu lysu force-pushed the lysu:dev-chunk-grow-distsql branch from 835ca83 to 627242e Jan 4, 2019
@lysu

This comment has been minimized.

Copy link
Member Author

commented Jan 4, 2019

/run-all-tests tidb-test=pr/711

@codecov-io

This comment has been minimized.

Copy link

commented Jan 4, 2019

Codecov Report

❗️ No coverage uploaded for pull request base (master@243d611). Click here to learn what that means.
The diff coverage is 57.14%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master    #8480   +/-   ##
=========================================
  Coverage          ?   67.49%           
=========================================
  Files             ?      363           
  Lines             ?    75134           
  Branches          ?        0           
=========================================
  Hits              ?    50711           
  Misses            ?    19937           
  Partials          ?     4486
Impacted Files Coverage Δ
sessionctx/variable/sysvar.go 100% <ø> (ø)
table/tables/tables.go 52.81% <0%> (ø)
statistics/handle.go 83.43% <100%> (ø)
session/session.go 72.82% <100%> (ø)
executor/executor.go 66.76% <100%> (ø)
util/mock/context.go 21.11% <100%> (ø)
sessionctx/variable/session.go 30.84% <33.33%> (ø)
sessionctx/variable/varsutil.go 27.67% <50%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 243d611...a7dce62. Read the comment docs.

sessionctx/variable/varsutil.go Show resolved Hide resolved
sessionctx/variable/varsutil.go Show resolved Hide resolved
sessionctx/variable/varsutil.go Show resolved Hide resolved
session/session_test.go Outdated Show resolved Hide resolved
executor/join_test.go Outdated Show resolved Hide resolved
executor/aggregate_test.go Outdated Show resolved Hide resolved
executor/join_test.go Outdated Show resolved Hide resolved
@lysu

This comment has been minimized.

Copy link
Member Author

commented Jan 4, 2019

PTAL again if free @zz-jason @eurekaka @XuHuaiyu thx

and also tidb-test pingcap/tidb-test#711

@XuHuaiyu XuHuaiyu changed the title config: add InitChunkSize variable as chunk init capacity *: add InitChunkSize variable as chunk init capacity Jan 4, 2019
@lysu lysu force-pushed the lysu:dev-chunk-grow-distsql branch from ab23c54 to 2ea94f7 Jan 4, 2019
@lysu

This comment has been minimized.

Copy link
Member Author

commented Jan 4, 2019

@XuHuaiyu @eurekaka PTAL thx

@lysu

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2019

@XuHuaiyu @eurekaka PTAL thx

Copy link
Member

left a comment

LGTM

@zz-jason zz-jason added the status/LGT1 label Jan 5, 2019
@zz-jason zz-jason requested review from XuHuaiyu and eurekaka Jan 5, 2019
@zz-jason

This comment has been minimized.

Copy link
Member

commented Jan 5, 2019

/run-all-tests

@lysu

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2019

/run-all-tests tidb-test=pr/711

1 similar comment
@lysu

This comment has been minimized.

Copy link
Member Author

commented Jan 5, 2019

/run-all-tests tidb-test=pr/711

Copy link
Contributor

left a comment

LGTM

@eurekaka eurekaka added status/LGT2 and removed status/LGT1 labels Jan 7, 2019
eurekaka and others added 2 commits Jan 7, 2019
@jackysp jackysp merged commit 084aa5a into pingcap:master Jan 7, 2019
3 of 4 checks passed
3 of 4 checks passed
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
ci/circleci Your tests passed on CircleCI!
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.