Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

executor: trace and control memory usage in DistSQL layer #10003

Merged
merged 17 commits into from Apr 12, 2019

Conversation

Projects
None yet
4 participants
@qw4990
Copy link
Contributor

qw4990 commented Apr 2, 2019

What problem does this PR solve?

TiDB only traces and controls memory used in Executor layer now.
Some operators in DistSQL layer like selectResult and copIterator also have their own buffer that may use a lot of memory.
Memory usage in these operators should be traced and controlled by MemTracker.

What is changed and how it works?

Introduce a new field MemTracker in kv.Request to let operators in DistSQL layer use to trace and control memory usage.

For simplification, this PR only support tracing DistSQL operators created by TableReader, IndexReader and IndexLookup.
DistSQL operators created by Analyze will be supported by an another PR.

Check List

Tests

  • Unit test
@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 2, 2019

/rebuild

@qw4990 qw4990 force-pushed the qw4990:mem_trace branch from d663c4d to 57ffc8c Apr 2, 2019

Show resolved Hide resolved sessionctx/variable/session.go Outdated
Show resolved Hide resolved kv/kv.go Outdated
Show resolved Hide resolved kv/kv.go Outdated
Show resolved Hide resolved kv/kv.go Outdated
Show resolved Hide resolved distsql/request_builder.go Outdated
Show resolved Hide resolved store/tikv/coprocessor.go
Show resolved Hide resolved store/tikv/coprocessor.go Outdated
@codecov

This comment has been minimized.

Copy link

codecov bot commented Apr 3, 2019

Codecov Report

Merging #10003 into master will increase coverage by 0.0361%.
The diff coverage is 93.3333%.

@@               Coverage Diff               @@
##            master     #10003        +/-   ##
===============================================
+ Coverage   77.489%   77.5252%   +0.0361%     
===============================================
  Files          403        403                
  Lines        81898      81954        +56     
===============================================
+ Hits         63462      63535        +73     
+ Misses       13723      13706        -17     
  Partials      4713       4713
@lamxTyler
Copy link
Member

lamxTyler left a comment

LGTM

Show resolved Hide resolved distsql/distsql_test.go Outdated
Show resolved Hide resolved kv/kv.go Outdated
Show resolved Hide resolved store/tikv/coprocessor.go Outdated
Show resolved Hide resolved store/tikv/coprocessor.go Outdated
Show resolved Hide resolved distsql/select_result.go

@qw4990 qw4990 force-pushed the qw4990:mem_trace branch from 6456527 to 3e1cc4c Apr 11, 2019

@zz-jason
Copy link
Member

zz-jason left a comment

LGTM

@zz-jason zz-jason added status/LGT2 and removed status/LGT1 labels Apr 11, 2019

@zz-jason

This comment has been minimized.

Copy link
Member

zz-jason commented Apr 11, 2019

/run-all-tests

1 similar comment
@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 11, 2019

/run-all-tests

@zhouqiang-cl

This comment has been minimized.

Copy link
Member

zhouqiang-cl commented Apr 11, 2019

/rebuild

@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 11, 2019

/run-all-tests

2 similar comments
@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 11, 2019

/run-all-tests

@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 11, 2019

/run-all-tests

@qw4990 qw4990 force-pushed the qw4990:mem_trace branch from b7a4d41 to 0edd284 Apr 11, 2019

@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 11, 2019

/run-all-tests

1 similar comment
@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 11, 2019

/run-all-tests

@qw4990 qw4990 force-pushed the qw4990:mem_trace branch from 0edd284 to 23f3c48 Apr 12, 2019

@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 12, 2019

/run-all-tests

@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 12, 2019

/run-all-tests

@qw4990

This comment has been minimized.

Copy link
Contributor Author

qw4990 commented Apr 12, 2019

All CI problems have been fixed, PTAL~

@qw4990 qw4990 merged commit 7447f04 into pingcap:master Apr 12, 2019

6 checks passed

ci/circleci Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/check_dev Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/code_coverage Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.