Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tikvclient: fix a bug that double close channels. #10991

Merged
merged 31 commits into from Jul 5, 2019

Conversation

@hicqu
Copy link
Contributor

commented Jun 30, 2019

Signed-off-by: qupeng qupeng@pingcap.com

What problem does this PR solve?

There is a bug in tikvclient module about double closing channels, which causes some threads panic when holding a lock. Although these threads can recover, they will be blocked on that lock so they can't work finally.

What is changed and how it works?

This PR fixed the channel double closing problem, and use defer to make the lock logic more clear.

Check List

Tests

  • It's hard to add unit tests. We need to cover the case in our tikv client test framework.

Related changes

  • Need to cherry-pick to the release branch
tikvclient: fix a bug that double close channels.
Signed-off-by: qupeng <qupeng@pingcap.com>

@hicqu hicqu requested review from lysu and zz-jason Jun 30, 2019

@hicqu hicqu added the type/bug-fix label Jun 30, 2019

@zz-jason
Copy link
Member

left a comment

please add an unit test

@hicqu

This comment has been minimized.

Copy link
Contributor Author

commented Jun 30, 2019

@zz-jason I guess you didn't read the initial comment. Unit tests can't work for these cases because we need some real TiKV services to run tikvclient code. Currently we test tikvclient.go in 2 schrodinger test cases, we can add more cases there instead of here.

@siddontang

This comment has been minimized.

Copy link
Member

commented Jun 30, 2019

you can use faketikv

seem we also need to care client-go , @disksing

@lysu lysu requested a review from tiancaiamao Jul 1, 2019

Show resolved Hide resolved store/tikv/client.go Outdated
@lysu

This comment has been minimized.

Copy link
Member

commented Jul 1, 2019

almost lgtm, but have some question..

can we do not call failPendingRequests in sendLoop for errors execept io.EOF?

it seems when sendMsg fail will finish csAttempt https://github.com/grpc/grpc-go/blob/master/stream.go#L670

so it will trigger retry or http2Client#closeStream(will write error to recvBuffer), so it seems recvMsg always got error - -?

..... but maybe do it in both side will more safe

@hicqu

This comment has been minimized.

Copy link
Contributor Author

commented Jul 3, 2019

@lysu I think don't depend gRPC's internal behaviors is better. So let's call failPendingRequests in a locked context.

@codecov

This comment has been minimized.

Copy link

commented Jul 3, 2019

Codecov Report

Merging #10991 into master will decrease coverage by 0.4708%.
The diff coverage is 72.2222%.

@@               Coverage Diff                @@
##             master     #10991        +/-   ##
================================================
- Coverage   81.5977%   81.1269%   -0.4709%     
================================================
  Files           420        420                
  Lines         90902      89466      -1436     
================================================
- Hits          74174      72581      -1593     
- Misses        11421      11630       +209     
+ Partials       5307       5255        -52

hicqu added some commits Jul 3, 2019

Show resolved Hide resolved store/tikv/client_fail_test.go Outdated
Show resolved Hide resolved store/tikv/mock_tikv_service.go Outdated

hicqu added some commits Jul 3, 2019

@hicqu

This comment has been minimized.

Copy link
Contributor Author

commented Jul 3, 2019

PTAL @lysu @zz-jason @lonng thanks!

Show resolved Hide resolved store/tikv/client.go Outdated

hicqu added some commits Jul 3, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

commented Jul 4, 2019

Please fix CI @hicqu

hicqu added some commits Jul 4, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2019

/run-all-tests

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2019

/run-all-tests

hicqu and others added some commits Jul 5, 2019

@tiancaiamao

This comment has been minimized.

Copy link
Contributor

commented Jul 5, 2019

/run-all-tests

@tiancaiamao tiancaiamao merged commit 21d2590 into pingcap:master Jul 5, 2019

14 checks passed

ci/circleci Your tests passed on CircleCI!
Details
idc-jenkins-ci-tidb/build Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/build_check_race Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/check_dev Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/check_dev_2 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/common-test job succeeded
Details
idc-jenkins-ci-tidb/integration-common-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-compatibility-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/integration-ddl-test Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/mybatis-test job succeeded
Details
idc-jenkins-ci-tidb/sqllogic-test-1 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/sqllogic-test-2 Jenkins job succeeded.
Details
idc-jenkins-ci-tidb/unit-test Jenkins job succeeded.
Details
license/cla Contributor License Agreement is signed.
Details

@hicqu hicqu deleted the hicqu:fix-tikvclient-double-close branch Jul 5, 2019

hicqu added a commit to hicqu/tidb that referenced this pull request Jul 5, 2019

hicqu added a commit to hicqu/tidb that referenced this pull request Jul 5, 2019

hicqu added a commit to hicqu/tidb that referenced this pull request Jul 5, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.