pd: cancel call when refreshing client

[BusyJay]

This is a quick fix for possible stale heartbeat streaming call.

Streaming call won't report error when messages are dropped silently in network. However there are sync requests in pd client can detect this error by timeout. When the error is detected, client will be refreshed. At that time, streaming call created by old client should be canceled automatically. But this doesn't happen as expected, so this pr cancels it explicitly.

We need to add a test case to tested this with iptables.

The issue needs to be investaged further too. /cc tikv/grpc-rs#150

[siddontang]

can we use failpoint to reproduce it? E.g, force breaking the loop and take the receiver.

[overvenus]

Do we need to call cancel on raft client?
https://github.com/pingcap/tikv/blob/65f8525a45c916c55476cffc5882bd379bb59b27/src/server/raft_client.rs#L78-L96

[siddontang]

Do we need to call cancel on raft client?

I think we need. Maybe we should add keep timeout too, but @disksing meets some problems with it.

[BusyJay]

I don't think so. If client is refreshed in raft client, then the streaming call is dropped and recreated immediately.

[BusyJay]

E.g, force breaking the loop and take the receiver.

I don't get it. What loop to be break? And how and why to take the receiver?

[siddontang]

what I mean is not returning Ready and going here https://github.com/BusyJay/tikv/blob/709096a3ec8e0011b6ac4e434e89f3ca2d3a2278/src/pd/util.rs#L70

Is it OK?

[BusyJay]

I don't see any connections between what you mention and the bug trying to fix here. Actually the poll method is never called again until cancel is called.

[siddontang]

Got it.

[BusyJay]

PTAL

[siddontang]

LGTM

[overvenus]

lgtm

[BusyJay]

/run-integration-tests

[overvenus]

/rebuild

[BusyJay]

/run-integration-tests