Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

raftstore: retry pending read index requests #6348

Merged
merged 49 commits into from Jan 10, 2020
Merged

Conversation

@hicqu
Copy link
Contributor

hicqu commented Dec 27, 2019

Signed-off-by: qupeng qupeng@pingcap.com

What have you changed?

On follwer replicas pending read index requests will be retried. So that read commands won't be blocked forever in raftstore.

What is the type of the changes?

Bugfix.

How is the PR tested?

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

No.

Does this PR affect tidb-ansible?

No.

Refer to a related PR or issue link (optional)

Benchmark result if necessary (optional)

Any examples? (optional)

Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu

This comment has been minimized.

Copy link
Contributor Author

hicqu commented Dec 27, 2019

/run-all-tests

Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu

This comment has been minimized.

Copy link
Contributor Author

hicqu commented Dec 27, 2019

/run-all-tests

1 similar comment
@hicqu

This comment has been minimized.

Copy link
Contributor Author

hicqu commented Dec 27, 2019

/run-all-tests

src/raftstore/store/peer.rs Outdated Show resolved Hide resolved
hicqu added 2 commits Dec 30, 2019
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu

This comment has been minimized.

Copy link
Contributor Author

hicqu commented Dec 30, 2019

/run-all-tests

Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu hicqu force-pushed the hicqu:read-index-retry branch from e156a04 to 41ea574 Dec 30, 2019
@hicqu

This comment has been minimized.

Copy link
Contributor Author

hicqu commented Dec 30, 2019

/run-all-tests

Copy link
Contributor

5kbpers left a comment

LGTM

hicqu added 4 commits Dec 31, 2019
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu

This comment has been minimized.

Copy link
Contributor Author

hicqu commented Dec 31, 2019

/run-all-tests

@hicqu

This comment has been minimized.

Copy link
Contributor Author

hicqu commented Jan 2, 2020

/release

hicqu added 2 commits Jan 2, 2020
Signed-off-by: qupeng <qupeng@pingcap.com>
hicqu added 3 commits Jan 8, 2020
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu

This comment has been minimized.

Copy link
Contributor Author

hicqu commented Jan 9, 2020

/run-all-tests

@hicqu hicqu added this to the v4.0.0-beta milestone Jan 9, 2020
hicqu added 4 commits Jan 9, 2020
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
@@ -1871,6 +1871,25 @@ impl Peer {
Ok(())
}

/// `ReadIndex` requests could be lost in network, so on followers commands could queue in
/// `pending_reads` forever. Sending a new `ReadIndex` periodically can resolve this.
pub(super) fn retry_pending_reads(&mut self) {

This comment has been minimized.

Copy link
@BusyJay

BusyJay Jan 9, 2020

Contributor

How about put the if check here? And I think pub is enough.

}
}

pub fn check_needs_retry(&mut self) -> bool {

This comment has been minimized.

Copy link
@BusyJay

BusyJay Jan 9, 2020

Contributor

Why not pass configuration here instead? So that only one field is necessary.

This comment has been minimized.

Copy link
@hicqu

hicqu Jan 9, 2020

Author Contributor

So push_back will also needs the configuration, which makes it more complex.

return false;
}

self.last_retried = total + self.handled_cnt - 1;

This comment has been minimized.

Copy link
@BusyJay

BusyJay Jan 9, 2020

Contributor

Why minus one?

This comment has been minimized.

Copy link
@hicqu

hicqu Jan 9, 2020

Author Contributor

So last_retried is an index, not a length.

}
}

pub fn check_needs_retry(&mut self) -> bool {

This comment has been minimized.

Copy link
@BusyJay

BusyJay Jan 9, 2020

Contributor

Should add some comment explaining the retry strategy.

hicqu added 4 commits Jan 9, 2020
Signed-off-by: qupeng <qupeng@pingcap.com>
Signed-off-by: qupeng <qupeng@pingcap.com>
@@ -191,7 +191,7 @@ pub struct Peer {

leader_missing_time: Option<Instant>,
leader_lease: Lease,
pending_reads: ReadIndexQueue,
pub pending_reads: ReadIndexQueue,

This comment has been minimized.

Copy link
@BusyJay

BusyJay Jan 9, 2020

Contributor

Unnecessary public.

}

let read = self.pending_reads.back_mut().unwrap();
if read.read_index.is_none() {

This comment has been minimized.

Copy link
@BusyJay

BusyJay Jan 9, 2020

Contributor

It must be none.

}

if self.retry_countdown == usize::MAX {
self.retry_countdown = cfg.raft_election_timeout_ticks.checked_sub(1).unwrap();

This comment has been minimized.

Copy link
@BusyJay

BusyJay Jan 9, 2020

Contributor

Election timeout tick has to be greater than 0. count down can be zero after minusing one.

hicqu added 2 commits Jan 9, 2020
Signed-off-by: qupeng <qupeng@pingcap.com>
@BusyJay
BusyJay approved these changes Jan 9, 2020
@BusyJay

This comment has been minimized.

Copy link
Contributor

BusyJay commented Jan 9, 2020

@5kbpers PTAL, I think it's completely different from what you has reviewed.

@BusyJay BusyJay dismissed 5kbpers’s stale review Jan 9, 2020

I think it's completely different from what you has reviewed.

Copy link
Contributor

5kbpers left a comment

LGTM

@hicqu

This comment has been minimized.

Copy link
Contributor Author

hicqu commented Jan 10, 2020

/run-all-tests

Signed-off-by: qupeng <qupeng@pingcap.com>
@hicqu hicqu merged commit 76866ae into tikv:master Jan 10, 2020
3 checks passed
3 checks passed
DCO All commits are signed off!
Details
idc-jenkins-ci-tikv/integration-common-test Jenkins job succeeded.
Details
idc-jenkins-ci/test Jenkins job succeeded.
Details
@hicqu hicqu deleted the hicqu:read-index-retry branch Jan 10, 2020
@sre-bot

This comment has been minimized.

Copy link
Collaborator

sre-bot commented Jan 10, 2020

cherry pick to release-3.1 failed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.