raft: Make flow control more aggressive #9985
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## master #9985 +/- ##
==========================================
- Coverage 69.34% 69.11% -0.23%
==========================================
Files 386 386
Lines 35914 35918 +4
==========================================
- Hits 24905 24826 -79
- Misses 9212 9297 +85
+ Partials 1797 1795 -2
Continue to review full report at Codecov.
|
xiang90
reviewed
Aug 7, 2018
raft/raft.go
Outdated
| // argument controls whether messages with no entries will be sent | ||
| // ("empty" messages are useful to convey updated Commit indexes, but | ||
| // are undesirable when we're sending multiple messages in a batch). | ||
| func (r *raft) sendAppend(to uint64, sendIfEmpty bool) bool { |
There was a problem hiding this comment.
can we keep the old sendAppend? and add a new func maybeSendAppend (or sendAppendIfNotEmpty)?
|
@bdarnell Have you tried this in CockroachDB setup? |
|
LGTM after fixing the nit. |
We allow multiple in-flight append messages, but prior to this change the only way we'd ever send them is if there is a steady stream of new proposals. Catching up a follower that is far behind would be unnecessarily slow (this is exacerbated by a quirk of CockroachDB's use of raft which limits our ability to catch up via snapshot in some cases). See cockroachdb/cockroach#27983
|
Yes, we've tested it. We had a node that had accumulated a multi-gigabyte raft log. Without this change it was going to take weeks to catch up, and with this it caught up in hours. |
bdarnell
added a commit
to cockroachdb/vendored
that referenced
this pull request
Aug 13, 2018
Picks up etcd-io/etcd#9982 and etcd-io/etcd#9985 (and no other changes to packages we use).
bdarnell
added a commit
to bdarnell/cockroach
that referenced
this pull request
Aug 13, 2018
Picks up etcd-io/etcd#9982 and etcd-io/etcd#9985 (and no other changes to packages we use). Fixes cockroachdb#27983 Fixes cockroachdb#27804 Release note (bug fix): Additional fixes for out-of-memory errors caused by very large raft logs. Release note (performance improvement): Greatly improved performance when catching up followers that are behind when raft logs are large.
craig bot
pushed a commit
to cockroachdb/cockroach
that referenced
this pull request
Aug 13, 2018
28511: vendor: Update etcd r=tschottdorf a=bdarnell Picks up etcd-io/etcd#9982 and etcd-io/etcd#9985 (and no other changes to packages we use). Fixes #27983 Fixes #27804 Release note (bug fix): Additional fixes for out-of-memory errors caused by very large raft logs. Release note (performance improvement): Greatly improved performance when catching up followers that are behind when raft logs are large. Co-authored-by: Ben Darnell <ben@bendarnell.com>
Hi @benbjohnson Why don't you use snapshot here? seem your Range size is 64 MB, the total log size has exceeded this threshold too much. |
BusyJay
added a commit
to BusyJay/raft-rs
that referenced
this pull request
Feb 27, 2020
Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
BusyJay
added a commit
to tikv/raft-rs
that referenced
this pull request
Mar 3, 2020
BusyJay
added a commit
to BusyJay/raft-rs
that referenced
this pull request
Mar 26, 2020
The patch is to speed up log replication when a node is way behind than leader and logs are not compacted yet. Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
gengliqi
pushed a commit
to tikv/raft-rs
that referenced
this pull request
Mar 27, 2020
shbieng
pushed a commit
to shbieng/raft-rs
that referenced
this pull request
Sep 19, 2022
The patch is to speed up log replication when a node is way behind than leader and logs are not compacted yet. Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
siennathesane
pushed a commit
to shieldmaidens/pleiades
that referenced
this pull request
Nov 4, 2023
The patch is to speed up log replication when a node is way behind than leader and logs are not compacted yet. Signed-off-by: Jay Lee <BusyJayLee@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We allow multiple in-flight append messages, but prior to this change
the only way we'd ever send them is if there is a steady stream of new
proposals. Catching up a follower that is far behind would be
unnecessarily slow (this is exacerbated by a quirk of CockroachDB's
use of raft which limits our ability to catch up via snapshot in some
cases).
See cockroachdb/cockroach#27983
Please read https://github.com/coreos/etcd/blob/master/CONTRIBUTING.md#contribution-flow.