TiKV OOM causes by reading raft conf changes during raft.hup #15770
Labels
affects-5.0
This bug affects 5.0.x versions.
affects-5.1
This bug affects 5.1.x versions.
affects-5.2
This bug affects 5.2.x versions.
affects-5.3
This bug affects 5.3.x versions.
affects-5.4
affects-6.0
affects-6.1
affects-6.2
affects-6.3
affects-6.4
affects-6.5
affects-6.6
affects-7.0
affects-7.1
affects-7.2
affects-7.3
affects-7.4
affects-7.5
severity/major
type/bug
Type: Issue - Confirmed a bug
Comments
overvenus
added
type/bug
This was referenced Oct 16, 2023
ti-chi-bot bot
added a commit
that referenced
this issue
Oct 23, 2023
…#15806) close #15770 Before start election, raft-rs has to check if there is any unapplied conf change entry. In the current implementation, this needs to scan logs from [unapplied_index, committed_index]. It essentially takes unbounded memory when raft peers that has many unapplied logs. To fix the issue, TiKV can paginate scan raft log which has a fixed memory usage upper bound. Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot
pushed a commit
to ti-chi-bot/tikv
that referenced
this issue
Oct 23, 2023
close tikv#15770 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
pushed a commit
to ti-chi-bot/tikv
that referenced
this issue
Oct 23, 2023
close tikv#15770 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
pushed a commit
to ti-chi-bot/tikv
that referenced
this issue
Oct 23, 2023
close tikv#15770 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
ti-chi-bot
pushed a commit
to ti-chi-bot/tikv
that referenced
this issue
Oct 23, 2023
close tikv#15770 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
This was referenced Oct 23, 2023
ti-chi-bot
pushed a commit
to ti-chi-bot/tikv
that referenced
this issue
Oct 23, 2023
close tikv#15770 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Smityz
pushed a commit
to Smityz/tikv
that referenced
this issue
Oct 23, 2023
…tikv#15806) close tikv#15770 Before start election, raft-rs has to check if there is any unapplied conf change entry. In the current implementation, this needs to scan logs from [unapplied_index, committed_index]. It essentially takes unbounded memory when raft peers that has many unapplied logs. To fix the issue, TiKV can paginate scan raft log which has a fixed memory usage upper bound. Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
ti-chi-bot bot
pushed a commit
that referenced
this issue
Nov 17, 2023
…#15806) (#15814) close #15770 Before start election, raft-rs has to check if there is any unapplied conf change entry. In the current implementation, this needs to scan logs from [unapplied_index, committed_index]. It essentially takes unbounded memory when raft peers that has many unapplied logs. To fix the issue, TiKV can paginate scan raft log which has a fixed memory usage upper bound. Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com>
ti-chi-bot bot
added a commit
that referenced
this issue
Nov 17, 2023
…#15806) (#15812) close #15770 Before start election, raft-rs has to check if there is any unapplied conf change entry. In the current implementation, this needs to scan logs from [unapplied_index, committed_index]. It essentially takes unbounded memory when raft peers that has many unapplied logs. To fix the issue, TiKV can paginate scan raft log which has a fixed memory usage upper bound. Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
overvenus
added a commit
to ti-chi-bot/tikv
that referenced
this issue
Nov 17, 2023
…tikv#15806) close tikv#15770 Before start election, raft-rs has to check if there is any unapplied conf change entry. In the current implementation, this needs to scan logs from [unapplied_index, committed_index]. It essentially takes unbounded memory when raft peers that has many unapplied logs. To fix the issue, TiKV can paginate scan raft log which has a fixed memory usage upper bound. Signed-off-by: Neil Shen <overvenus@gmail.com>
ti-chi-bot bot
added a commit
that referenced
this issue
Nov 17, 2023
…#15806) (#15813) close #15770 Before start election, raft-rs has to check if there is any unapplied conf change entry. In the current implementation, this needs to scan logs from [unapplied_index, committed_index]. It essentially takes unbounded memory when raft peers that has many unapplied logs. To fix the issue, TiKV can paginate scan raft log which has a fixed memory usage upper bound. Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
15 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug Report
Before start election, raft-rs has to check if there is any unapplied conf change
entry. In the current implementation, this needs to scan logs from
[unapplied_index, committed_index]. It essentially takes unbounded memory when
raft peers that has many unapplied logs.
To fix the issue, TiKV can paginate scan raft log which has a fixed memory usage
upper bound.
Results below:
TiKV 3 OOM immediately after restart, until rolling update to a paginate-scan TiKV.
What version of TiKV are you using?
v7.4.0
Steps to reproduce
The case occurs during sysbench bulk insert, but I'm not sure if it can reproduce reliably.
What did you expect?
No OOM
What did happened?
OOM
The text was updated successfully, but these errors were encountered: