Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(dr-autosync) v6.5.8 scale in one node from primary datacenter failed during async mode #16465

Closed
mayjiang0203 opened this issue Jan 30, 2024 · 2 comments · Fixed by #16467 or #16486
Closed

(dr-autosync) v6.5.8 scale in one node from primary datacenter failed during async mode #16465

mayjiang0203 opened this issue Jan 30, 2024 · 2 comments · Fixed by #16467 or #16486
Labels
affects-6.5 affects-7.1 affects-7.5 severity/major type/bug Type: Issue - Confirmed a bug

Comments

@mayjiang0203
Copy link

Bug Report

What version of TiKV are you using?

v6.5.8

What operating system and CPU are you using?

Steps to reproduce

  1. deploy one tidb cluster with 12 stores, and config it with 5 replicas in dr-auto-sync mode.
  2. down two tikvs in the backup center, make replication mode switch to async.
  3. Try to scale-in one node in the primary center.

What did you expect?

Scale in succeeded.

What did happened?

Scale in hung.

Didn't hit it before #16174.
@mayjiang0203 mayjiang0203 added the type/bug Type: Issue - Confirmed a bug label Jan 30, 2024
@mayjiang0203
Copy link
Author

/severity major

@tonyxuqqi
Copy link
Contributor

In this case, does the back-up site have two learners for each region?
If that's the case, it would be blocked by today's check as we likely run into 3 healthy peers and 3 unhealthy peers. And if the remove peer happens on the healthy peer, it would be blocked.
The argument here is that if the unhealthy peer is learner, then it should not be counted. Then we will have only 3 healthy peers plus one unhealthy learner. By demoting a healthy peer, we will still have 2 healthy voter which can achieve majority.

@tonyxuqqi tonyxuqqi mentioned this issue Jan 30, 2024
Merged
9 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jan 31, 2024
@tonyxuqqi
raftstore: improve the remove peer check (#16467)
b67dd09 
close #16465

improve the remove peer check. Only check when the updating role is voter

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
@ti-chi-bot ti-chi-bot mentioned this issue Jan 31, 2024
Merged
9 tasks
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Jan 31, 2024
@tonyxuqqi @ti-chi-bot
This is an automated cherry-pick of tikv#16467
69351a3 
close tikv#16465

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Jan 31, 2024
Closed
9 tasks
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Jan 31, 2024
@tonyxuqqi @ti-chi-bot
This is an automated cherry-pick of tikv#16467
b5f389a 
close tikv#16465

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Jan 31, 2024
Closed
9 tasks
ti-chi-bot bot pushed a commit that referenced this issue Jan 31, 2024
@ti-chi-bot @tonyxuqqi
raftstore: improve the remove peer check (#16467) (#16470)
13bbe32 
close #16465

improve the remove peer check. Only check when the updating role is voter

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>

Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com>
tonyxuqqi added a commit to ti-chi-bot/tikv that referenced this issue Jan 31, 2024
@tonyxuqqi
raftstore: improve the remove peer check (tikv#16467)
6c39048 
close tikv#16465

improve the remove peer check. Only check when the updating role is voter

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
tonyxuqqi added a commit to ti-chi-bot/tikv that referenced this issue Jan 31, 2024
@tonyxuqqi
raftstore: improve the remove peer check (tikv#16467)
cf96723 
close tikv#16465

improve the remove peer check. Only check when the updating role is voter

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
@v01dstar v01dstar mentioned this issue Jan 31, 2024
Merged
4 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 1, 2024
@v01dstar
raftstore: Improve readability (#16476)
5d190fa 
ref #16465

Improve readability

Signed-off-by: Yang Zhang <yang.zhang@pingcap.com>
@tonyxuqqi tonyxuqqi mentioned this issue Feb 1, 2024
Merged
9 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 1, 2024
@tonyxuqqi
raftstore: Revert remove peer check (#16479)
4653fc6 
close #16465

revert the remove peer check. Without this revert, the scale-in will be blocked when there're one slow or down peer in a region of the scale-in node.

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
@tonyxuqqi tonyxuqqi mentioned this issue Feb 1, 2024
Merged
9 tasks
RidRisR pushed a commit to RidRisR/tikv that referenced this issue Feb 2, 2024
@v01dstar @RidRisR
raftstore: Improve readability (tikv#16476)
ac56676 
ref tikv#16465

Improve readability

Signed-off-by: Yang Zhang <yang.zhang@pingcap.com>
Signed-off-by: RidRisR <79858083+RidRisR@users.noreply.github.com>
ti-chi-bot bot pushed a commit that referenced this issue Feb 2, 2024
@tonyxuqqi
raftstore: polish the availability check on conf change requests (#16486
04370e9 
)

close #16465

When calculating the impact of conf change, include all operations into considerations.

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
tonyxuqqi added a commit to ti-chi-bot/tikv that referenced this issue Feb 2, 2024
@tonyxuqqi
raftstore: polish the availability check on conf change requests (tik…
92f72a5 
…v#16486)

close tikv#16465

When calculating the impact of conf change, include all operations into considerations.

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Feb 2, 2024
@tonyxuqqi @ti-chi-bot
This is an automated cherry-pick of tikv#16486
37a9852 
close tikv#16465

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Feb 2, 2024
Merged
9 tasks
ti-chi-bot bot pushed a commit that referenced this issue Feb 6, 2024
@ti-chi-bot @tonyxuqqi
raftstore: polish the availability check on conf change requests (#16486
24fe68e 
) (#16489)

close #16465

When calculating the impact of conf change, include all operations into considerations.

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>

Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com>
@tonyxuqqi tonyxuqqi mentioned this issue Feb 20, 2024
Merged
5 tasks
dbsid pushed a commit to dbsid/tikv that referenced this issue Mar 24, 2024
@tonyxuqqi @dbsid
raftstore: improve the remove peer check (tikv#16467)
bef9884 
close tikv#16465

improve the remove peer check. Only check when the updating role is voter

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
Signed-off-by: dbsid <chenhuansheng@pingcap.com>
dbsid pushed a commit to dbsid/tikv that referenced this issue Mar 24, 2024
@v01dstar @dbsid
raftstore: Improve readability (tikv#16476)
9535b60 
ref tikv#16465

Improve readability

Signed-off-by: Yang Zhang <yang.zhang@pingcap.com>
Signed-off-by: dbsid <chenhuansheng@pingcap.com>
dbsid pushed a commit to dbsid/tikv that referenced this issue Mar 24, 2024
@tonyxuqqi @dbsid
raftstore: polish the availability check on conf change requests (tik…
6632298 
…v#16486)

close tikv#16465

When calculating the impact of conf change, include all operations into considerations.

Signed-off-by: tonyxuqqi <tonyxuqi@outlook.com>
Signed-off-by: dbsid <chenhuansheng@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 affects-7.1 affects-7.5 severity/major type/bug Type: Issue - Confirmed a bug
Projects
None yet
Milestone
No milestone
3 participants
@jebter @tonyxuqqi @mayjiang0203