Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dr-autosync] during sync_recover, workload report error: Region is unavailable #14975

Closed
mayjiang0203 opened this issue Jun 20, 2023 · 2 comments · Fixed by #15103
Closed

[dr-autosync] during sync_recover, workload report error: Region is unavailable #14975

mayjiang0203 opened this issue Jun 20, 2023 · 2 comments · Fixed by #15103
Assignees
@nolouch
Labels
affects-6.5 affects-7.1 severity/major type/bug Type: Issue - Confirmed a bug type/enhancement Type: Issue - Enhancement

Comments

@mayjiang0203
Copy link

Bug Report

What version of TiKV are you using?

v6.5.3

What operating system and CPU are you using?

Steps to reproduce

a. down backup dc and make cluster work in the async mode for more than 10h with workload running.
b. recover backup dc，restart workload hit error of "Region is unavailable"

What did you expect?

no error reported

What did happened?

[2023/06/20 10:51:08.959 +08:00] [INFO] [utils.go:55] ["current status is "] [curStatus=sync_recover]
java -cp .:lib/mysql-connector-java-5.1.49.jar batch_hy_zd "tiup-peer.e2e-dr-auto-sync-5r-pref-tps-1809605-1-603:3390" "root" "" 30 ecasdb 20
java.sql.SQLException: Region is unavailable
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3933)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3869)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2675)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2439)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1365)
at batch_hy_zd.getMax(batch_hy_zd.java:193)
at batch_hy_zd.main(batch_hy_zd.java:69)

test cluster id: 7246458230958436285
@mayjiang0203 mayjiang0203 changed the title [dr-autosync] during synv_recover, workload report error: Region is unavailable [dr-autosync] during sync_recover, workload report error: Region is unavailable Jun 20, 2023
@mayjiang0203
Copy link
Author

should set replication-mode.pause-region-split: true in async mode

@nolouch nolouch reopened this Jun 25, 2023
@nolouch nolouch mentioned this issue Jun 25, 2023
Closed
@nolouch
Copy link
Contributor

nolouch commented Jun 25, 2023

ref #7945

@nolouch nolouch mentioned this issue Jul 11, 2023
Merged
@nolouch nolouch added type/enhancement Type: Issue - Enhancement type/bug Type: Issue - Confirmed a bug severity/major labels Jul 11, 2023
ti-chi-bot bot added a commit that referenced this issue Jul 14, 2023
@nolouch @ti-chi-bot
raftstore: allow DrAutoSync to majority commit log during SyncRecover…
31a6299 
… phase (#15103)

close #14975

raftstore: do not enable group commit when replication mode is SyncRecover
- during DR mode, the SyncRecover state should be committed in majority mode. otherwise, the qps may drop

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
@ti-chi-bot ti-chi-bot mentioned this issue Jul 14, 2023
Merged
ti-chi-bot pushed a commit to ti-chi-bot/tikv that referenced this issue Jul 14, 2023
@nolouch @ti-chi-bot
This is an automated cherry-pick of tikv#15103
6411460 
close tikv#14975

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
@ti-chi-bot ti-chi-bot mentioned this issue Jul 14, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jul 18, 2023
@ti-chi-bot @nolouch
raftstore: allow DrAutoSync to majority commit log during SyncRecover…
1c1eec4 
… phase (#15103) (#15130)

close #14975

raftstore: do not enable group commit when replication mode is SyncRecover
- during DR mode, the SyncRecover state should be committed in majority mode. otherwise, the qps may drop

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: ShuNing <nolouch@gmail.com>
Co-authored-by: nolouch <nolouch@gmail.com>
@nolouch nolouch self-assigned this Jul 28, 2023
ti-chi-bot bot added a commit that referenced this issue Sep 1, 2023
@ti-chi-bot @nolouch @ti-chi-bot
raftstore: allow DrAutoSync to majority commit log during SyncRecover…
f11bf5c 
… phase (#15103) (#15131)

close #14975

raftstore: do not enable group commit when replication mode is SyncRecover
- during DR mode, the SyncRecover state should be committed in majority mode. otherwise, the qps may drop

Signed-off-by: nolouch <nolouch@gmail.com>

Co-authored-by: nolouch <nolouch@gmail.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nolouch nolouch

Labels
affects-6.5 affects-7.1 severity/major type/bug Type: Issue - Confirmed a bug type/enhancement Type: Issue - Enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

raftstore: allow DrAutoSync to majority commit log during SyncRecover phase nolouch/tikv
2 participants
@nolouch @mayjiang0203