Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dr-autosync] too much logs during online recovery after switching to back dc in sync_recovery mode #15112

Closed
mayjiang0203 opened this issue Jul 13, 2023 · 6 comments · Fixed by #15119
Closed

[dr-autosync] too much logs during online recovery after switching to back dc in sync_recovery mode #15112

mayjiang0203 opened this issue Jul 13, 2023 · 6 comments · Fixed by #15119
Assignees
@v01dstar
Labels
affects-7.1 type/bug Type: Issue - Confirmed a bug

Comments

@mayjiang0203
Copy link

Bug Report

What version of TiKV are you using?

6.5.3

What operating system and CPU are you using?

Steps to reproduce

What did you expect?

What did happened?

[2023/07/11 18:21:44.936 +08:00] [INFO] [cluster.go:384] ["will run cmd"] [cmd:="tiup ctl:v6.5.3 pd -u http://pd3-peer.e2e-dr-auto-sync-func-thin-tps-1818031-1-434:2379 unsafe remove-failed-stores --auto-detect"]

....
  {
    "info": "Unsafe recovery finished",
    "time": "2023-07-11 18:23:34.459",
    "details": [
      "affected table ids: 10, 72, 281474976710654, 34, 48, 88, 46, 82, 89, 281474976710652, 281474976710653, 26, 76, 104, 8, 38, 52, 78, 83, 22, 44, 60, 64, 66, 68, 85, 4, 24, 36, 84, 6, 42, 30, 40, 87, 100, 102, 32, 50, 54, 58, 62, 108, 96, 98, 106, 20, 12, 56, 70, 18, 28, 94, 281474976710651, 14, 16, 74, 86, 90"
    ]
  }
]

sh-4.2# ls -lrh
total 522M
-rw-r--r--. 1 tidb tidb  466 Jul 11 18:25 tikv_stderr.log
-rw-r--r--. 1 tidb tidb 8.8M Jul 12 15:16 tikv.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-50.912.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-40.594.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-33.361.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-29.736.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-26.884.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-23.517.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-19.517.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-15.533.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-11.542.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-07.535.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:23 tikv-2023-07-11T18-23-03.505.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-59.481.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-55.478.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-51.478.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-47.494.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-43.484.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-39.443.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-35.412.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-31.420.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-27.419.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-23.386.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-19.353.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-15.291.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-11.215.log
-rw-r--r--. 1 tidb tidb  11M Jul 11 18:22 tikv-2023-07-11T18-22-06.588.log
-rw-r--r--. 1 tidb tidb 263M Jul 11 18:21 tikv-2023-07-11T18-21-30.055.log
@mayjiang0203
Copy link
Author

mayjiang0203 commented Jul 13, 2023 •
edited

/type enhancement

@ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Jul 13, 2023

@mayjiang0203: The label(s) type/enhencement cannot be applied, because the repository doesn't have them.

In response to this:

/type enhencement

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@ti-chi-bot ti-chi-bot bot added the type/enhancement Type: Issue - Enhancement label Jul 13, 2023
@mayjiang0203
Copy link
Author

/assign @v01dstar

@v01dstar v01dstar mentioned this issue Jul 13, 2023
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jul 14, 2023
@v01dstar
cdc: Exponentialize resolved ts scan backoff (#15119)
c73b6d3 
close #15112

Make the scan task retry wait time exponential, so that less logs will be produced.

Signed-off-by: Yang Zhang <yang.zhang@pingcap.com>
@ti-chi-bot ti-chi-bot mentioned this issue Feb 1, 2024
Merged
@TonsnakeLin
Copy link
Contributor

/label affects-7.1

@TonsnakeLin
Copy link
Contributor

/type bug

@ti-chi-bot ti-chi-bot bot added the type/bug Type: Issue - Confirmed a bug label Mar 1, 2024
@TonsnakeLin
Copy link
Contributor

/remove-type enhancement

@ti-chi-bot ti-chi-bot bot removed the type/enhancement Type: Issue - Enhancement label Mar 1, 2024
ti-chi-bot bot added a commit that referenced this issue Mar 1, 2024
@ti-chi-bot @v01dstar
@tonyxuqqi @ti-chi-bot
cdc: Exponentialize resolved ts scan backoff (#15119) (#16483)
3443254 
close #15112

Make the scan task retry wait time exponential, so that less logs will be produced.

Signed-off-by: Yang Zhang <yang.zhang@pingcap.com>

Co-authored-by: Yang Zhang <yang.zhang@pingcap.com>
Co-authored-by: tonyxuqqi <tonyxuqi@outlook.com>
Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@v01dstar v01dstar

Labels
affects-7.1 type/bug Type: Issue - Confirmed a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

cdc: Exponentialize resolved ts scan backoff v01dstar/tikv
3 participants
@mayjiang0203 @v01dstar @TonsnakeLin