Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Too many logs: "synchronize is taking too long, report a bug" occupied a lot disk capacity #2706

Closed
Tammyxia opened this issue Sep 1, 2021 · 4 comments · Fixed by #3775
Closed

Too many logs: "synchronize is taking too long, report a bug" occupied a lot disk capacity #2706

Tammyxia opened this issue Sep 1, 2021 · 4 comments · Fixed by #3775
Assignees
@overvenus
Labels
area/ticdc Issues or PRs related to TiCDC. priority/p0 Issue with the highest priority, should be resolved as soon as possible. severity/major This is a major bug. type/bug This is a bug.
Projects
Capturing 60K Tables Question and Bug Reports

Comments

@Tammyxia
Copy link

Tammyxia commented Sep 1, 2021 •
edited
Loading

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do? If possible, provide a recipe for reproducing the error.
  • Workload: DML + 20 new creating tables.
  • TiKV scale-in from 24 to 20. Total region 20w.

  1. What did you expect to see?

  2. What did you see instead?

  • Owner node new generated 230+ cdc log in 20hours, each log file is 300MB, which occupied too much disk capacity.

image
image

  1. Versions of the cluster

    • Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

      4.0.14

    • TiCDC version (execute cdc version):

      4.0.14 cloud
@Tammyxia Tammyxia added type/bug This is a bug. severity/moderate This is a moderate bug. labels Sep 1, 2021
@liuzix liuzix added severity/major This is a major bug. and removed severity/moderate This is a moderate bug. labels Sep 2, 2021
@liuzix liuzix added this to To do in Capturing 60K Tables via automation Sep 2, 2021
@github-actions github-actions bot added this to Need Triage in Question and Bug Reports Sep 2, 2021
@liuzix liuzix self-assigned this Sep 2, 2021
@liuzix liuzix moved this from Need Triage to In Progress in Question and Bug Reports Sep 2, 2021
@liuzix liuzix added the priority/p0 Issue with the highest priority, should be resolved as soon as possible. label Sep 2, 2021
@liuzix
Copy link
Contributor

liuzix commented Sep 2, 2021

This seems to be the exact problem solved by #2078

@liuzix liuzix moved this from To do to In progress in Capturing 60K Tables Sep 2, 2021
@overvenus
Copy link
Member

@liuzix Can we set the level to debug or reduce frequency? In this case, it takes lots of CPU and IO and may block the entire TiCDC process.

image

@liuzix liuzix closed this as completed Sep 7, 2021
Capturing 60K Tables automation moved this from In progress to Done Sep 7, 2021
Question and Bug Reports automation moved this from In Progress to Done Sep 7, 2021
@cyliu0 cyliu0 reopened this Sep 9, 2021
Capturing 60K Tables automation moved this from Done to In progress Sep 9, 2021
Question and Bug Reports automation moved this from Done to In Progress Sep 9, 2021
@amyangfei
Copy link
Contributor

amyangfei commented Sep 9, 2021 •
edited
Loading

Should add this commit to master branch d6ac981

@overvenus
Copy link
Member

@liuzix Please cherry-pick d6ac981 to master branch, thanks!

@AkiraXie AkiraXie added the area/ticdc Issues or PRs related to TiCDC. label Dec 6, 2021
@overvenus overvenus assigned overvenus and unassigned liuzix Dec 7, 2021
@overvenus overvenus mentioned this issue Dec 7, 2021
Merged
Capturing 60K Tables automation moved this from In progress to Done Dec 8, 2021
Question and Bug Reports automation moved this from In Progress to Done Dec 8, 2021
This was referenced Dec 8, 2021
Merged
Merged
Merged
This was referenced Dec 8, 2021
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@overvenus overvenus

Labels
area/ticdc Issues or PRs related to TiCDC. priority/p0 Issue with the highest priority, should be resolved as soon as possible. severity/major This is a major bug. type/bug This is a bug.
Projects
Question and Bug Reports
Done
Capturing 60K Tables
  
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

workerpool: limit the rate to output deadlock warning overvenus/ticdc
6 participants
@amyangfei @overvenus @liuzix @cyliu0 @AkiraXie @Tammyxia