TiCDC can't recover after TiDB of upstream OOM #8561
Assignees
Labels
affects-6.0
affects-6.1
affects-6.2
affects-6.3
affects-6.4
affects-6.5
affects-6.6
affects-7.0
area/ticdc
Issues or PRs related to TiCDC.
severity/major
This is a major bug.
type/bug
This is a bug.
Projects
Comments
WangLe1321
added
area/ticdc
|
The issue can be work around per #4516 (comment) |
|
/severity major |
ti-chi-bot
added
severity/major
|
/assign @overvenus |
ti-chi-bot
added a commit
to tikv/tikv
that referenced
this issue
Apr 7, 2023
…ock (#14465) close pingcap/tiflow#8561, ref #11993 cdc: batch send resolved ts exponentially to speed up TiCDC resolve lock Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
ti-chi-bot
pushed a commit
to ti-chi-bot/tikv
that referenced
this issue
May 23, 2023
close pingcap/tiflow#8561, ref tikv#11993 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>
overvenus
added a commit
to ti-chi-bot/tikv
that referenced
this issue
May 23, 2023
…ock (tikv#14465) close pingcap/tiflow#8561, ref tikv#11993 Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io> Signed-off-by: Neil Shen <overvenus@gmail.com>
ti-chi-bot bot
pushed a commit
to tikv/tikv
that referenced
this issue
May 23, 2023
…ock (#14465) (#14796) close pingcap/tiflow#8561, ref #11993, ref #14465 cdc: batch send resolved ts exponentially to speed up TiCDC resolve lock Signed-off-by: Neil Shen <overvenus@gmail.com> Co-authored-by: Neil Shen <overvenus@gmail.com> Co-authored-by: 二手掉包工程师 <rustin.liu@gmail.com> Co-authored-by: Ti Chi Robot <ti-community-prow-bot@tidb.io>
17 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What did you do?
TiDB of upstream has too high qps and connections, causing TIDB OOM.
metrics of TiDB:
https://clinic.pingcap.com/grafana/d/000000013tidbcloudv6_1_x/tidb-runtime-cluster-clinic?orgId=1&var-tidb_cluster=1379661944640134080&var-instance=All&var-store=All&from=1678407999000&to=1678415199000&var-tidb_version=v6.1.x&var-org=1372813089196917631&refresh=15m&var-org_type=cloud
https://clinic.pingcap.com/grafana/d/2bbdcd7d4d88a2dtidbcloudv6_1_x/tidb-summary-cluster-clinic?orgId=1&var-tidb_cluster=1379661944640134080&var-instance=All&var-store=All&from=1678407999000&to=1678415199000&var-tidb_version=v6.1.x&var-org=1372813089196917631&refresh=15m&var-org_type=cloud
https://clinic.pingcap.com/portal/#/orgs/1372813089196917631/clusters/1379661944640134080/report/5/fed8c3b6-fe7a-48f7-80e1-100dacf18938
after scale up TiDB nodes, TiDB recovered.
What did you expect to see?
After TiDB recovered, cdc should reduce lag.
What did you see instead?
cdc lag is still high.
Versions of the cluster
Upstream TiDB cluster version (execute
SELECT tidb_version();in a MySQL client):(paste TiDB cluster version here)Upstream TiKV version (execute
tikv-server --version):(paste TiKV version here)TiCDC version (execute
cdc version):(paste TiCDC version here)https://clinic.pingcap.com/portal/#/orgs/1372813089196917631/clusters/1379661944640134080
cluster version is v6.1.3
The text was updated successfully, but these errors were encountered: