TiKV OOM during CDC initial scan

[fubinzh]

Bug Report

What version of TiKV are you using?

/ # /tikv-server -V
TiKV
Release Version: 6.5.3
Edition: Enterprise
Git Commit Hash: 5165943
Git Commit Branch: heads/refs/tags/v6.5.3-20231116-5165943

What operating system and CPU are you using?

K8S

Steps to reproduce

1. TiDB cluster with 6 TiKV (16c 40g), 2 TiCDC
2. Create cdc changefeed and pause it
3. Run workload to generate 1TB data for CDC to sync.
4. Resume cdc changefeed

What did you expect?

TiKV should not OOM

What did happened?

TiKV OOM

[fubinzh]

/assign @hicqu

[fubinzh]

/severity major

[overvenus]

The OOM is caused by initial scan pending task surge. The size of initial scan tasks is roughly 5848 bytes, 1649639 tasks take about 8.9 GB memory.

TiKV-2 Pending Tasks	TiKV-2 Memory

tikv/components/cdc/src/endpoint.rs

Lines 765 to 780 in 5165943

	self.workers.spawn(async move {
	CDC_SCAN_TASKS.with_label_values(&["total"]).inc();
	match init
	.initialize(change_cmd, raft_router, concurrency_semaphore)
	.await
	{
	Ok(()) => {
	CDC_SCAN_TASKS.with_label_values(&["finish"]).inc();
	}
	Err(e) => {
	CDC_SCAN_TASKS.with_label_values(&["abort"]).inc();
	error!("cdc initialize fail: {}", e; "region_id" => region_id);
	init.deregister_downstream(e)
	}
	}
	});

---

To fix the issue,

• TiKV needs to limit the total number of pending tasks, it must reject requets from TiCDC when there are too many pending tasks.
• TiCDC need to limit concurrent requests, maybe revert kvclient(cdc): remove rate limiter for scanning and error handling pingcap/tiflow#8860 (which essentially reverts the effort of kv/client: use region based rate limiter and decouple it from event handler pingcap/tiflow#3118 and cdc/kv: limit EventFeed retry pingcap/tiflow#853 and results in tikv overload)

[hicqu]

On TiCDC instances, there are lots of such logs:

[2023/11/19 16:11:13.208 +08:00] [INFO] [client.go:981] ["stream to store closed"] [namespace=default] [changefeed=test1] [addr=tc-tikv-2.tc-tikv-peer.webank-cdc-tps-4590005-1-42.svc:20160] [storeID=8]

After a gRPC stream is re-established, all subscribed regions will be re-sent to TiKV instances. That's why there are only 50K subscribed regions on a TiKV but 15000K pending region tasks.

Limiting the total number of pending tasks is a good idea. However we still need to resolve the stream-recreation case.

[hicqu]

@overvenus pingcap/tiflow#8860 is expected to reduce changefeed initialization time, and it does works. So I suggest to just limit total number of pending tasks on TiKVs.