Skip to content

improve log puller memory quota mechanism #5519

Description

@lidezhu

Is your feature request related to a problem?

The current log puller memory quota mechanism is too coarse-grained and depends on dynstream memory control. Today, when pending events in dynstream exceed the threshold, SubscriptionClient stops pushing TiKV events into dynstream.
This control point is too late. It only throttles events locally after TiCDC has already received them from TiKV, but it does not control the real source of memory growth. For example, even when the puller is already full, new region incremental scans may still be scheduled and sent to TiKV, and TiKV may continue producing events for these regions.
In large-table, many-region, high-write-throughput, or slow-downstream scenarios, memory can still keep growing, so the current quota mechanism is not very effective.
Essentially, the current behavior is closer to “stop enqueueing into the local queue after it is full” than a complete puller-side flow control mechanism.

Describe the feature you'd like

Completely remove log puller’s dependency on dynstream memory control and introduce a dedicated memory control mechanism for log puller.
The new mechanism should move the control point closer to the source. When the puller is under high memory pressure, it should pause or slow down scheduling new region incremental scans, so TiCDC does not keep expanding the set of regions that TiKV is producing events for. After memory usage drops below a safe threshold, region scheduling can resume.
In log puller, dynstream should only be responsible for event dispatching and handling. It should no longer be responsible for puller memory quota. The puller should make quota decisions based on its own memory usage, with metrics and logs showing memory usage, quota state, and whether region scheduling is paused due to quota.

Describe alternatives you've considered

No response

Teachability, Documentation, Adoption, Migration Strategy

No response

Metadata

Metadata

Assignees

Labels

affects-8.5This bug affects the 8.5.x(LTS) versions.type/enhancementThe issue or PR belongs to an enhancement.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions