Is your feature request related to a problem?
The current log puller memory quota mechanism is too coarse-grained and depends on dynstream memory control. Today, when pending events in dynstream exceed the threshold, SubscriptionClient stops pushing TiKV events into dynstream.
This control point is too late. It only throttles events locally after TiCDC has already received them from TiKV, but it does not control the real source of memory growth. For example, even when the puller is already full, new region incremental scans may still be scheduled and sent to TiKV, and TiKV may continue producing events for these regions.
In large-table, many-region, high-write-throughput, or slow-downstream scenarios, memory can still keep growing, so the current quota mechanism is not very effective.
Essentially, the current behavior is closer to “stop enqueueing into the local queue after it is full” than a complete puller-side flow control mechanism.
Describe the feature you'd like
Completely remove log puller’s dependency on dynstream memory control and introduce a dedicated memory control mechanism for log puller.
The new mechanism should move the control point closer to the source. When the puller is under high memory pressure, it should pause or slow down scheduling new region incremental scans, so TiCDC does not keep expanding the set of regions that TiKV is producing events for. After memory usage drops below a safe threshold, region scheduling can resume.
In log puller, dynstream should only be responsible for event dispatching and handling. It should no longer be responsible for puller memory quota. The puller should make quota decisions based on its own memory usage, with metrics and logs showing memory usage, quota state, and whether region scheduling is paused due to quota.
Describe alternatives you've considered
No response
Teachability, Documentation, Adoption, Migration Strategy
No response
Is your feature request related to a problem?
The current log puller memory quota mechanism is too coarse-grained and depends on dynstream memory control. Today, when pending events in dynstream exceed the threshold,
SubscriptionClientstops pushing TiKV events into dynstream.This control point is too late. It only throttles events locally after TiCDC has already received them from TiKV, but it does not control the real source of memory growth. For example, even when the puller is already full, new region incremental scans may still be scheduled and sent to TiKV, and TiKV may continue producing events for these regions.
In large-table, many-region, high-write-throughput, or slow-downstream scenarios, memory can still keep growing, so the current quota mechanism is not very effective.
Essentially, the current behavior is closer to “stop enqueueing into the local queue after it is full” than a complete puller-side flow control mechanism.
Describe the feature you'd like
Completely remove log puller’s dependency on dynstream memory control and introduce a dedicated memory control mechanism for log puller.
The new mechanism should move the control point closer to the source. When the puller is under high memory pressure, it should pause or slow down scheduling new region incremental scans, so TiCDC does not keep expanding the set of regions that TiKV is producing events for. After memory usage drops below a safe threshold, region scheduling can resume.
In log puller, dynstream should only be responsible for event dispatching and handling. It should no longer be responsible for puller memory quota. The puller should make quota decisions based on its own memory usage, with metrics and logs showing memory usage, quota state, and whether region scheduling is paused due to quota.
Describe alternatives you've considered
No response
Teachability, Documentation, Adoption, Migration Strategy
No response