redo: make row flush batch size configurable

[wlwilliamx]

What problem does this PR solve?

Issue Number: close #5936

The redo DML writers use a hard-coded 1024-row flush threshold. For
workloads with small row events and object storage, this can force frequent
small-file uploads before max-log-size or flush-interval is reached,
amplifying object-store latency and checkpoint lag.

What is changed and how it works?

• Add consistent.flush-batch-size to the changefeed configuration and v2 API.
• Default it to 0. Zero disables count-based flushing, while a positive value
  flushes after exactly that many row events.
• Apply the configuration to both memory and file redo writer backends.
• Keep the sink-to-writer input batch independent from the persistent flush
  policy.
• Reject negative values during changefeed configuration validation.

Check List

Tests

• Unit test
  • go test -race --tags=intest ./pkg/config ./api/v2 ./pkg/redo/writer ./pkg/redo/writer/file ./pkg/redo/writer/memory ./downstreamadapter/sink/redo
• Manual test

Questions

Will it cause performance regression or break compatibility?

The default batching behavior changes intentionally: count-based flushing is
disabled instead of flushing every 1024 row events. Redo logs are still flushed
by max-log-size and flush-interval.

This reduces object-storage request amplification for workloads with many small
row events. It can retain more callbacks between periodic or size-based flushes.
Setting flush-batch-size = 1024 restores the previous count-based behavior.
The redo log data format is unchanged.

Do you need to update user documentation, design documentation or monitoring documentation?

Yes. The TiCDC configuration documentation should describe
consistent.flush-batch-size, its default value, and the meaning of zero.

Release note

Add the `consistent.flush-batch-size` changefeed option. It defaults to `0`,
which disables row-count-based redo log flushing. Set it to a positive number
to flush redo logs after that many row events.

Summary by CodeRabbit

• New Features

  • Added configurable redo log flush batch sizing.
  • Added API and TOML support for setting flush-batch-size.
  • Positive values trigger flushing after the configured number of rows; zero disables count-based flushing while retaining other flush triggers.
  • Improved batching for data transfer to the DML writer.

• Bug Fixes

  • Added validation to reject negative flush batch sizes and apply defaults when unspecified.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign lidezhu for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS
• pkg/config/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: aa7878e7-afb1-4327-b888-cb89445e66b2

📥 Commits

Reviewing files that changed from the base of the PR and between 6fc0494 and 41f3158.

📒 Files selected for processing (4)

• pkg/redo/writer/file/file.go
• pkg/redo/writer/file/file_test.go
• pkg/redo/writer/memory/file_worker.go
• pkg/redo/writer/memory/file_worker_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/redo/writer/file/file_test.go

---

📝 Walkthrough

Walkthrough

The change adds Consistent.FlushBatchSize to API and internal configuration. Redo writers use positive values for count-based flushing and use zero to disable that trigger. Callback release now follows durable file rotation. The redo sink uses a separate fixed input batch size.

Changes

Redo flush batching

Layer / File(s)	Summary
Flush batch configuration contract api/v2/model.go, api/v2/*test.go, pkg/config/*, pkg/redo/config.go, pkg/redo/testutil/config.go	The API and internal configuration carry FlushBatchSize. Validation defaults unset values, preserves zero, and rejects negative values.
Writer configuration propagation pkg/redo/writer/config.go, pkg/redo/writer/writer_test.go	The redo writer loads and exposes the configured flush batch size.
File writer flush behavior pkg/redo/writer/file/file.go, pkg/redo/writer/file/file_test.go	The file writer reports rotation, releases callbacks after durable writes, and applies configured count-based flushing. Zero disables count-based flushing.
Memory worker flush behavior pkg/redo/writer/memory/file_worker.go, pkg/redo/writer/memory/file_worker_test.go	The memory worker associates callbacks with files and releases them in order after durable rotation. Tests cover configured thresholds and zero-based disabling.
Independent sink input batching downstreamadapter/sink/redo/sink.go, downstreamadapter/sink/redo/sink_test.go	The redo sink uses a fixed 1024-item transfer batch, separate from persistent flush settings.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant APIModel
  participant ConsistentConfig
  participant WriterConfig
  participant MemoryWorker
  participant RedoFile
  APIModel->>ConsistentConfig: provide FlushBatchSize
  ConsistentConfig->>WriterConfig: pass validated value
  WriterConfig->>MemoryWorker: configure row-count threshold
  MemoryWorker->>RedoFile: write events
  RedoFile-->>MemoryWorker: report file rotation
  MemoryWorker-->>MemoryWorker: release callbacks after durable files

Loading

Possibly related PRs

• pingcap/ticdc#5595: Both changes modify redo sink batching and callback behavior.
• pingcap/ticdc#5862: Both changes extend replica configuration fields and API conversion tests.

Suggested reviewers: wk989898, 3aceshowhand

Poem

A rabbit sets the batch just right,
Zero keeps count flushes out of sight.
Config hops through each layer,
Callbacks wait till files are fair.
The sink packs its input with care.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 58.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: making the redo row flush batch size configurable.
Description check	✅ Passed	The description includes the issue, problem, implementation, tests, compatibility impact, documentation note, and release note.
Linked Issues check	✅ Passed	The changes implement the linked issue objectives by making row-count flushing configurable and disabling the default threshold for both writer backends.
Out of Scope Changes check	✅ Passed	The changes are limited to configuration, API wiring, redo writer behavior, sink batching separation, and related tests.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[wlwilliamx]

/test all