kv: reduce gRPC window size to avoid OOM

[overvenus]

What problem does this PR solve?

Note: TiCDC in the following tests includes #2214.

Memory usage (pause sorter)

Tested with following changes

diff --git a/cdc/puller/sorter/unified_sorter.go b/cdc/puller/sorter/unified_sorter.go
index 9cd9c898..6e033429 100644
--- a/cdc/puller/sorter/unified_sorter.go
+++ b/cdc/puller/sorter/unified_sorter.go
@@ -197,9 +197,9 @@ func (s *UnifiedSorter) Run(ctx context.Context) error {
                }
        })
 
-       errg.Go(func() error {
-               return printError(runMerger(subctx, numConcurrentHeaps, heapSorterCollectCh, s.outputCh, ioCancelFunc))
-       })
+       // errg.Go(func() error {
+       //      return printError(runMerger(subctx, numConcurrentHeaps, heapSorterCollectCh, s.outputCh, ioCancelFunc))
+       // })
 
        errg.Go(func() error {
                captureAddr := util.CaptureAddrFromCtx(ctx)

64MB vs 64KB

The last image shows TiKV snapshot is holded for a long time, because CDC scan is flow controlled. Through, I doubt if it will actually happen as TiCDC sorter can always make progress.

Performance

No significant difference.

64MB (OOM) vs 64KB

Close #2673

What is changed and how it works?

Check List

Tests

• Manual test (add detailed scripts or steps below)

Related changes

• Need to cherry-pick to the release branch

Release note

Fix OOM when TiCDC captures too many regions

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• amyangfei
• liuzix

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[amyangfei]

By the way, is there any metric exposed from gRPC API we can use to monitor the stream/connection status, such as window size, receive buffer, etc.

[overvenus]

By the way, is there any metric exposed from gRPC API we can use to monitor the stream/connection status, such as window size, receive buffer, etc.

Maybe you are looking for channelz https://pkg.go.dev/google.golang.org/grpc/internal/channelz#ChannelInternalMetric

[amyangfei]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: e83449a

[3AceShowHand]

how about 1 << 16 ?

[overvenus]

/merge

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: 2ff79b1

[codecov-commenter]

Codecov Report

Merging #2699 (a5a48f8) into master (81c22b1) will increase coverage by 5.2636%.
The diff coverage is 68.8027%.

@@               Coverage Diff                @@
##             master      #2699        +/-   ##
================================================
+ Coverage   55.8716%   61.1353%   +5.2636%     
================================================
  Files           169        161         -8     
  Lines         20667      17916      -2751     
================================================
- Hits          11547      10953       -594     
+ Misses         8012       5970      -2042     
+ Partials       1108        993       -115     

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #2723.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #2724.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #2725.

[ti-chi-bot]

In response to a cherrypick label: new pull request created: #2726.