mysql (ticdc): Improve the performance of the mysql sink by refining the transaction event batching logic

[hongyunyan]

What problem does this PR solve?

Issue Number: close #11241

What is changed and how it works?

1. When there are a lot of txns in worker's txnCh, we try to do flush until save batches to maxTxnRows, instead of doing flush with tiny txns because of ticker.C.
2. Adjust the calculation of worker-busy-ratio to make it more precise and more clear.
   The original calculation of worker-busy-ratio may have certain deviation when the flush time is long (such as larger than hundreds of milliseconds). And the longer the flush time, the larger the deviation. The reason for the deviation is that when the flsuh time is long, the interval between each time we increase the worker busy ratio value clearly exceeds 1s, resulting in the growth rate of the worker-busy-ratio per second being less than actual rate.

Performance Test Result

We take a large workload on upstream to keep worker-busy-ratio as 100%, and compare the improvement of sink performance before and after optimization.

1. Simulate network latency between the upstream and downstream is 2ms.
   After Optimization -- 33339 rows/s (+14%). Before Optimization -- 29075 rows/s
2. Simulate network latency between the upstream and downstream is 5ms.
   After Optimization -- 20528 rows/s(+13%). Before Optimization -- 18129 rows/s
3. Simulate network latency between the upstream and downstream is 20ms.
   After Optimization -- 7273 rows/s(+20%). Before Optimization -- 6048 rows/s
   

New panel of worker busy ratio

Check List

Tests

• Manual test (add detailed scripts or steps below)

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

None

[CharlesCheung96]

It seems that the core idea here is to ensure that the flush interval is greater than 10ms. Maybe we could record lastFlushTime at the end of each flush and check it in each ticker? such as:

case <-ticker.C:
    if time.Since(lastFlushTime) >= w.flushInterval {
    	needFlush = true
    }

[hongyunyan]

I believe both of these two methods are feasible, and I'm wondering if the current code might be a bit more straightforward and easier to understand?

[CharlesCheung96]

In general, the nested logic is more complex.

[hongyunyan]

I think it's ok here orz

[codecov]

Codecov Report

Merging #10466 (7d8bd0b) into master (3e9c9de) will decrease coverage by 5.9918%.
Report is 296 commits behind head on master.
The diff coverage is 53.8765%.

Additional details and impacted files

Components	Coverage Δ
cdc	61.6267% <53.8765%> (-1.7802%)	⬇️
dm	51.2640% <ø> (∅)
engine	63.4141% <ø> (∅)

Flag	Coverage Δ
unit	57.4151% <53.8765%> (-5.9918%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master     #10466        +/-   ##
================================================
- Coverage   63.4069%   57.4151%   -5.9918%     
================================================
  Files           392        849       +457     
  Lines         51067     125864     +74797     
================================================
+ Hits          32380      72265     +39885     
- Misses        16385      48192     +31807     
- Partials       2302       5407      +3105     

[hongyunyan]

/test verify

[hicqu]

/retest

[hongyunyan]

/cherry-pick release-6.5

[ti-chi-bot]

@hongyunyan: new pull request created to branch release-6.5: #11242.

In response to this:

/cherry-pick release-6.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[hongyunyan]

/cherry-pick 7.1

[hongyunyan]

/cherry-pick release-7.1

[ti-chi-bot]

@hongyunyan: cannot checkout 7.1: error checking out 7.1: exit status 1. output: error: pathspec '7.1' did not match any file(s) known to git

In response to this:

/cherry-pick 7.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@hongyunyan: new pull request created to branch release-7.1: #11244.

In response to this:

/cherry-pick release-7.1

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

In response to a cherrypick label: new pull request could not be created: failed to create pull request against pingcap/tiflow#release-7.1 from head ti-chi-bot:cherry-pick-10466-to-release-7.1: status code 422 not one of [201], body: {"message":"Validation Failed","errors":[{"resource":"PullRequest","code":"custom","message":"A pull request already exists for ti-chi-bot:cherry-pick-10466-to-release-7.1."}],"documentation_url":"https://docs.github.com/rest/pulls/pulls#create-a-pull-request"}

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-7.5: #11248.