[Enhancement] Optimize compaction resource usage for cloud native primary table

[luohaha]

Why I'm doing:
We need a way that it can:

1. limit the number of rowsets per compaction merge, thus reducing resource usage
2. does not affect the real compaction score calculation results

What I'm doing:

1. Add new BE config lake_pk_compaction_merge_rowset_delta, to limit the number of rowsets per compaction merge.
2. Make sure this config won't affect compaction score calculation.

Test result:

Continuously ingest 7GB of data into a tablet and observe the time consumed by compaction before and after optimization:
Before:

mysql> show proc '/compactions';
+----------------------------------------+-------+---------------------+---------------------+---------------------+-------+
| Partition                              | TxnID | StartTime           | CommitTime          | FinishTime          | Error |
+----------------------------------------+-------+---------------------+---------------------+---------------------+-------+
| test_pri_load.tbl_pk_withvarchar.10076 | 13    | 2024-01-20 07:25:14 | 2024-01-20 07:25:25 | 2024-01-20 07:25:32 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10076 | 23    | 2024-01-20 07:26:03 | 2024-01-20 07:26:30 | 2024-01-20 07:26:41 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10076 | 33    | 2024-01-20 07:26:54 | 2024-01-20 07:27:46 | 2024-01-20 07:28:03 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10076 | 43    | 2024-01-20 07:28:03 | 2024-01-20 07:29:16 | 2024-01-20 07:29:39 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10076 | 58    | 2024-01-20 07:29:39 | 2024-01-20 07:31:23 | 2024-01-20 07:31:56 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10076 | 74    | 2024-01-20 07:31:57 | 2024-01-20 07:34:05 | 2024-01-20 07:34:50 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10076 | 93    | 2024-01-20 07:34:50 | 2024-01-20 07:37:49 | 2024-01-20 07:38:52 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10076 | 114   | 2024-01-20 07:38:52 | 2024-01-20 07:42:21 | 2024-01-20 07:43:26 | NULL  |
+----------------------------------------+-------+---------------------+---------------------+---------------------+-------+

The longest compaction task cost 5min.

After:

mysql> show proc '/compactions';
+----------------------------------------+-------+---------------------+---------------------+---------------------+-------+
| Partition                              | TxnID | StartTime           | CommitTime          | FinishTime          | Error |
+----------------------------------------+-------+---------------------+---------------------+---------------------+-------+

| test_pri_load.tbl_pk_withvarchar.10101 | 130   | 2024-01-20 07:46:26 | 2024-01-20 07:46:33 | 2024-01-20 07:46:37 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10101 | 135   | 2024-01-20 07:46:45 | 2024-01-20 07:46:52 | 2024-01-20 07:46:57 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10101 | 140   | 2024-01-20 07:47:06 | 2024-01-20 07:47:13 | 2024-01-20 07:47:19 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10101 | 145   | 2024-01-20 07:47:27 | 2024-01-20 07:47:33 | 2024-01-20 07:47:42 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10101 | 150   | 2024-01-20 07:47:51 | 2024-01-20 07:47:58 | 2024-01-20 07:48:05 | NULL  |
...
| test_pri_load.tbl_pk_withvarchar.10101 | 204   | 2024-01-20 07:54:28 | 2024-01-20 07:54:35 | 2024-01-20 07:54:58 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10101 | 209   | 2024-01-20 07:55:09 | 2024-01-20 07:55:24 | 2024-01-20 07:56:05 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10101 | 216   | 2024-01-20 07:56:18 | 2024-01-20 07:56:26 | 2024-01-20 07:56:52 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10101 | 221   | 2024-01-20 07:57:05 | 2024-01-20 07:57:24 | 2024-01-20 07:58:07 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10101 | 227   | 2024-01-20 07:58:20 | 2024-01-20 07:58:27 | 2024-01-20 07:58:56 | NULL  |
| test_pri_load.tbl_pk_withvarchar.10101 | 232   | 2024-01-20 07:59:09 | 2024-01-20 07:59:24 | 2024-01-20 08:00:18 | NULL  |
...
| test_pri_load.tbl_pk_withvarchar.10101 | 245   | 2024-01-20 08:01:50 | 2024-01-20 08:02:07 | 2024-01-20 08:02:42 | NULL  |

The longest compaction task cost 1min.

This test result shows that new compaction policy can reduce 80% compaction resource usage.

What type of PR is this:

• BugFix
• Feature
• Enhancement
• Refactor
• UT
• Doc
• Tool

Does this PR entail a change in behavior?

• Yes, this PR will result in a change in behavior.
• No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

• Interface/UI changes: syntax, type conversion, expression evaluation, display information
• Parameter changes: default values, similar parameters but with different default values
• Policy changes: use new policy to replace old one, functionality automatically enabled
• Feature removed
• Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

• I have added test cases for my bug fix or my new feature
• This pr needs user documentation (for new or modified features or behaviors)
  • I have added documentation for my new feature or new function
• This is a backport pr

Bugfix cherry-pick branch check:

• I have checked the version labels which the pr will be auto-backported to the target branch
  • 3.2
  • 3.1
  • 3.0
  • 2.5

[sevev]

Limit the number of rowsets per compaction merge but increase compaction times?

[github-actions]

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

[github-actions]

[BE Incremental Coverage Report]

✅ pass : 4 / 5 (80.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/storage/lake/compaction_policy.cpp	4	5	80.00%	[188]

[github-actions]

@Mergifyio backport branch-3.2

[github-actions]

@Mergifyio backport branch-3.1

[mergify]

backport branch-3.2

✅ Backports have been created

• #39618 [Enhancement] Optimize compaction resource usage for cloud native primary table (backport #39611) has been created for branch branch-3.2

[mergify]

backport branch-3.1

✅ Backports have been created

• #39619 [Enhancement] Optimize compaction resource usage for cloud native primary table (backport #39611) has been created for branch branch-3.1 but encountered conflicts