fix: [Cherry-pick]Pick major compaction fixs and optimizations

[wayblink]

This PR cherry-picks the following commits:

• fix: sync partitiion stats blocking balance task fix: sync partitiion stats blocking balance task(#33741) #33742
• fix: Fix meta prefix overlap bug fix: Fix meta prefix overlap bug #33830
• fix: Small fixs of major compaction fix: Small fixs of major compaction #33929
• fix: Fix memory buffer error & some renaming fix: Fix memory buffer error & some renaming #33850
• fix: sync part stats task cannot be finished fix: sync part stats task cannot be finished(#30376) #34027
• Add an option to enable/disable vector field clustering key Add an option to enable/disable vector field clustering key #34097
• fix: fix error ignore in compactor fix: fix error ignore in compactor #34169
• fix:load major compaction partial result fix:load major compaction partial result(#34051) #34052
• Use new stream segment reader in clustering compaction Use new stream segment reader in clustering compaction #34232

issue: #30633
pr: #33742 #33830 #33929 #33850 #34027 #34097 #34169 #34052 #34232

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wayblink
To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed.
You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mergify]

@wayblink Thanks for your contribution. Please submit with DCO, see the contributing guide https://github.com/milvus-io/milvus/blob/master/CONTRIBUTING.md#developer-certificate-of-origin-dco.

[mergify]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[wayblink]

/run-cpu-e2e

[codecov]

Codecov Report

Attention: Patch coverage is 69.23077% with 56 lines in your changes missing coverage. Please review.

Project coverage is 80.76%. Comparing base (7ee4ccb) to head (a72b9bb).
Report is 9 commits behind head on 2.4.

Additional details and impacted files

@@            Coverage Diff             @@
##              2.4   #34360      +/-   ##
==========================================
- Coverage   81.46%   80.76%   -0.70%     
==========================================
  Files        1028     1034       +6     
  Lines      132427   134672    +2245     
==========================================
+ Hits       107879   108766     +887     
- Misses      20474    21714    +1240     
- Partials     4074     4192     +118     

Files	Coverage Δ
internal/datacoord/compaction_task_clustering.go	47.72% <100.00%> (ø)
internal/datanode/compaction/segment_writer.go	84.55% <100.00%> (+0.79%)	⬆️
internal/proxy/impl.go	86.19% <100.00%> (+<0.01%)	⬆️
internal/proxy/task.go	87.70% <100.00%> (-0.03%)	⬇️
internal/querycoordv2/checkers/balance_checker.go	95.12% <100.00%> (ø)
internal/querycoordv2/task/task.go	97.70% <100.00%> (+0.01%)	⬆️
internal/querynodev2/delegator/delegator.go	86.87% <100.00%> (+0.21%)	⬆️
internal/storage/partition_stats.go	81.39% <100.00%> (+2.08%)	⬆️
internal/storage/serde.go	88.05% <100.00%> (+0.04%)	⬆️
pkg/common/map.go	100.00% <100.00%> (ø)
... and 9 more

... and 35 files with indirect coverage changes

[mergify]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@wayblink E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.