fix: starve lock caused by slow GetCompactionTo method when too much segments

[chyezh]

issue: #30823
pr: #30963

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[chyezh]

/run-cpu-e2e

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[chyezh]

/run-cpu-e2e

[mergify]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

[chyezh]

/run-cpu-e2e

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[codecov]

Codecov Report

Attention: Patch coverage is 94.87179% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 82.22%. Comparing base (483a32b) to head (94c31b2).
Report is 13 commits behind head on 2.3.

Additional details and impacted files

@@            Coverage Diff             @@
##              2.3   #30965      +/-   ##
==========================================
+ Coverage   82.17%   82.22%   +0.04%     
==========================================
  Files         841      843       +2     
  Lines      121490   121873     +383     
==========================================
+ Hits        99833   100204     +371     
- Misses      18424    18440      +16     
+ Partials     3233     3229       -4     

Files	Coverage Δ
internal/datacoord/meta.go	89.61% <100.00%> (+0.84%)	⬆️
internal/datacoord/segment_info.go	75.60% <96.77%> (+2.74%)	⬆️
internal/datacoord/services.go	85.88% <80.00%> (+0.03%)	⬆️

... and 40 files with indirect coverage changes

[xiaocai2333]

/lgtm

[czs007]

/approve
/lgtm

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chyezh, czs007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• internal/datacoord/OWNERS [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment