enhance: add graceful stop timeout to avoid node stop hang under extreme cases

[chyezh]

1. add coordinator and proxy graceful stop timeout to 5s.
2. add other work node graceful stop timeout to 900s, and we should potentially change this to 600s when graceful stop is smooth
3. change the order of datacoord component while stop.
4. LivenessCheck do not perform graceful shutdown now.

issue: #30310
pr: #30317
also see: #30306

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[chyezh]

/run-cpu-e2e

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[chyezh]

/run-cpu-e2e

[xiaofan-luan]

/approve
/lgtm

[mergify]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chyezh, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/OWNERS [xiaofan-luan]
• internal/datacoord/OWNERS [xiaofan-luan]
• internal/datanode/OWNERS [xiaofan-luan]
• internal/distributed/OWNERS [xiaofan-luan]
• internal/indexnode/OWNERS [xiaofan-luan]
• internal/proxy/OWNERS [xiaofan-luan]
• internal/querycoordv2/OWNERS [xiaofan-luan]
• internal/querynodev2/OWNERS [xiaofan-luan]
• internal/rootcoord/OWNERS [xiaofan-luan]
• pkg/util/OWNERS [xiaofan-luan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

Attention: 10 lines in your changes are missing coverage. Please review.

Comparison is base (26df754) 82.06% compared to head (b6cf703) 82.10%.
Report is 1 commits behind head on 2.3.

Additional details and impacted files

@@            Coverage Diff             @@
##              2.3   #30320      +/-   ##
==========================================
+ Coverage   82.06%   82.10%   +0.03%     
==========================================
  Files         840      840              
  Lines      121139   121170      +31     
==========================================
+ Hits        99413    99483      +70     
+ Misses      18487    18446      -41     
- Partials     3239     3241       +2     

Files	Coverage Δ
internal/distributed/datacoord/service.go	87.75% <100.00%> (+0.15%)	⬆️
internal/distributed/datanode/service.go	82.19% <100.00%> (+0.33%)	⬆️
internal/distributed/indexnode/service.go	71.51% <100.00%> (+0.70%)	⬆️
internal/distributed/proxy/service.go	80.44% <100.00%> (+0.07%)	⬆️
internal/distributed/querycoord/service.go	73.87% <100.00%> (+0.23%)	⬆️
internal/distributed/querynode/service.go	80.58% <100.00%> (+0.38%)	⬆️
internal/distributed/rootcoord/service.go	79.43% <100.00%> (+0.29%)	⬆️
pkg/util/paramtable/component_param.go	98.37% <100.00%> (+0.03%)	⬆️
internal/datacoord/server.go	71.02% <91.66%> (+1.01%)	⬆️
internal/datanode/data_node.go	79.91% <0.00%> (+2.18%)	⬆️
... and 5 more

... and 14 files with indirect coverage changes

[mergify]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

[jaime0815]

/lgtm

[mergify]

@chyezh ut workflow job failed, comment rerun ut can trigger the job again.

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[chyezh]

/run-cpu-e2e

[chyezh]

/run-cpu-e2e

[chyezh]

/run-cpu-e2e

[chyezh]

/run-cpu-e2e

[chyezh]

rerun ut

[jaime0815]

/lgtm

[mergify]

@chyezh E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[chyezh]

/run-cpu-e2e

[yanliang567]

e2e passed, but pipeline hang at log uploading. manual pass