[fix] [broker] add timeout for health check read.

[thetumbled]

Fixes #21989

Motivation

There could be cases that the health-check message is lost or the reader just can't read the expected message for any reason, which will result into the producers and readers can't be closed and therefore the resource leak problem occur.

Modifications

Add timeout for health check read operation.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: thetumbled#39

[thetumbled]

PTAL, thanks. @mattisonchao @lhotari

[lhotari]

LGTM. Good work @thetumbled

[poorbarcode]

LGTM. @thetumbled could you add a test for this case?

[thetumbled]

LGTM. @thetumbled could you add a test for this case?

Done.

[thetumbled]

/pulsarbot rerun-failure-checks

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (8a18043) 73.65% compared to head (5150b88) 73.63%.
Report is 2 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #21990      +/-   ##
============================================
- Coverage     73.65%   73.63%   -0.03%     
- Complexity    32471    32477       +6     
============================================
  Files          1863     1863              
  Lines        138772   138777       +5     
  Branches      15203    15203              
============================================
- Hits         102213   102187      -26     
- Misses        28665    28695      +30     
- Partials       7894     7895       +1     

Flag	Coverage Δ
inttests	24.09% <85.71%> (-0.02%)	⬇️
systests	23.96% <42.85%> (+0.09%)	⬆️
unittests	72.91% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...g/apache/pulsar/broker/admin/impl/BrokersBase.java	72.45% <100.00%> (+1.58%)	⬆️

... and 85 files with indirect coverage changes