[Branch-2.8] Fix Broker HealthCheck Endpoint Exposes Race Conditions.

[Technoboy-]

Cherry-pick #14367

Master Issue: #14362

Motivation

See #14362

According to relative PR #7724, we will force delete all subscriptions when calling healthCheck REST API. but it has a race condition when two threads call this API.
Please consider this case:

Thread A: Clean up all subscriptions, then create a reader.
Thread B: Prepare to force delete all subscriptions.

So, in this case, the reader of thread A is deleted and then an NPE or other exception occurs.

Modifications

• Use Completable#handle to fix problem 1, the reader needs to be closed regardless of whether there is an exception.
• Recheck the subscription after closing reading, and force deletion if it still exists after closing reading.
• Added multi-threaded tests for health checks.

Verifying this change

• Make sure that the change passes the CI checks.

Documentation

• no-need-doc

[nicoloboschi]

LGTM

[eolivelli]

@Technoboy-
I little suggestion for the future:
when you cherry pick a patch from someone else it is a good practice to start by cherry picking the original patch and try to keep the original author, otherwise it seems that you are the original author, in this case it is @mattisonchao

I am looking here
https://github.com/apache/pulsar/pull/14618/commits

this is the original commit
4f1e39b

it is fine to amend it and adapt it to the target branch, but my point is that in the git history @mattisonchao should be listed as author of the patch (whenever it is possible).

[eolivelli]

LGTM

committing as soon as CI is green

[Technoboy-]

@Technoboy- I little suggestion for the future: when you cherry pick a patch from someone else it is a good practice to start by cherry picking the original patch and try to keep the original author, otherwise it seems that you are the original author, in this case it is @mattisonchao

I am looking here https://github.com/apache/pulsar/pull/14618/commits

this is the original commit 4f1e39b

it is fine to amend it and adapt it to the target branch, but my point is that in the git history @mattisonchao should be listed as author of the patch (whenever it is possible).

Yes, I see. Thanks. But there are a lot of conflicts when cherry-picking. So I have to rewrite it according to the original one. We can add the author info when merging. (I'm not for any commits record here, but for speeding up the release.)

[Technoboy-]

LGTM

committing as soon as CI is green

Ok, fixed.