[ISSUE #314] Heartbeat handler use independently thread pool

[fuyou001]

What is the purpose of the change

thousands of client Sub message, when subscription change, broker notify all client with same cid,
causing related ThreadPoolExecutor pool full, so Heartbeat use independently thread pool

Brief changelog

increase notify consumerIds changed semaphore size

Verifying this change

Follow this checklist to help us incorporate your contribution quickly and easily:

• Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
• Format the pull request title like [ROCKETMQ-XXX] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
• Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
• Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
• Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test to make sure integration-test pass.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

[lizhanhui]

I am wondering why notifying clients of membership change could take up the thread pool? Note notify is performed in an asynchronous non-blocking and one-way manner. Can you share your case details justifying this is indeed the root cause?

[vongosling]

@fuyou001 thoughts?

[fuyou001]

some reasons, I can't say more about it. @lizhanhui @vongosling

[vsair]

@fuyou001 I agree with @lizhanhui , the others command would not take up the thread pool. It's useless to isolate the pool.

[coveralls]

Coverage increased (+0.003%) to 42.318% when pulling 25aa641 on fuyou001:issues-314 into b054a9d on apache:develop.

[lizhanhui]

Considering heartbeat plays a major role in maintaining cluster members in sync, I'd agree to move it to a dedicated thread-pool so that it won't be delayed too much in case broker is under heavy pressure.

[lizhanhui]

@vsair I talked to @fuyou001 privately via email. The reason he is reluctant to share is that disclosing these technical details may bring about security threats to their commercial service before applying patches to all serving brokers. I'd post updates to share the scenario and explain more at the earliest convenient time.

[lollipopjin]

It seems to be more reasonable to isolate the time-taking notification works with Heartbeat, so +1.

[vongosling]

LGTM if we have finished the checklist before a merge @fuyou001 Another question, do you consider it is rational if we have created so many threadpool?

[fuyou001]

@vongosling