HDDS-3933. Fix memory leak because of too many Datanode State Machine Thread

[runzhiwang]

What changes were proposed in this pull request?

What's problem ?
Datanode creates more than 20K Datanode State Machine Thread, then OOM happened.

What's the reason ?
20K Datanode State Machine Thread were created by newCachedThreadPool

Almost all of them were wait lock.

Only one Datanode State Machine Thread got the lock, and block when submitRequest. Because this thread was blocked and can not free the lock, newCachedThreadPool will create new thread infinitely.

How to fix ?

1. Avoid use newCachedThreadPool, because it will create new thread infinitely, if no thread available in pool.
2. Cancel future when task time out.
3. Besides there is bug in such following code which want to loop until timeout or returned >= count.
   But timeout can not work, for example when init startTime = 0 and timeLeft = 100, if monotonicNow = 10, timeLeft = 100 - (10 - 0) = 90, and then if monotonicNow = 11, timeLeft = 90 - (11 - 0) = 79 which actually should be 89.

 while (returned < count && timeLeft > 0) {
    timeLeft = timeLeft - (Time.monotonicNow() - startTime);
 }

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3933

How was this patch tested?

Existed UT.

[runzhiwang]

I will check the failed ut.

[ChenSammi]

Kindly add new unit test to show case the new logic works as expected. And check the failed UT and accpetance test.

[xiaoyuyao]

Thanks @runzhiwang for reporting issue and the fix. Patch LGTM, just have a minor question inline.

[codecov-commenter]

Codecov Report

Merging #1185 into master will increase coverage by 2.93%.
The diff coverage is 74.96%.

@@             Coverage Diff              @@
##             master    #1185      +/-   ##
============================================
+ Coverage     70.56%   73.49%   +2.93%     
- Complexity     9427    10048     +621     
============================================
  Files           965      978      +13     
  Lines         49063    49918     +855     
  Branches       4803     4848      +45     
============================================
+ Hits          34620    36689    +2069     
+ Misses        12137    10925    -1212     
+ Partials       2306     2304       -2     

Impacted Files	Coverage Δ	Complexity Δ
.../apache/hadoop/hdds/conf/HddsPrometheusConfig.java	50.00% <ø> (ø)	2.00 <0.00> (ø)
...op/hdds/utils/LegacyHadoopConfigurationSource.java	50.00% <ø> (ø)	5.00 <0.00> (ø)
.../java/org/apache/hadoop/ozone/OzoneConfigKeys.java	100.00% <ø> (ø)	1.00 <0.00> (ø)
...main/java/org/apache/hadoop/ozone/OzoneConsts.java	85.71% <ø> (+1.50%)	1.00 <0.00> (ø)
...g/apache/hadoop/hdds/conf/ConfigurationSource.java	22.72% <ø> (ø)	7.00 <0.00> (ø)
...rg/apache/hadoop/ozone/HddsDatanodeHttpServer.java	100.00% <ø> (+28.57%)	13.00 <0.00> (+4.00)
...mon/transport/server/ratis/XceiverServerRatis.java	88.10% <0.00%> (-4.50%)	62.00 <0.00> (+5.00)	⬇️
...iner/ozoneimpl/ContainerScrubberConfiguration.java	81.81% <ø> (-18.19%)	7.00 <0.00> (-1.00)
...org/apache/hadoop/hdds/server/http/HttpConfig.java	76.47% <0.00%> (ø)	1.00 <0.00> (ø)
.../hadoop/hdds/scm/container/ReplicationManager.java	87.35% <0.00%> (-0.70%)	102.00 <0.00> (ø)
... and 453 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25d3e52...2c381fa. Read the comment docs.

[ChenSammi]

LGTM + 1. Thanks @runzhiwang for the contribution and @xiaoyuyao for the review.