HDDS-3642. Stop/Pause Background services while replacing OM DB with checkpoint from Leader

[hanishakoneru]

What changes were proposed in this pull request?

When a follower OM needs to replace its DB with a checkpoint from Leader (to catch up on the transactions), it should pause or stop services which read/ write to the DB.

During OM HA testing, found that OM could crash with JVM error on RocksDB. This happened because KeyDeletingService was trying to access a memory which is already freed up.

Please see Jira link below for error logs.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-3642

How was this patch tested?

Tested on a local docker cluster.

[bharatviswa504]

@hanishakoneru This PR needs a rebase.

[bharatviswa504]

Overall LGTM.
One minor comment, and it needs a rebase.

[bharatviswa504]

Right now, we don't have any stop() for S3SecretManager and PrefixManager.
And also for these, only read/write operations will happen. Write will go through double buffer and read's will not happen anyway as this is not leader. Do we still see issue?

[hanishakoneru]

If writes can happen then we should stop it, right?
But writes would have to come through Ratis server which will be paused. So it should be okay.

[bharatviswa504]

Thanks for confirming.
Then we can remove the comments.

[bharatviswa504]

Looks like it is failing in compilation and also check style issues.

[codecov-commenter]

Codecov Report

Merging #1002 into master will decrease coverage by 0.01%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #1002      +/-   ##
============================================
- Coverage     69.46%   69.45%   -0.02%     
+ Complexity     9121     9118       -3     
============================================
  Files           961      961              
  Lines         48151    48158       +7     
  Branches       4679     4679              
============================================
- Hits          33448    33447       -1     
- Misses        12489    12495       +6     
- Partials       2214     2216       +2     

Impacted Files	Coverage Δ	Complexity Δ
.../java/org/apache/hadoop/ozone/om/OzoneManager.java	64.27% <0.00%> (-0.32%)	186.00 <0.00> (ø)
...otocol/commands/RetriableDatanodeEventWatcher.java	55.55% <0.00%> (-44.45%)	3.00% <0.00%> (-1.00%)
...er/common/transport/server/GrpcXceiverService.java	70.00% <0.00%> (-10.00%)	3.00% <0.00%> (ø%)
...apache/hadoop/hdds/scm/block/BlockManagerImpl.java	66.66% <0.00%> (-5.41%)	18.00% <0.00%> (ø%)
...apache/hadoop/hdds/server/events/EventWatcher.java	77.77% <0.00%> (-4.17%)	14.00% <0.00%> (ø%)
...va/org/apache/hadoop/ozone/lease/LeaseManager.java	90.80% <0.00%> (-2.30%)	15.00% <0.00%> (-1.00%)
...hdds/scm/container/common/helpers/ExcludeList.java	95.91% <0.00%> (-2.05%)	23.00% <0.00%> (-1.00%)
...ine/commandhandler/DeleteBlocksCommandHandler.java	62.50% <0.00%> (-1.57%)	9.00% <0.00%> (-2.00%)
.../apache/hadoop/ozone/client/io/KeyInputStream.java	78.19% <0.00%> (-1.51%)	42.00% <0.00%> (-1.00%)
.../statemachine/background/BlockDeletingService.java	75.73% <0.00%> (-1.48%)	12.00% <0.00%> (-1.00%)
... and 16 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cee76a8...37ff3de. Read the comment docs.

[bharatviswa504]

+1 LGTM.
Thank You @hanishakoneru for the contribution

[bharatviswa504]

Kicked off CI run, test has not run due to unavailability of artificats.

[bharatviswa504]

Thank You @hanishakoneru for the contribution.