HDDS-6000. Freon datanode chunk validator fails checksum validation

[adoroszlai]

What changes were proposed in this pull request?

Datanode chunk validator was failing because datanode would not return the requested chunk due to bcsID mismatch (check added in HDDS-4986):

StorageContainerException: Unable to find the block with bcsID 2 .Container 1 bcsId is 0.
   at org.apache.hadoop.ozone.container.keyvalue.helpers.BlockUtils.verifyBCSId(BlockUtils.java:206)
   at org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handleReadChunk(KeyValueHandler.java:604)

https://issues.apache.org/jira/browse/HDDS-6000

How was this patch tested?

Added smoketest.

$ ozone freon dcg -p dcg
[main] INFO freon.BaseFreonGenerator: Executing test with prefix dcg
[Thread-4] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1000)
...
[Thread-4] INFO freon.ProgressBar: Progress: 100.00 % (1000 out of 1000)
[shutdown-hook-0] INFO metrics: type=TIMER, name=chunk-write, count=1000, min=28.5967, max=1469.2974, mean=111.2364816185868, stddev=132.9111794457411, median=92.822, p75=119.2345, p95=178.0347, p98=194.6324, p99=222.7369, p999=1462.9769, mean_rate=81.13738779672916, m1=56.79755719538722, m5=53.30907003981104, m15=52.70470835973068, rate_unit=events/second, duration_unit=milliseconds
[shutdown-hook-0] INFO freon.BaseFreonGenerator: Total execution time (sec): 13
[shutdown-hook-0] INFO freon.BaseFreonGenerator: Failures: 0
[shutdown-hook-0] INFO freon.BaseFreonGenerator: Successful executions: 1000

$ ozone freon dcv -p dcg
[main] INFO freon.BaseFreonGenerator: Executing test with prefix dcg
[Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1000)
[main] INFO freon.DatanodeChunkValidator: Using pipeline PipelineID=f9081b03-f892-42da-ba5f-35a5d7d1d1eb
[Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0 out of 1000)
...
[Thread-3] INFO freon.ProgressBar: Progress: 100.00 % (1000 out of 1000)
[shutdown-hook-0] INFO metrics: type=TIMER, name=chunk-validate, count=1000, min=4.8452, max=64.9278, mean=15.33763111608525, stddev=7.410829411618411, median=13.1337, p75=18.5535, p95=29.2186, p98=34.7298, p99=41.1294, p999=61.8825, mean_rate=476.15807476719795, m1=0.0, m5=0.0, m15=0.0, rate_unit=events/second, duration_unit=milliseconds
[shutdown-hook-0] INFO freon.BaseFreonGenerator: Total execution time (sec): 4
[shutdown-hook-0] INFO freon.BaseFreonGenerator: Failures: 0
[shutdown-hook-0] INFO freon.BaseFreonGenerator: Successful executions: 1000

https://github.com/adoroszlai/hadoop-ozone/actions/runs/1475697452

[sodonnel]

These changes LGTM

[adoroszlai]

Thanks @sodonnel for the review.