HDDS-2411. add a datanode chunk validator fo datanode chunk generator #1312
Conversation
elek
changed the title
There was a problem hiding this comment.
Thanks the patch @frischHWC (and sorry for the late review).
Based on the code it looks good to me. One minor comment: I prefer to remove the g from the name of the command (the current convention is using g for generate and v for validate). To make the merge faster I pushed this change (hope, you don't mind).
I am trying to test but I have some problem with my local env. Will merge it soon when I am able to test it...
| /** | ||
| * Data validator of chunks to use pure datanode XCeiver interface. | ||
| */ | ||
| @Command(name = "dcgv", |
There was a problem hiding this comment.
NIT: g usually used as an acronym of "validator", dcv might be a better abbrev.
|
One check unsuccessful as Ozone Freon testing on Minikube did not passed. |
No, it's not related, it's happening elsewhere (other PRs and commits), too. |
Yes, but it doesn't seem to be a false positive. I checked and somehow the SCM couldn't come out from the safe mode under 90 sec. Which is strange... (I mean unrelated, but not a test problem) |
|
@frischHWC Would you be so kind to add ANY comment to the issue? (I need your Jira username to add you to the project to make it possible to be assigned.) |
|
@elek : my jira username: frisch. |
* master: (26 commits) HDDS-4167. Acceptance test logs missing if fails during cluster startup (apache#1366) HDDS-4121. Implement OmMetadataMangerImpl#getExpiredOpenKeys. (apache#1351) HDDS-3867. Extend the chunkinfo tool to display information from all nodes in the pipeline. (apache#1154) HDDS-4077. Incomplete OzoneFileSystem statistics (apache#1329) HDDS-3903. OzoneRpcClient support batch rename keys. (apache#1150) HDDS-4151. Skip the inputstream while offset larger than zero in s3g (apache#1354) HDDS-4147. Add OFS to FileSystem META-INF (apache#1352) HDDS-4137. Turn on the verbose mode of safe mode check on testlib (apache#1343) HDDS-4146. Show the ScmId and ClusterId in the scm web ui. (apache#1350) HDDS-4145. Bump version to 1.1.0-SNAPSHOT on master (apache#1349) HDDS-4109. Tests in TestOzoneFileSystem should use the existing MiniOzoneCluster (apache#1316) HDDS-4149. Implement OzoneFileStatus#toString (apache#1356) HDDS-4153. Increase default timeout in kubernetes tests (apache#1357) HDDS-2411. add a datanode chunk validator fo datanode chunk generator (apache#1312) HDDS-4140. Auto-close /pending pull requests after 21 days of inactivity (apache#1344) HDDS-4152. Archive container logs for kubernetes check (apache#1355) HDDS-4056. Convert OzoneAdmin to pluggable model (apache#1285) HDDS-3972. Add option to limit number of items displaying through ldb tool. (apache#1206) HDDS-4068. Client should not retry same OM on network connection failure (apache#1324) HDDS-4062. Non rack aware pipelines should not be created if multiple racks are alive. (apache#1291) ...
What changes were proposed in this pull request?
Solve HDDS-2411 by adding a Datanode chunk Validator for Ozone Freon test.
Note that a new checksum is computed for each chunk.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2411
How was this patch tested?
Manual tests were made by setting up a docker-compose cluster of 3 nodes (they are required as generator check for a pipeline with a factor THREE).
A login to a node with following commands:
ozone freon rk --replicationType RATIS --factor THREE
ozone freon dcg -p tester
ozone freon dcgv -p tester
To show that test was well working, a change in chunk size for validator involves (as expected) a failure:
ozone freon dcg -p test -s 1000