HDDS-6971. EC: Add EC block checksum computer.#3790
Merged
umamaheswararao merged 8 commits intoapache:masterfrom Nov 16, 2022
Merged
HDDS-6971. EC: Add EC block checksum computer.#3790umamaheswararao merged 8 commits intoapache:masterfrom
umamaheswararao merged 8 commits intoapache:masterfrom
Conversation
Contributor
Contributor
umamaheswararao
left a comment
umamaheswararao
left a comment
There was a problem hiding this comment.
@aswinshakil thank you for working on this patch. I have reviewed this patch, please find my initial comments on this patch.
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ECBlockOutputStream.java
Outdated
Show resolved
Hide resolved
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ECBlockOutputStream.java
Show resolved
Hide resolved
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ECBlockOutputStream.java
Show resolved
Hide resolved
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/ECBlockOutputStream.java
Outdated
Show resolved
Hide resolved
...ain/java/org/apache/hadoop/ozone/container/ec/reconstruction/ECContainerOperationClient.java
Outdated
Show resolved
Hide resolved
...p-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECBlockOutputStreamEntry.java
Show resolved
Hide resolved
...one/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
Outdated
Show resolved
Hide resolved
...one/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestRootedOzoneFileSystem.java
Outdated
Show resolved
Hide resolved
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientUtils.java
Show resolved
Hide resolved
...ne/client/src/main/java/org/apache/hadoop/ozone/client/checksum/ECBlockChecksumComputer.java
Show resolved
Hide resolved
Contributor
|
Mostly changes looks ok to me. @jojochuang could you please take a final look? Thanks |
jojochuang
reviewed
Nov 15, 2022
Contributor
jojochuang
left a comment
jojochuang
left a comment
There was a problem hiding this comment.
Overall looks good. A few suggestions for future improvements.
...p-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileChecksum.java
Show resolved
Hide resolved
...p-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECBlockOutputStreamEntry.java
Show resolved
Hide resolved
...p-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/ECBlockOutputStreamEntry.java
Outdated
Show resolved
Hide resolved
...ozone/client/src/main/java/org/apache/hadoop/ozone/client/checksum/ECFileChecksumHelper.java
Show resolved
Hide resolved
...p-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileChecksum.java
Outdated
Show resolved
Hide resolved
Contributor
|
Thanks @aswinshakil for working on this. Could you also file doc JIRA for updating the docs with EC constraints? Ex: what is comparable between EC and Ratis and HDFS etc. |
umamaheswararao
approved these changes
Nov 15, 2022
Member
Author
|
Created DOC JIRA HDDS-7491 for EC FileChecksum |
jojochuang
approved these changes
Nov 16, 2022
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
EC keys do not have a way to provide checksum for
getFileChecksumAPI. This JIRA provides an implementation for EC Checksum which can also be used for Data Integrity check later.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6971
TODO
How was this patch tested?
This patch was tested manually and using integration tests.