HADOOP-17999: Do not initialize all target FileSystems for setWriteChecksum and setVerifyChecksum #3639
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ecksum and setVerifyChecksum
|
🎊 +1 overall
This message was automatically generated. |
…hecksum in ViewFileSystem The same has already been done in ViewFs.
|
💔 -1 overall
This message was automatically generated. |
shvachko
reviewed
Nov 15, 2021
| for (InodeTree.MountPoint<FileSystem> mount : mountPoints) { | ||
| fsMap.get(mount.src).setVerifyChecksum(verifyChecksum); | ||
| } | ||
| public void setVerifyChecksum(final boolean verifyChecksum) { |
There was a problem hiding this comment.
Looks like the default impl. in FileSystem is already no-op. So the value of overriding is only in the comment. May be we should just remove this methods and add a comment to the jira explaining the change.
There was a problem hiding this comment.
We still want to keep overriding the two methods in ViewFileSystem (same as ViewFs) so that we explicitly say these are no-ops and doesn't depend on the FileSystem implementation (if anything changes in future)
|
🎊 +1 overall
This message was automatically generated. |
|
+1 for the last patch. |
asfgit
pushed a commit
that referenced
this pull request
Nov 17, 2021
…hecksum in ViewFileSystem. Contributed by Abhishek Das. (#3639)
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
…hecksum in ViewFileSystem. Contributed by Abhishek Das. (apache#3639)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: https://issues.apache.org/jira/browse/HADOOP-17999
Description of PR
All the target filesystems corresponding to mount points are initialized in setWriteChecksum and setVerifyChecksum methods. This PR initializes the filesystems lazily
How was this patch tested?
Added unit tests. Also for large number of mount points, hadoop cli commands such as get, put or copyFromLocal were throwing OOM. With the fix, the commands were successful.
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?