Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-1503. Reduce garbage generated by non-netty threads in datanode ratis server #813

Merged
merged 2 commits into from
May 14, 2019

Conversation

lokeshj1703
Copy link
Contributor

We use GRPC protocol for rpc communication in Ratis. By default thread caches are generated even for non-netty threads. This Jira aims to add a default JVM parameter for disabling thread caches for non-netty threads in datanode ratis server.

Copy link
Contributor

@mukul1987 mukul1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, the patch looks good to me.
Lets add a link to the grpc-java or RATIS issue here for details.

@lokeshj1703
Copy link
Contributor Author

@mukul1987 I have added the corresponding Ratis issue link.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 837 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 460 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 839 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 441 the patch passed
+1 mvnsite 0 the patch passed
+1 shellcheck 26 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 737 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 unit 99 hadoop-hdds in the patch passed.
+1 unit 137 hadoop-ozone in the patch passed.
+1 asflicense 33 The patch does not generate ASF License warnings.
4694
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-813/1/artifact/out/Dockerfile
GITHUB PR #813
Optional Tests dupname asflicense mvnsite unit shellcheck shelldocs
uname Linux 720ca2584f5c 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 1a47c2b
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-813/1/testReport/
Max. process+thread count 306 (vs. ulimit of 5500)
modules C: hadoop-ozone/common U: hadoop-ozone/common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-813/1/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 143 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
0 shelldocs 0 Shelldocs was not available.
+1 @author 0 The patch does not contain any @author tags.
-1 test4tests 0 The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 mvninstall 435 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 844 branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 mvninstall 437 the patch passed
+1 mvnsite 0 the patch passed
+1 shellcheck 26 There were no new shellcheck issues.
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 723 patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 unit 99 hadoop-hdds in the patch passed.
+1 unit 136 hadoop-ozone in the patch passed.
+1 asflicense 34 The patch does not generate ASF License warnings.
3031
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-813/2/artifact/out/Dockerfile
GITHUB PR #813
Optional Tests dupname asflicense mvnsite unit shellcheck shelldocs
uname Linux 73d4705ab1fa 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / 1a47c2b
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-813/2/testReport/
Max. process+thread count 298 (vs. ulimit of 5500)
modules C: hadoop-ozone/common U: hadoop-ozone/common
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-813/2/console
versions git=2.7.4 maven=3.3.9 shellcheck=0.4.6
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

@xiaoyuyao
Copy link
Contributor

+1 from me too. Do we need to update ozone with a new Ratis release to take the newly added config?

@lokeshj1703
Copy link
Contributor Author

Do we need to update ozone with a new Ratis release to take the newly added config?

@xiaoyuyao That is not needed. The JVM parameter sets a static field in netty - PooledByteBufAllocator#DEFAULT_USE_CACHE_FOR_ALL_THREADS. Therefore it does not require new Ratis release.

@lokeshj1703 lokeshj1703 merged commit 6bcc1dc into apache:trunk May 14, 2019
@lokeshj1703 lokeshj1703 deleted the HDDS-1503 branch May 14, 2019 09:19
@lokeshj1703
Copy link
Contributor Author

@mukul1987 @xiaoyuyao Thanks for reviewing the PR! I have merged it to trunk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants