Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDDS-1750. Add block allocation metrics for pipelines in SCM #1047

Merged
merged 2 commits into from
Jul 9, 2019

Conversation

lokeshj1703
Copy link
Contributor

This Jira aims to add block allocation metrics for pipelines in SCM. This would help in determining the distribution of block allocations among various pipelines in SCM.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 31 Docker mode activated.
_ Prechecks _
+1 dupname 1 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 69 Maven dependency ordering for branch
+1 mvninstall 475 trunk passed
+1 compile 241 trunk passed
+1 checkstyle 65 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 800 branch has no errors when building and testing our client artifacts.
+1 javadoc 147 trunk passed
0 spotbugs 323 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 509 trunk passed
_ Patch Compile Tests _
0 mvndep 32 Maven dependency ordering for patch
+1 mvninstall 454 the patch passed
+1 compile 262 the patch passed
+1 javac 262 the patch passed
+1 checkstyle 68 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 649 patch has no errors when building and testing our client artifacts.
+1 javadoc 156 the patch passed
+1 findbugs 568 the patch passed
_ Other Tests _
+1 unit 251 hadoop-hdds in the patch passed.
-1 unit 2609 hadoop-ozone in the patch failed.
+1 asflicense 39 The patch does not generate ASF License warnings.
7577
Reason Tests
Failed junit tests hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
hadoop.ozone.om.TestOzoneManager
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient
hadoop.ozone.TestMiniOzoneCluster
hadoop.ozone.container.server.TestSecureContainerServer
hadoop.hdds.scm.pipeline.TestRatisPipelineProvider
hadoop.ozone.container.metrics.TestContainerMetrics
hadoop.ozone.TestStorageContainerManager
hadoop.ozone.client.rpc.TestBlockOutputStream
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestCommitWatcher
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1047/1/artifact/out/Dockerfile
GITHUB PR #1047
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux 16659bc524d1 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / e966edd
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1047/1/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1047/1/testReport/
Max. process+thread count 3787 (vs. ulimit of 5500)
modules C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1047/1/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@mukul1987 mukul1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @lokeshj1703. The patch generally looks good to me.

@@ -152,6 +152,7 @@ public synchronized Pipeline createPipeline(
stateManager.addPipeline(pipeline);
nodeManager.addPipeline(pipeline);
metrics.incNumPipelineCreated();
metrics.createNumBlocksAllocatedMetric(pipeline);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can line 154 and 155 be done in one state to pipelineMetrics ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be named as createPerPipelineMetrics or something like that :)

@@ -362,6 +364,7 @@ private void finalizePipeline(PipelineID pipelineId) throws IOException {
for (ContainerID containerID : containerIDs) {
eventPublisher.fireEvent(SCMEvents.CLOSE_CONTAINER, containerID);
}
metrics.clearMetrics(pipelineId);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets rename this to removePipelineMetrics

@lokeshj1703
Copy link
Contributor Author

@mukul1987 Thanks for reviewing the PR! I have addresses the review commits in the latest commit.

@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Comment
0 reexec 29 Docker mode activated.
_ Prechecks _
+1 dupname 0 No case conflicting files found.
+1 @author 0 The patch does not contain any @author tags.
+1 test4tests 0 The patch appears to include 1 new or modified test files.
_ trunk Compile Tests _
0 mvndep 63 Maven dependency ordering for branch
+1 mvninstall 476 trunk passed
+1 compile 250 trunk passed
+1 checkstyle 68 trunk passed
+1 mvnsite 0 trunk passed
+1 shadedclient 856 branch has no errors when building and testing our client artifacts.
+1 javadoc 145 trunk passed
0 spotbugs 304 Used deprecated FindBugs config; considering switching to SpotBugs.
+1 findbugs 496 trunk passed
_ Patch Compile Tests _
0 mvndep 36 Maven dependency ordering for patch
+1 mvninstall 446 the patch passed
+1 compile 269 the patch passed
+1 javac 269 the patch passed
+1 checkstyle 72 the patch passed
+1 mvnsite 0 the patch passed
+1 whitespace 0 The patch has no whitespace issues.
+1 shadedclient 671 patch has no errors when building and testing our client artifacts.
+1 javadoc 158 the patch passed
+1 findbugs 514 the patch passed
_ Other Tests _
+1 unit 259 hadoop-hdds in the patch passed.
-1 unit 1796 hadoop-ozone in the patch failed.
+1 asflicense 46 The patch does not generate ASF License warnings.
6812
Reason Tests
Failed junit tests hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures
hadoop.ozone.client.rpc.TestOzoneAtRestEncryption
hadoop.ozone.client.rpc.TestOzoneRpcClient
hadoop.hdds.scm.pipeline.TestRatisPipelineProvider
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion
hadoop.ozone.client.rpc.TestSecureOzoneRpcClient
hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis
Subsystem Report/Notes
Docker Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1047/2/artifact/out/Dockerfile
GITHUB PR #1047
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
uname Linux bcbd8d8691c2 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/hadoop.sh
git revision trunk / ec851e4
Default Java 1.8.0_212
unit https://builds.apache.org/job/hadoop-multibranch/job/PR-1047/2/artifact/out/patch-unit-hadoop-ozone.txt
Test Results https://builds.apache.org/job/hadoop-multibranch/job/PR-1047/2/testReport/
Max. process+thread count 4452 (vs. ulimit of 5500)
modules C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: .
Console output https://builds.apache.org/job/hadoop-multibranch/job/PR-1047/2/console
versions git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1
Powered by Apache Yetus 0.10.0 http://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@mukul1987 mukul1987 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, the changes looks good to me.

@mukul1987 mukul1987 merged commit b5d30e4 into apache:trunk Jul 9, 2019
bshashikant pushed a commit to bshashikant/hadoop that referenced this pull request Jul 10, 2019
shanthoosh pushed a commit to shanthoosh/hadoop that referenced this pull request Oct 15, 2019
… doing reflection (apache#1047)

* SAMZA-2210: Initial majority migration for injecting classloader when doing reflection
* apply reordering of arguments for ReflectionUtil methods
* rename test method
* fixing issues after merge
* use buildContainerAllocator in ContainerProcessManager after merge
* clean up variables and naming according to PR comments
amahussein pushed a commit to amahussein/hadoop that referenced this pull request Oct 29, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants