HDDS-11206. Statistics storage usage indicators include min, max, median, stdev #6977

jianghuazhu · 2024-07-22T11:04:13Z

What changes were proposed in this pull request?

For SCM, it is very useful to count the min, max, median, and stdev of node storage.
After adding the patch:
jmx:

ui:

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11206

How was this patch tested?

It is necessary to ensure that jmx can display NodeStatics correctly.
It is necessary to ensure that the UI can display Statistical correctly.

…ian, stdev

jianghuazhu · 2024-07-25T13:07:46Z

Can you help review this PR, @sumitagrawl @ArafatKhan2198 .
Thanks.

ivandika3

Thank for the patch @jianghuazhu , left a few comments.

ivandika3 · 2024-07-25T10:21:44Z

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManagerMXBean.java

@@ -48,4 +49,8 @@ public interface NodeManagerMXBean {
   */
  Map<String, Map<String, String>> getNodeStatusInfo();

+  default Map<String, String> getNodeStatics() {


Typo: Statics -> Statistics.

Also applies to other instances of Statics in the patch.

ivandika3 · 2024-07-25T10:22:35Z

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

+  private enum UsageStatics {
+    MIN("Min"),
+    MAX("Max"),
+    MEDINNA("Medina"),


Typo: MEDINA -> MEDIAN. Also applies to similar typos.

Thanks for your comment and review, @ivandika3 .
I'll update later.

hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm-overview.html

ivandika3 · 2024-07-26T02:56:53Z

hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm.js

+            $scope.statistical = {
+                usages : {}
+            }
+            $scope.statistical.usages.min = "";
+            $scope.statistical.usages.max = "";
+            $scope.statistical.usages.medina = "";
+            $scope.statistical.usages.stdev = "";


Nit: statistical -> statistics.

I think it's also better to be more specific that it is the statistics for DNs (since technically we can have statistics for SCM in the future). Maybe scope.statistics.nodes.usages. prefix is better.

I have updated it.

ivandika3

Thanks for updating the patch. LGTM +1.

ivandika3 · 2024-07-26T06:08:41Z

hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java

+    Map<String, String> nodeStatics = new HashMap<>();
+    // Statistics node usaged
+    nodeUsageStatics(nodeStatics);


Nit: Statics -> Stats / Statistics

ArafatKhan2198 · 2024-07-26T06:12:19Z

@jianghuazhu Can you please update the PR description with the latest screenshots.

devabhishekpal · 2024-07-26T06:42:27Z

Hello @jianghuazhu, thank you for this patch.
It is really helpful that we are able to see the statistics for storage usage.
I had one suggestion, if we could divide the data into different rows instead of a single row:

Statistical


DataNodes usages% (Min / Median / Max / stdDev)	11.79% / 52.96% / 95.03% / 22.91%

to

Statistics

Datanode Usage	in percentage (%)
Min	11.79
Median	52.96
Max	95.03
Standard Deviation	22.91

jianghuazhu · 2024-07-26T09:59:29Z

Thanks for your comments and reviews, @ArafatKhan2198 @devabhishekpal .
I updated the description of this PR, here are the latest UI and JMX.
JMX：

UI：

devabhishekpal · 2024-07-28T06:07:33Z

hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm.js

+            $scope.statistics = {
+                nodes : {
+                    usages : {}
+                }
+            }
+            $scope.statistics.nodes.usages.min = "";
+            $scope.statistics.nodes.usages.max = "";
+            $scope.statistics.nodes.usages.median = "";
+            $scope.statistics.nodes.usages.stdev = "";


Hi @jianghuazhu, was there any reason behind separating these values out?
I believe we can declare them in the statistics object.
Also I think initializing them to something like "N/A" would be better as in case there is any error, the default value won't be empty but rather show something to let the user know something went wrong.

Suggested change

$scope.statistics = {

nodes : {

usages : {}

}

}

$scope.statistics.nodes.usages.min = "";

$scope.statistics.nodes.usages.max = "";

$scope.statistics.nodes.usages.median = "";

$scope.statistics.nodes.usages.stdev = "";

$scope.statistics = {

nodes : {

usages : {

min: "N/A",

max: "N/A",

median: "N/A",

stdev: "N/A"

}

}

}

devabhishekpal · 2024-07-28T06:27:33Z

hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm.js

@@ -81,6 +90,18 @@
                    $scope.totalItems = nodeStatusCopy.length;
                    $scope.lastIndex = Math.ceil(nodeStatusCopy.length / $scope.RecordsToDisplay);
                    $scope.nodeStatus = nodeStatusCopy.slice(0, $scope.RecordsToDisplay);
+
+                    ctrl.nodemanagermetrics.NodeStatistics.map(({ key, value }) => {


While using a .map() will give us the same result, but it will also return an array (empty in this case) which we are ignoring as the value is not being assigned.
This might not be affecting the performance or anything, as it is a minor operation, but I would suggest to use .forEach in this scenario.

devabhishekpal

Thanks @jianghuazhu for your changes.
Just a few minor comments, and then it looks good to me.

devabhishekpal · 2024-07-29T06:42:35Z

hadoop-hdds/server-scm/src/main/resources/webapps/scm/scm.js

@@ -81,6 +91,18 @@
                    $scope.totalItems = nodeStatusCopy.length;
                    $scope.lastIndex = Math.ceil(nodeStatusCopy.length / $scope.RecordsToDisplay);
                    $scope.nodeStatus = nodeStatusCopy.slice(0, $scope.RecordsToDisplay);
+
+                    ctrl.nodemanagermetrics.NodeStatistics.forEach(function(obj) {


Just fyi, can de-structure this as well like you did for map().
Since obj = { key: abc, value: xyz }, we can destructure it as:
{key, value} = obj

So the function could also be written as:

ctrl.nodemanagermetrics.NodeStatistics.forEach(({key, value}) => { .... });

But your current change is good as well

devabhishekpal

Thanks a lot @jianghuazhu for your changes, this addition is great.
Thank you @ivandika3 for reviewing this PR
The changes LGTM, +1

jianghuazhu · 2024-07-29T11:52:07Z

Thanks for your comments and reviews, @devabhishekpal , @ivandika3 .

ivandika3 · 2024-07-29T11:53:35Z

Thank you for the patch @jianghuazhu, and @devabhishekpal @ArafatKhan2198 for the reviews.

HDDS-11206. Statistics storage usage indicators include min, max, med…

69a5215

…ian, stdev

ivandika3 reviewed Jul 26, 2024

View reviewed changes

Fix some code

e973a44

ivandika3 approved these changes Jul 26, 2024

View reviewed changes

Fix some UI styles

5840f02

devabhishekpal reviewed Jul 28, 2024

View reviewed changes

devabhishekpal suggested changes Jul 28, 2024

View reviewed changes

Update some javascript

307a772

devabhishekpal reviewed Jul 29, 2024

View reviewed changes

devabhishekpal approved these changes Jul 29, 2024

View reviewed changes

ivandika3 merged commit a532b89 into apache:master Jul 29, 2024
39 checks passed

ivandika3 added the UI label Aug 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDDS-11206. Statistics storage usage indicators include min, max, median, stdev #6977

HDDS-11206. Statistics storage usage indicators include min, max, median, stdev #6977

jianghuazhu commented Jul 22, 2024 •

edited

Loading

jianghuazhu commented Jul 25, 2024

ivandika3 left a comment

ivandika3 Jul 25, 2024

ivandika3 Jul 25, 2024

jianghuazhu Jul 26, 2024

ivandika3 Jul 26, 2024

jianghuazhu Jul 26, 2024

ivandika3 left a comment

ivandika3 Jul 26, 2024

ArafatKhan2198 commented Jul 26, 2024

devabhishekpal commented Jul 26, 2024 •

edited

Loading

jianghuazhu commented Jul 26, 2024

devabhishekpal Jul 28, 2024

devabhishekpal Jul 28, 2024

devabhishekpal left a comment

devabhishekpal Jul 29, 2024

devabhishekpal left a comment

jianghuazhu commented Jul 29, 2024

ivandika3 commented Jul 29, 2024

HDDS-11206. Statistics storage usage indicators include min, max, median, stdev #6977

HDDS-11206. Statistics storage usage indicators include min, max, median, stdev #6977

Conversation

jianghuazhu commented Jul 22, 2024 • edited Loading

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

jianghuazhu commented Jul 25, 2024

ivandika3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ivandika3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ArafatKhan2198 commented Jul 26, 2024

devabhishekpal commented Jul 26, 2024 • edited Loading

Statistical

Statistics

jianghuazhu commented Jul 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devabhishekpal left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

devabhishekpal left a comment

Choose a reason for hiding this comment

jianghuazhu commented Jul 29, 2024

ivandika3 commented Jul 29, 2024

jianghuazhu commented Jul 22, 2024 •

edited

Loading

devabhishekpal commented Jul 26, 2024 •

edited

Loading