[SPARK-23432][UI] Add executor peak jvm memory metrics in executors page #30186

warrenzhu25 · 2020-10-29T18:16:16Z

What changes were proposed in this pull request?

Add executor peak jvm memory metrics in executors page

Why are the changes needed?

Users can know executor peak jvm metrics on in executors page

Does this PR introduce any user-facing change?

Users can know executor peak jvm metrics on in executors page

How was this patch tested?

Manually tested

tgravescs · 2020-10-29T19:11:38Z

can you provide screen shots?

dongjoon-hyun · 2020-10-30T17:25:02Z

ok to test

dongjoon-hyun · 2020-10-30T17:28:35Z

cc @sarutak and @gengliangwang

tgravescs · 2020-10-30T17:38:21Z

core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html

+          <th>JVM Memory OnHeap / OffHeap</th>
+          <th>Execution Memory OnHeap / OffHeap</th>
+          <th>Storage Memory OnHeap / OffHeap</th>
+          <th>Pool Memory Direct / Mapped</th>


add tooltips describing what these are

tgravescs · 2020-10-30T17:47:50Z

core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html

@@ -86,6 +86,10 @@ <h4 class="title-table">Executors</h4>
            <span data-toggle="tooltip" data-placement="top"
                  title="Memory used / total available memory for off heap storage of data like RDD partitions cached in memory.">
              Off Heap Storage Memory</span></th>
+          <th>JVM Memory OnHeap / OffHeap</th>
+          <th>Execution Memory OnHeap / OffHeap</th>
+          <th>Storage Memory OnHeap / OffHeap</th>


this is very close to the above On/Off Heap Storage Memory - so I think we need to differentiate the titles more. I think that goes for all of these, if they are Peak we should call them Peak.

Added Peak for all.

SparkQA · 2020-10-30T18:14:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35068/

SparkQA · 2020-10-30T18:39:29Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35068/

SparkQA · 2020-10-30T19:59:50Z

Test build #130466 has finished for PR 30186 at commit f2c556f.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-10-31T02:01:33Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35083/

SparkQA · 2020-10-31T02:30:15Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35083/

SparkQA · 2020-10-31T03:57:16Z

Test build #130479 has finished for PR 30186 at commit 92d1112.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs · 2020-11-02T16:08:38Z

core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html

+              Peak Execution Memory OnHeap / OffHeap</span></th>
+          <th>
+            <span data-toggle="tooltip" data-placement="top"
+            title="Peak storage onHeap / OffHeap memory used by JVM.">


I think we should expand this to be "Peak onHeap/ OffHeap memory used for storage of data like RDD partitions cached in memory"

tgravescs · 2020-11-02T16:14:04Z

core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html

+              Peak JVM Memory OnHeap / OffHeap</span></th>
+          <th>
+            <span data-toggle="tooltip" data-placement="top"
+                  title="Peak execution onHeap / OffHeap memory used by JVM.">


Peak OnHeap/OffHeap memory used for execution. This refers to memory used for computation in shuffles, joins, user data structures, etc. See the Memory Management Overview documentation for more details.

tgravescs · 2020-11-02T16:17:00Z

core/src/main/resources/org/apache/spark/ui/static/executorspage-template.html

+              Peak Storage Memory OnHeap / OffHeap</span></th>
+          <th>
+            <span data-toggle="tooltip" data-placement="top"
+                  title="Peak direct / mapped pool memory used by JVM.">


Similarly I think we should explain more or point to java docs.
At least say this is direct byte buffers and mapped are memory-mapped or perhaps better would be to point to java.nio:type=BufferPool,name=direct and java.nio:type=BufferPool,name=mapped

Good point. Updated.

SparkQA · 2020-11-03T02:20:47Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35138/

SparkQA · 2020-11-03T02:41:23Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35138/

SparkQA · 2020-11-03T03:45:20Z

Test build #130538 has finished for PR 30186 at commit fc8fe85.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-03T04:30:42Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35147/

SparkQA · 2020-11-03T04:58:55Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35147/

SparkQA · 2020-11-03T06:35:35Z

Test build #130548 has finished for PR 30186 at commit 28cfbdf.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-04T01:07:32Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35181/

SparkQA · 2020-11-04T01:36:46Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35181/

SparkQA · 2020-11-04T03:07:09Z

Test build #130580 has finished for PR 30186 at commit 30da8f1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

tgravescs

looks good pending Jenkins

HeartSaVioR · 2020-11-06T02:45:27Z

retest this, please

SparkQA · 2020-11-06T03:28:41Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35293/

SparkQA · 2020-11-06T03:57:44Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35293/

SparkQA · 2020-11-06T05:27:41Z

Test build #130682 has finished for PR 30186 at commit 30da8f1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR

+1

HeartSaVioR · 2020-11-06T07:56:33Z

Thanks! Merged into master.
@warrenzhu25 Could you please let me know your JIRA ID? I'm not 100% sure so I left the assignee as unassigned. Once I get the information I'll update it.

HeartSaVioR · 2020-11-06T07:58:55Z

Never mind. I tried your Github account and found it. Please let me know if I misassigned.

warrenzhu25 · 2020-11-06T16:10:22Z

@HeartSaVioR thanks, it's correct.

warrenzhu25 force-pushed the 23432 branch from bf899db to f2c556f Compare October 29, 2020 19:37

warrenzhu25 changed the title ~~[SPARK-23432][UI]Add executor peak jvm memory metrics in executors page~~ [SPARK-23432][UI] Add executor peak jvm memory metrics in executors page Oct 30, 2020

tgravescs reviewed Oct 30, 2020

View reviewed changes

warrenzhu25 force-pushed the 23432 branch from f2c556f to 92d1112 Compare October 31, 2020 01:19

tgravescs reviewed Nov 2, 2020

View reviewed changes

[SPARK-23432][UI]Add executor peak jvm memory metrics in executors page

fc8fe85

warrenzhu25 force-pushed the 23432 branch from 92d1112 to fc8fe85 Compare November 3, 2020 01:34

Trigger Build

28cfbdf

Trigger Build

30da8f1

tgravescs approved these changes Nov 4, 2020

View reviewed changes

HeartSaVioR approved these changes Nov 6, 2020

View reviewed changes

HeartSaVioR closed this in 93ad26b Nov 6, 2020

gengliangwang mentioned this pull request Nov 26, 2020

[SPARK-33562][UI] Improve the style of the checkbox in executor page #30500

Closed

dongjoon-hyun mentioned this pull request Dec 24, 2020

[SPARK-33906][WEBUI] Fix the bug of UI Executor page stuck due to undefined peakMemoryMetrics #30920

Closed

warrenzhu25 deleted the 23432 branch September 29, 2021 18:14

[SPARK-23432][UI] Add executor peak jvm memory metrics in executors page #30186

[SPARK-23432][UI] Add executor peak jvm memory metrics in executors page #30186

Conversation

warrenzhu25 commented Oct 29, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

tgravescs commented Oct 29, 2020

dongjoon-hyun commented Oct 30, 2020

dongjoon-hyun commented Oct 30, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Oct 30, 2020

SparkQA commented Oct 30, 2020

SparkQA commented Oct 30, 2020

SparkQA commented Oct 31, 2020

SparkQA commented Oct 31, 2020

SparkQA commented Oct 31, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Nov 3, 2020

SparkQA commented Nov 3, 2020

SparkQA commented Nov 3, 2020

SparkQA commented Nov 3, 2020

SparkQA commented Nov 3, 2020

SparkQA commented Nov 3, 2020

SparkQA commented Nov 4, 2020

SparkQA commented Nov 4, 2020

SparkQA commented Nov 4, 2020

tgravescs left a comment

Choose a reason for hiding this comment

HeartSaVioR commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

SparkQA commented Nov 6, 2020

HeartSaVioR left a comment

Choose a reason for hiding this comment

HeartSaVioR commented Nov 6, 2020

HeartSaVioR commented Nov 6, 2020

warrenzhu25 commented Nov 6, 2020

warrenzhu25 commented Oct 29, 2020 •

edited

Loading