[BEAM-11213] Display Beam Metrics in Spark History Server #13743

ghost · 2021-01-13T11:52:12Z

Create event log following Spark History Server format in default /tmp/spark-events log directory, so it could be read by Spark History Server.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

tysonjh · 2021-01-13T16:02:55Z

R: @ibzib

ibzib

Thanks @tszerszen! A couple questions:

Does this PR affect the regular Spark UI at all, or does it only show metrics in the history server?
These changes only apply to the portable Spark runner. What about the "classic"/non-portable version?

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

runners/spark/job-server/build.gradle

ibzib · 2021-01-20T22:17:46Z

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java

@@ -34,6 +34,12 @@
 */
 public interface SparkPipelineOptions extends SparkCommonPipelineOptions {

+  @Description("The directory to save Spark History Server logs")
+  @Default.String("/tmp/spark-events/")


Should we set spark.eventLog.dir in the Spark conf? Or does that not matter?

beam/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkContextFactory.java

Line 88 in d1c8c24

SparkConf conf = new SparkConf();

It doesn't matter, however for consistency I think it would be good to configure it in such a way.

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

ghost · 2021-01-24T17:42:13Z

Run Java PreCommit

ghost · 2021-01-24T19:24:34Z

R: @ibzib

runners/spark/job-server/build.gradle

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineOptions.java

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/SparkBeamMetric.java

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

ibzib · 2021-01-26T03:38:09Z

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

+                }
+              }));
+      eventLoggingListener.onApplicationEnd(
+          new SparkListenerApplicationEnd(Instant.now().getMillis()));


I'm pretty sure pipeline end time (and also start time for that matter) is itself a metric. To keep things consistent, it'd be better to use that metric here instead of Instant.now().

When I printed out the results of renderAll method I didn't found such metrics for whole pipeline only for it's parts. Maybe not all metrics appear in renderAll method or should I filter for them specifically?

I'm not sure, it's not a blocker for this PR though. Thanks for checking.

ghost · 2021-01-27T16:46:40Z

R: @ibzib

ibzib

Thanks @tszerszen, this is looking a lot better. I have a few more comments.

runners/spark/job-server/build.gradle

ibzib · 2021-01-27T20:43:28Z

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

+                }
+              }));
+      eventLoggingListener.onApplicationEnd(
+          new SparkListenerApplicationEnd(Instant.now().getMillis()));


I'm not sure, it's not a blocker for this PR though. Thanks for checking.

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

runners/spark/src/test/java/org/apache/beam/runners/spark/metrics/SparkBeamMetricTest.java

runners/spark/src/main/java/org/apache/beam/runners/spark/metrics/SparkBeamMetric.java

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

iemejia · 2021-01-27T21:35:29Z

I have not checked what is the current status on the classic runners, are the Beam metrics shown correctly? Do you think we can reuse this work (if they are not shown correctly) there too @tszerszen ?

ghost · 2021-01-28T12:25:03Z

@iemejia I think this work can be reused there, since it's calling native Spark EventLoggingListener.

iemejia · 2021-01-28T12:43:23Z

Great to know @tszerszen, thanks . Sounds like a nice follow up issue to be created/fixed in case you feel motivated after this one is merged.

ghost · 2021-01-30T07:58:58Z

R: @ibzib

ibzib

Thanks Tomasz, I think this will be good to go after addressing a couple things:

Remove the new options from the Spark job server configuration, and all related code. (getSparkHistoryDir and getEventLogEnabled should only be pipeline options.)
Fix the SparkListenerExecutorAdded logic.

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

…stener executor added logic

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkCommonPipelineOptions.java

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

…stener executor added logic

ibzib

Some minor cleanup and then we can merge this.

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java

ghost · 2021-02-03T14:28:30Z

Run Java PreCommit

ghost · 2021-02-03T15:40:28Z

Run Java PreCommit

ghost · 2021-02-03T17:30:56Z

R: @ibzib

ibzib

LGTM, thank you!

ibzib · 2021-02-03T18:57:20Z

Run Java PreCommit

ghost · 2021-02-03T21:22:17Z

Run Java PreCommit

ibzib · 2021-02-03T22:15:03Z

Java test failure seems to be a flake: BEAM-11746

ibzib · 2021-02-03T22:15:09Z

Run Java PreCommit

Tomasz Szerszeń added 2 commits January 13, 2021 12:48

Create Spark Metrics in directory using Spark History Server format

291ced1

java spotless apply

f044ec1

ghost changed the title ~~[BEAM-11213] Beam metrics should be displayed in Spark UI~~ [BEAM-11213] Display Beam Metrics in Spark History Server Jan 13, 2021

Tomasz Szerszeń added 4 commits January 14, 2021 10:42

add --spark-history-dir option in spark job server

c578c84

add driver distribution logs

cb1b46f

remove start import check

864603a

spotless apply

f0c7412

ibzib reviewed Jan 20, 2021

View reviewed changes

Tomasz Szerszeń added 6 commits January 24, 2021 15:13

render all metrics & add eventLogEnabled

ad359f5

add exception when eventLogEnabled is true

bb7c05e

handle eventLogEnabled

cac34a7

apply spotless

e669894

remove star import

8be02d6

run java spotless

5392f66

fix renderName is static from now on

1da04b4

ibzib requested changes Jan 26, 2021

View reviewed changes

Tomasz Szerszeń added 2 commits January 27, 2021 16:01

spark.executor.id loop and switch to boolean

9bec263

spotless apply

31f86b8

ibzib reviewed Jan 27, 2021

View reviewed changes

remove uncessary code

c98a89e

scope.Option.apply

6529c7a

ibzib reviewed Jan 30, 2021

View reviewed changes

runners/spark/src/main/java/org/apache/beam/runners/spark/SparkPipelineRunner.java Outdated Show resolved Hide resolved

Tomasz Szerszeń added 3 commits January 31, 2021 12:09

remove options from spark job server configuration & fix the spark li…

03c2457

…stener executor added logic

remove options from spark job server configuration & fix the spark li…

d3b7838

…stener executor added logic

remove options from spark job server configuration & fix the spark li…

ba68363

…stener executor added logic

ibzib reviewed Jan 31, 2021

View reviewed changes

Tomasz Szerszeń added 3 commits January 31, 2021 18:48

remove options from spark job server configuration & fix the spark li…

6acf592

…stener executor added logic

remove options from spark job server configuration & fix the spark li…

a513fbf

…stener executor added logic

remove options from spark job server configuration & fix the spark li…

f383e17

…stener executor added logic

ibzib reviewed Feb 2, 2021

View reviewed changes

Tomasz Szerszeń added 2 commits February 3, 2021 14:44

minor cleanup

e105b9f

minor cleanup

068a33e

minor improvments

b35c282

Tomasz Szerszeń added 2 commits February 3, 2021 18:25

remove whitespace changes

4b64b5f

remove whitespace changes

6329b57

ibzib approved these changes Feb 3, 2021

View reviewed changes

ibzib merged commit 654ad2b into apache:master Feb 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BEAM-11213] Display Beam Metrics in Spark History Server #13743

[BEAM-11213] Display Beam Metrics in Spark History Server #13743

ghost commented Jan 13, 2021 •

edited by ghost

tysonjh commented Jan 13, 2021

ibzib left a comment

ibzib Jan 20, 2021

ghost Jan 24, 2021

ghost commented Jan 24, 2021

ghost commented Jan 24, 2021

ibzib Jan 26, 2021

ghost Jan 27, 2021 •

edited by ghost

ibzib Jan 27, 2021

ghost commented Jan 27, 2021

ibzib left a comment

ibzib Jan 27, 2021

iemejia commented Jan 27, 2021

ghost commented Jan 28, 2021

iemejia commented Jan 28, 2021

ghost commented Jan 30, 2021

ibzib left a comment

ibzib left a comment

ghost commented Feb 3, 2021

ghost commented Feb 3, 2021

ghost commented Feb 3, 2021

ibzib left a comment

ibzib commented Feb 3, 2021

ghost commented Feb 3, 2021

ibzib commented Feb 3, 2021

ibzib commented Feb 3, 2021

[BEAM-11213] Display Beam Metrics in Spark History Server #13743

[BEAM-11213] Display Beam Metrics in Spark History Server #13743

Conversation

ghost commented Jan 13, 2021 • edited by ghost

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

tysonjh commented Jan 13, 2021

ibzib left a comment

Choose a reason for hiding this comment

ibzib Jan 20, 2021

Choose a reason for hiding this comment

ghost Jan 24, 2021

Choose a reason for hiding this comment

ghost commented Jan 24, 2021

ghost commented Jan 24, 2021

ibzib Jan 26, 2021

Choose a reason for hiding this comment

ghost Jan 27, 2021 • edited by ghost

Choose a reason for hiding this comment

ibzib Jan 27, 2021

Choose a reason for hiding this comment

ghost commented Jan 27, 2021

ibzib left a comment

Choose a reason for hiding this comment

ibzib Jan 27, 2021

Choose a reason for hiding this comment

iemejia commented Jan 27, 2021

ghost commented Jan 28, 2021

iemejia commented Jan 28, 2021

ghost commented Jan 30, 2021

ibzib left a comment

Choose a reason for hiding this comment

ibzib left a comment

Choose a reason for hiding this comment

ghost commented Feb 3, 2021

ghost commented Feb 3, 2021

ghost commented Feb 3, 2021

ibzib left a comment

Choose a reason for hiding this comment

ibzib commented Feb 3, 2021

ghost commented Feb 3, 2021

ibzib commented Feb 3, 2021

ibzib commented Feb 3, 2021

ghost commented Jan 13, 2021 •

edited by ghost

ghost Jan 27, 2021 •

edited by ghost