[SPARK-49241][CORE] Add `OpenTelemetryPush` Sink with `opentelemetry` profile #47763

TQJADE · 2024-08-14T22:21:09Z

What changes were proposed in this pull request?

OpenTelemetryPushSink and OpenTelemetryPushReporter has been added in the supported Sink Classes.

Why are the changes needed?

From OpenTelemetry side, they have added the support from a pull model: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/apachesparkreceiver
However, for a very short jobs, only the push model is applicable. Therefore, a new sink OpenTelemetryPushSink and a new reporter OpenTelemetryPushReporter have been introduced here.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Build the Apache Spark Core Snapshot. Reference the Snapshot and giving the local OpenTelemetry endpoint 127.0.0.1:4317 as the endpoint for OpenTelemetryReporter.
Start the open-telemetry collector locally, per: https://opentelemetry.io/docs/collector/quick-start/#generate-and-collect-telemetry

docker run \
  -p 127.0.0.1:4317:4317 \
  -p 127.0.0.1:55679:55679 \
  otel/opentelemetry-collector:0.107.0 \
  2>&1 | tee collector-output.txt # Optionally tee output for easier search later

Start Spark Applications and publishing the metrics
Observing the metrics to be printed.

Was this patch authored or co-authored using generative AI tooling?

No

dongjoon-hyun · 2024-08-14T22:30:07Z

Thank you for making a PR, @TQJADE .

dongjoon-hyun

Could you make CIs happy?

dongjoon-hyun · 2024-08-16T16:19:59Z

It seems that there are multiple failures in CIs. Do you think those failures related to this PR?
When we enable this profile, it seems that this PR introduces many dependencies. Do we need all?

$ dev/make-distribution.sh -Popentelemetry-reporter
$ ls -alh dist/jars/*opentelemetry*
-rw-r--r--@ 1 dongjoon  staff   138K Aug 16 09:20 dist/jars/opentelemetry-api-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff    42K Aug 16 09:20 dist/jars/opentelemetry-api-incubator-1.40.0-alpha.jar
-rw-r--r--@ 1 dongjoon  staff    46K Aug 16 09:20 dist/jars/opentelemetry-context-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff   100K Aug 16 09:20 dist/jars/opentelemetry-exporter-common-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff    68K Aug 16 09:20 dist/jars/opentelemetry-exporter-otlp-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff   188K Aug 16 09:20 dist/jars/opentelemetry-exporter-otlp-common-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff    25K Aug 16 09:20 dist/jars/opentelemetry-exporter-sender-okhttp-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff   6.6K Aug 16 09:20 dist/jars/opentelemetry-sdk-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff    53K Aug 16 09:20 dist/jars/opentelemetry-sdk-common-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff    19K Aug 16 09:20 dist/jars/opentelemetry-sdk-extension-autoconfigure-spi-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff    53K Aug 16 09:20 dist/jars/opentelemetry-sdk-logs-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff   302K Aug 16 09:20 dist/jars/opentelemetry-sdk-metrics-1.40.0.jar
-rw-r--r--@ 1 dongjoon  staff   125K Aug 16 09:20 dist/jars/opentelemetry-sdk-trace-1.40.0.jar

Does this introduce any transitive dependencies too?

dongjoon-hyun · 2024-08-16T16:32:41Z

core/pom.xml

Let's use the latest one.

https://mvnrepository.com/artifact/io.opentelemetry/opentelemetry-api/1.41.0

dongjoon-hyun · 2024-08-25T22:54:41Z

core/src/main/scala/org/apache/spark/metrics/sink/opentelemetry/OpenTelemetryPushReporter.scala

+    pollInterval: Int = 10,
+    pollUnit: TimeUnit = TimeUnit.SECONDS,
+    host: String = "http://localhost",
+    port: String = "4317",


Could you double-check this? It seems that OpenTelemetry changed this. What is the current status?

Change OTLP/HTTP port from 4317 to 4318 (#1839) open-telemetry/opentelemetry-specification#1970

https://github.com/open-telemetry/opentelemetry-specification/blob/0b3328bb17fa6f86af521f8d22dbbcddea2ac914/CHANGELOG.md?plain=1#L2164

@dongjoon-hyun 4317 is port for grpc and 4318 is for HTTP/HTTPS. Should I rename the port to grpcPort?

https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/

To @TQJADE , you are using http:// in line 39, aren't you? Are you using grpc instead of HTTP here?

https://github.com/open-telemetry/opentelemetry-java/blob/7522bfe141524d52649b5c344c643104b57af608/exporters/common/src/main/java/io/opentelemetry/exporter/internal/ExporterBuilderUtil.java#L32-L47
Here there is a endpoint validation, it required the endpoint to start with http/https

What does that mean? Do you mean OpenTelemetry requires HTTP with 4317 ? If we need to use 4317 with HTTP, which HTTP call use 4318 port?

I tested with grpc endpoint, when I tested, the endpoint should be started with http or https. The default value of Opentelmetry GRPC endpoint port is 4317. HTTP endpoint port is 4318.

It seems that we are repeating the same questions and answers again and again. :)

What I asked to confirm is that why we are not using the default value of HTTP endpoint port (4318) here when the endpoint starts with http. This PR uses http://localhost:4317 instead of http://localhost:4318, doesn't it? We know that your sentence is correct, but it sounds like contradiction. We need an explanation for this contradiction.

the endpoint should be started with http or https. The default value of Opentelmetry GRPC endpoint port is 4317. HTTP endpoint port is 4318.

dongjoon-hyun · 2024-08-25T22:58:26Z

.../src/test/scala/org/apache/spark/metrics/sink/opentelemetry/OpenTelemetryPushSinkSuite.scala

+
+import org.apache.spark.SparkFunSuite
+
+class OpenTelemetryPushSinkSuite extends SparkFunSuite with PrivateMethodTester {


This looks like having a test coverage for only object OpenTelemetryPushSink.

Could you add more test coverage for class OpenTelemetryPushSink?

dongjoon-hyun · 2024-08-25T23:00:04Z

.../test/scala/org/apache/spark/metrics/sink/opentelemetry/OpenTelemetryPushReporterSuite.scala

+
+import org.apache.spark.SparkFunSuite
+
+class OpenTelemetryPushReporterSuite extends SparkFunSuite with PrivateMethodTester {


Could you add more test coverage?

dongjoon-hyun · 2024-08-27T05:55:26Z

core/pom.xml

+        <dependency>
+          <groupId>com.squareup.okhttp3</groupId>
+          <artifactId>okhttp</artifactId>
+          <version>3.13.0</version>


Please note that Apache Spark's okhttp is 3.12.12. So, we had better use 3.12.12 even for the test case in order to make it sure that there is no problem inside Apache Spark 4.0.0 release.

spark/dev/deps/spark-deps-hadoop-3-hive-2.3

Line 226 in 9fc1e05

okhttp/3.12.12//okhttp-3.12.12.jar

dongjoon-hyun

Thank you for updating this PR. We still need your replies for the following review comments.

dongjoon-hyun · 2024-08-27T16:57:38Z

core/pom.xml

Please remove this comment.

dongjoon-hyun

+1, LGTM. Thank you, @TQJADE .
Merged to master.

LuciferYang · 2024-09-02T11:50:21Z

Is it necessary to activate the opentelemetry profile to execute unit tests in IntelliJ, regardless of whether it's Maven or SBT? I've found that without activating the opentelemetry profile, I'm unable to execute unit tests in IntelliJ. For example, when I try to execute DataSourceV2SQLSuiteV2Filter in IntelliJ, I encounter the following error:

. Although manually activating the `opentelemetry` profile can serve as a workaround, this is not a good experience because `opentelemetry` is not activated by default. Or are there any other correct ways to ensure it works?

cc @TQJADE @dongjoon-hyun

also cc @panbingkun

dongjoon-hyun · 2024-09-03T16:22:49Z

Got it.

Actually, this PR follows the existing Volcano profile way which has the same side-effect. However, this seems more disruptive because this is in core while Volcano affects only K8s module.

To @TQJADE , could you find a better way to mitigate the above situation?

mridulm · 2024-09-04T03:02:21Z

QQ: Cant this not live outside of Apache Spark - and use the existing integrations in MetricSystem ?
I am trying to understand why we need to include it as part of Spark, and not as an external package - are the current extension mechanisms not sufficient ?

+CC @dongjoon-hyun

LuciferYang · 2024-09-04T08:23:06Z

Got it.

Actually, this PR follows the existing Volcano profile way which has the same side-effect. However, this seems more disruptive because this is in core while Volcano affects only K8s module.

To @TQJADE , could you find a better way to mitigate the above situation?

cc @dongjoon-hyun @TQJADE In fact, due to our use of sbt-pom-reader, on the Maven side, we can utilize the build-helper-maven-plugin to dynamically load additional src and test code paths. The configuration for build-helper-maven-plugin is also dynamically activated in SparkBuild.scala without requiring any additional configuration. I think this method should resolve the issue I mentioned.

I have submitted a pr: #47986.

Before supplementing the PR description, I will take some time to conduct additional verification.

I am trying to understand why we need to include it as part of Spark, and not as an external package - are the current extension mechanisms not sufficient ?

Of course, this question also requires a clear answer :)

TQJADE · 2024-09-04T18:17:41Z

I am trying to understand why we need to include it as part of Spark, and not as an external package - are the current extension mechanisms not sufficient ?

@mridulm @LuciferYang
I would like to use OpenTelemetry instrumentations to “translate” the codahale metrics of Spark Jobs to opentelemetry metrics. However this is disabled by default as below with reasons:
https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/supported-libraries.md#disabled-instrumentations

Some instrumentations can produce too many spans and make traces very noisy. For this reason, the following instrumentations are disabled by default. dropwizard-metrics which might create a very low quality metrics data, because of lack of label/attribute support in the Dropwizard metrics API.

That is the reason I would like to contribute to Apache Spark side. The OpenTelemetryPushSink is similar to StatsdSink, to push the metrics to target services.
cc: @dongjoon-hyun

mridulm · 2024-09-05T00:09:41Z

I am trying to understand whether there are any limitations due to which this integration has to be included as part of Apache Spark.
In other words, can this be an external integration/library that users include for the spark apps which need it: I am not familiar with opentelemetry - so trying to reason about the value of including it within Spark.

dongjoon-hyun · 2024-09-05T04:58:57Z

Your point is correct, @mridulm .

There is no limitation in Spark architecture and this can be an external library without any problem.
Initially, I misunderstood the context of the existing opentelemetry's Disabled instrumentations as discontinued or deprecated ones. It's my bad.

Let me revert this commit. Thank you for all your valuable reviews, @mridulm , @LuciferYang . Personally, sorry for wasting your time, @TQJADE .

dongjoon-hyun · 2024-09-05T05:04:35Z

This is reverted via 628cd75

mridulm · 2024-09-05T16:26:12Z

Thanks for the details @dongjoon-hyun ! Really appreciate it.

asafm · 2024-11-19T20:46:14Z

Why did you guys revert it? Did you put it in a Spark package? From my understanding of how Spark packages work, you can't place a MertricSink implementation in a package, since the sink it initialized before spark packages are loaded.

github-actions bot added BUILD CORE labels Aug 14, 2024

dongjoon-hyun changed the title ~~[SPARK-49241](Add OpenTelemetryPush Sink)~~ [SPARK-49241][CORE] Add OpenTelemetryPush Sink Aug 14, 2024

dongjoon-hyun reviewed Aug 14, 2024

View reviewed changes

TQJADE force-pushed the sink branch from b3b1e59 to ef4b19e Compare August 15, 2024 00:55

github-actions bot added INFRA DOCS labels Aug 15, 2024

TQJADE force-pushed the sink branch 7 times, most recently from 459fec6 to 5ad317b Compare August 16, 2024 01:03

dongjoon-hyun reviewed Aug 16, 2024

View reviewed changes

TQJADE added 12 commits August 21, 2024 14:22

Add OpenTelemetry Push Reporter and Sink

bd2bc94

Enable OpenTelemetry Profile when testing

1b23a98

Fix the test suite scala style error

6a15a95

Fix the test failure of mima

d305cb6

Fix the test failure of TPC-DS

0b21708

Fix the test failure of docs-generation

8932877

Try to fix Mima failures

4b3d7e8

Add build_profile_flags for opentelemetry-reporter in core

40a7db7

Bump opentelemtry version

ef9217a

Remove previous overriding profile

96a1b01

Add add-source for opentelemetry

f282c08

WIP

cee8f55

TQJADE force-pushed the sink branch from 5ad317b to 12c2990 Compare August 22, 2024 17:23

TQJADE requested a review from dongjoon-hyun August 24, 2024 05:49

dongjoon-hyun changed the title ~~[SPARK-49241][CORE] Add OpenTelemetryPush Sink~~ [SPARK-49241][CORE] Add OpenTelemetryPush Sink with opentelemetry profile Aug 25, 2024

dongjoon-hyun requested changes Aug 25, 2024

View reviewed changes

dongjoon-hyun reviewed Aug 25, 2024

View reviewed changes

Enrich tests

00bb1dc

dongjoon-hyun reviewed Aug 27, 2024

View reviewed changes

dongjoon-hyun requested changes Aug 27, 2024

View reviewed changes

dongjoon-hyun reviewed Aug 27, 2024

View reviewed changes

Modify okhttp version

dd3b49c

TQJADE force-pushed the sink branch from 79da886 to dd3b49c Compare August 27, 2024 17:02

Use endpoint

f38711a

TQJADE requested a review from dongjoon-hyun August 28, 2024 01:18

dongjoon-hyun approved these changes Aug 28, 2024

View reviewed changes

dongjoon-hyun closed this in 2eb2e39 Aug 28, 2024

LuciferYang mentioned this pull request Sep 4, 2024

[CORE][BUILD] Change to using build-helper-maven-plugin to manage the code for opentelemetry #47986

Closed


		import org.apache.spark.SparkFunSuite

		class OpenTelemetryPushSinkSuite extends SparkFunSuite with PrivateMethodTester {


		import org.apache.spark.SparkFunSuite

		class OpenTelemetryPushReporterSuite extends SparkFunSuite with PrivateMethodTester {

[SPARK-49241][CORE] Add OpenTelemetryPush Sink with opentelemetry profile #47763

[SPARK-49241][CORE] Add OpenTelemetryPush Sink with opentelemetry profile #47763

Uh oh!

Conversation

TQJADE commented Aug 14, 2024 • edited by dongjoon-hyun Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

dongjoon-hyun commented Aug 14, 2024

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TQJADE Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Aug 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

LuciferYang commented Sep 2, 2024

Uh oh!

dongjoon-hyun commented Sep 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mridulm commented Sep 4, 2024

Uh oh!

LuciferYang commented Sep 4, 2024

Uh oh!

TQJADE commented Sep 4, 2024

Uh oh!

mridulm commented Sep 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Sep 5, 2024

Uh oh!

dongjoon-hyun commented Sep 5, 2024

Uh oh!

mridulm commented Sep 5, 2024

Uh oh!

[SPARK-49241][CORE] Add `OpenTelemetryPush` Sink with `opentelemetry` profile #47763

[SPARK-49241][CORE] Add `OpenTelemetryPush` Sink with `opentelemetry` profile #47763

TQJADE commented Aug 14, 2024 •

edited by dongjoon-hyun

Loading

dongjoon-hyun commented Aug 16, 2024 •

edited

Loading

TQJADE Aug 27, 2024 •

edited

Loading

dongjoon-hyun Aug 27, 2024 •

edited

Loading

dongjoon-hyun Aug 27, 2024 •

edited

Loading

dongjoon-hyun commented Sep 3, 2024 •

edited

Loading

mridulm commented Sep 5, 2024 •

edited

Loading