Add spark.openlineage.appName tag to spark.application spans#11246
Add spark.openlineage.appName tag to spark.application spans#11246gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits intomasterfrom
Conversation
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
|
/merge |
|
View all feedbacks in Devflow UI.
This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
The expected merge time in
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c4bac7b335
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } | ||
|
|
||
| private void captureOpenlineageJobInfo(AgentTracer.SpanBuilder builder) { | ||
| String olAppName = sparkConf.get("spark.openlineage.appName", null); |
There was a problem hiding this comment.
captureOpenlineageJobInfo reads spark.openlineage.appName only from sparkConf, but this codebase already documents and handles cases where Spark config is not captured (see LiveListenerBusAdvice, which reflects getConf() and stores openLineageSparkConf for that reason). In those environments (notably the Databricks path called out in AbstractSparkInstrumentation), OpenLineage can still resolve an app name while this span tag remains missing, so the intended OL/DD correlation is still broken. Prefer reading from openLineageSparkConf when available, with fallback to sparkConf.
Useful? React with 👍 / 👎.
aboitreaud
added
inst: apache spark
gh-worker-dd-mergequeue-cf854d
Bot
merged commit bfbfa5b
into
master
2 participants
What Does This Do
Capture on the spans the OL job name. can be overriden in OL so we need to capture it and have it on the span
Motivation
OpenLineage names the application job using spark.openlineage.appName if set, falling back to spark.app.name otherwise.
The gap: when a user sets spark.openlineage.appName to something different from spark.app.name, OL uses that override as its job name but Datadog never reads that key — so job_name on the span = spark.app.name while OL's job name = spark.openlineage.appName. Correlation with our job_name attr breaks.
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.