Add Spark 4.0.0 test suite on JDK 17 to spark_2.13 - #12049
Add Spark 4.0.0 test suite on JDK 17 to spark_2.13#12049gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
Conversation
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
![datadog-official[bot]](https://avatars.githubusercontent.com/in/1358509?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The Spark 4 test suite is isolated to JDK 17 through 23, while the shared fixture changes preserve Spark 3 compatibility and account for Spark 4 constructor, helpful-NPE, and ANSI-mode differences. No diff-only behavioral regression was identified; no additional tests recommended: the new suite already exercises the relevant Spark 4 paths.
🤖 Datadog Autotest · Commit 70d750d · What is Autotest? · Any feedback? Reach out in #autotest
aboitreaud
added
inst: apache spark
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 70d750d676
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| "--add-opens=java.base/java.nio=ALL-UNNAMED", | ||
| "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" |
There was a problem hiding this comment.
When test_spark4 actually runs on JDK 17+, the in-process SparkSession is not launched through Spark's scripts, so it only receives the options listed here plus the repo's generic java.lang/java.util opens. Spark 4.0.0's own JavaModuleOptions marks a larger default set as required for Java 17 (for example java.lang.reflect, java.io, java.net, java.util.concurrent, jdk.internal.ref, and sun.nio.ch), and Spark startup/SQL execution can fail with module-access errors without those options; please feed JavaModuleOptions.defaultModuleOptionArray() or mirror the full Spark list for this suite.
Useful? React with 👍 / 👎.
This comment has been minimized.
This comment has been minimized.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
|
Confirming that the new test_spark4 suite actually ran its tests on this branch, on both supported JDKs: JDK 17 — job 1885974670 (https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1885974670) (test_inst: [17, 7/8])
JDK 21 — job 1885974678 (https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1885974678) (test_inst: [21, ...]),
As expected, test_spark4 is SKIPPED on JDK 25/tip due to the maxJavaVersion = VERSION_23 cap (Hadoop's UserGroupInformation breaks under JEP 486 / SecurityManager removal on JDK 24+), so no test events exist for those shards — consistent with the module's testJvmConstraints configuration. |
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
|
2 participants
What Does This Do
Adds a dedicated
test_spark4Gradle test suite tospark_2.13that runs the existing instrumentation test fixtures against real Spark 4.0.0 jars on JDK 17+ (Spark 4.0 requires JDK 17 minimum), complementing the existingtest_spark32suite.test_spark4suite (spark-core/spark-sql/spark-yarn4.0.0) with its own JDK constraints (17–23) and--add-opensflags mirroring Spark's ownJavaModuleOptionsdefaults, since the tests start aSparkSessionin-process rather than through Spark's launch scripts.protobuf-java:3.14.0on the new suite's classpath to avoid a binary-incompatible transitive protobuf version pulled in by Spark 4.0.0.AbstractSparkListenerTest,AbstractSparkTest) to handle Spark 4 API/behavior differences:SparkListenerApplicationEnd's new constructor param, ANSI mode being on by default, and JDK 15+'s more detailedNullPointerExceptionmessages surfacing under the new JDK 17 suite.No production/instrumentation code changes — this is purely test coverage catching up with Spark 4 support that already landed in earlier PRs.
Motivation
Better test coverage for recent Spark / Java version in the apache spark inst
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]