feat: expose Comet version as spark.comet.version runtime config - #5049
Open
andygrove wants to merge 1 commit into
Open
feat: expose Comet version as spark.comet.version runtime config#5049andygrove wants to merge 1 commit into
andygrove wants to merge 1 commit into
Conversation
Expose the loaded Comet build version through the Spark config spark.comet.version (set by the Comet driver plugin) so it can be queried at runtime via spark.conf.get or SET. Also document the existing org.apache.comet.COMET_VERSION programmatic accessor.
comphead
reviewed
Jul 27, 2026
| `spark.comet.version`. This can be queried at runtime from any supported language, for example: | ||
|
|
||
| ```scala | ||
| scala> spark.conf.get("spark.comet.version") |
Contributor
There was a problem hiding this comment.
would be nice to get an example output?
comphead
reviewed
Jul 27, 2026
| ``` | ||
|
|
||
| ```sql | ||
| SET spark.comet.version; |
Contributor
There was a problem hiding this comment.
I hardly can imagine scenarios for user to change comet version
Member
Author
There was a problem hiding this comment.
SET shows the current value.
spark-sql (default)> SET spark.sql.adaptive.enabled;
spark.sql.adaptive.enabled true
Time taken: 0.276 seconds, Fetched 1 row(s)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Supersedes #4852.
Rationale for this change
There is currently no convenient way for a Spark user to query which Comet version is loaded. The JVM
org.apache.comet.COMET_VERSIONconstant exists but is only reachable via Scala imports, and the native library logs the version on init but that requires driver log access.The previous attempt (#4852) exposed the version as a SQL function via
SparkSessionExtensions.injectFunction. Review feedback on that PR questioned whether a function was the right mechanism, and registering a function into the session registry also broke the upstream SparkShowFunctionsSuitetests, since those suites assert on the exact set of registered functions. Exposing the version as a Spark config avoids polluting the function registry entirely and matches how Spark surfaces similar build metadata.What changes are included in this PR?
spark.comet.versionon the SparkConf duringinit, so it is queryable at runtime viaspark.conf.get("spark.comet.version")orSET spark.comet.versionin SQL. It is set before the off-heap check so the version is reported even when Comet is otherwise disabled.spark.comet.versionruntime config and the existingorg.apache.comet.COMET_VERSION(plusCOMET_BRANCH/COMET_REVISION) programmatic accessors in the installation guide.How are these changes tested?
CometPluginsSuite(Comet version is exposed as a Spark config) assertingspark.comet.versionis visible on both the SparkContext conf and the session runtime config, using the same plugin-loading harness that covers the existing memory-overhead behavior.