-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-29543][SS][FOLLOWUP] Move spark.sql.streaming.ui.*
configs to StaticSQLConf
#27425
Conversation
spark.sql.streaming.ui.*
configs to StaticSQLConf
Test build #117709 has finished for PR 27425 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for fixing these. Left some nits. Otherwise, LGTM.
|
||
val STREAMING_UI_ENABLED = | ||
buildStaticConf("spark.sql.streaming.ui.enabled") | ||
.doc("Whether to run the structured streaming UI for the Spark application.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: Whether to run the Structured Streaming Web UI for the Spark application when the Spark Web UI is enabled.
|
||
val STREAMING_UI_RETAINED_PROGRESS_UPDATES = | ||
buildStaticConf("spark.sql.streaming.ui.retainedProgressUpdates") | ||
.doc("The number of progress updates to retain for a streaming query for structured " + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: structured streaming ui
-> Structured Streaming UI
|
||
val STREAMING_UI_RETAINED_QUERIES = | ||
buildStaticConf("spark.sql.streaming.ui.retainedQueries") | ||
.doc("The number of inactive queries to retain for structured streaming ui.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, doc changes done in d763abb.
@@ -146,12 +146,14 @@ private[sql] class SharedState( | |||
*/ | |||
lazy val streamingQueryStatusListener: Option[StreamingQueryStatusListener] = { | |||
val sqlConf = SQLConf.get |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: we can remove this and use conf
in this class directly.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, then StreamingQueryStatusListener need a little change, done in d763abb.
Test build #117740 has finished for PR 27425 at commit
|
Thanks for this work, LGTM. |
Thanks! Merging to master and branch-3.0. |
…o StaticSQLConf ### What changes were proposed in this pull request? Put the configs below needed by Structured Streaming UI into StaticSQLConf: - spark.sql.streaming.ui.enabled - spark.sql.streaming.ui.retainedProgressUpdates - spark.sql.streaming.ui.retainedQueries ### Why are the changes needed? Make all SS UI configs consistent with other similar configs in usage and naming. ### Does this PR introduce any user-facing change? Yes, add new static config `spark.sql.streaming.ui.retainedProgressUpdates`. ### How was this patch tested? Existing UT. Closes #27425 from xuanyuanking/SPARK-29543-follow. Authored-by: Yuanjian Li <xyliyuanjian@gmail.com> Signed-off-by: Shixiong Zhu <zsxwing@gmail.com> (cherry picked from commit a4912ce) Signed-off-by: Shixiong Zhu <zsxwing@gmail.com>
Thanks for the review! |
What changes were proposed in this pull request?
Put the configs below needed by Structured Streaming UI into StaticSQLConf:
Why are the changes needed?
Make all SS UI configs consistent with other similar configs in usage and naming.
Does this PR introduce any user-facing change?
Yes, add new static config
spark.sql.streaming.ui.retainedProgressUpdates
.How was this patch tested?
Existing UT.