[SPARK-49751][CONNECT] Fix deserialization of SparkListenerConnectServiceStarted event #48200
Conversation
…viceStarted event
| hostAddress: String, | ||
| bindingPort: Int, | ||
| sparkConf: SparkConf, | ||
| sparkConf: Map[String, String], |
There was a problem hiding this comment.
Can anyone explain why we are sending all spark confs? That seems to be a bit heavyweight.
There was a problem hiding this comment.
regarding use case mentioned in SPARK-47952, I think sparkConf is not necessary, maybe the author can answer this question @TakawaAkirayo
There was a problem hiding this comment.
The purpose is to send additional context, such as the spark.app.id set in SparkContext's _conf during startup, to the listener. This can be achieved through other means within the listener and this seems over-designed now, let's remove it @pan3793
There was a problem hiding this comment.
Okay, updated and verified again, please take another look, @hvanhovell and @TakawaAkirayo
|
kindly ping @hvanhovell |
|
Merged to master. |
4 participants
What changes were proposed in this pull request?
SparkListenerConnectServiceStartedis introduced in SPARK-47952, while the referenced fieldSparkConfis not serialized properly, then causes the SHS deserialization failure.According to the discussion, we can remove the
sparkConffield.Why are the changes needed?
Fix the event log deserialization and recover the SHS UI rendering.
Does this PR introduce any user-facing change?
Fix an unreleased feature, recover the SHS UI rendering from event logs produced by the connect server.
How was this patch tested?
Start a connect server with event log enabled, and then open the UI in SHS.
4.0.0-preview2
This PR.
Was this patch authored or co-authored using generative AI tooling?
No.