-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-47596][DSTREAMS] Streaming: Migrate logWarn with variables to structured logging framework #46079
Conversation
…structured logging framework
logWarning(s"isTimeValid called with $time whereas the last valid time " + | ||
s"is $lastValidTime") | ||
logWarning(log"isTimeValid called with ${MDC(TIME, time)} whereas the last valid time " + | ||
log"is ${MDC(LAST_VALID_TIME, lastValidTime)}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Their types are Time
, so the output time units are milliseconds
@@ -247,7 +247,8 @@ private[streaming] class ReceivedBlockTracker( | |||
true | |||
} catch { | |||
case NonFatal(e) => | |||
logWarning(s"Exception thrown while writing record: $record to the WriteAheadLog.", e) | |||
logWarning(log"Exception thrown while writing record: " + | |||
log"${MDC(RECEIVED_BLOCK_TRACKER_LOG_EVENT, record)} to the WriteAheadLog.", e) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure if it's called RECORD
, maybe it's more general
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using the class name RECEIVED_BLOCK_TRACKER_LOG_EVENT
is also fine
@@ -884,14 +885,16 @@ abstract class DStream[T: ClassTag] ( | |||
val alignedToTime = if ((toTime - zeroTime).isMultipleOf(slideDuration)) { | |||
toTime | |||
} else { | |||
logWarning(s"toTime ($toTime) is not a multiple of slideDuration ($slideDuration)") | |||
logWarning(log"toTime (${MDC(TO_TIME, toTime)}) is not a multiple of slideDuration " + | |||
log"(${MDC(SLIDE_DURATION, slideDuration)})") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The output time units are milliseconds
.
@panbingkun Thanks again for the works! Merging to master. |
Structured Streaming supports writing Spark SQL and using SQL to write stream processing logic. Is this possible, similar to the syntax of Flink. SQL can satisfy the flow processing process. |
What changes were proposed in this pull request?
The pr aims to migrate
logWarning
in moduleStreaming
with variables tostructured logging framework
.Why are the changes needed?
To enhance Apache Spark's logging system by implementing structured logging.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Was this patch authored or co-authored using generative AI tooling?
No.