spark-sql-streaming-StreamingQueryListener.adoc

StreamingQueryListener

StreamingQueryListener is the interface for notifications about the lifecycle events of streaming queries.

package org.apache.spark.sql.streaming

abstract class StreamingQueryListener {
  def onQueryStarted(event: QueryStartedEvent): Unit
  def onQueryProgress(event: QueryProgressEvent): Unit
  def onQueryTerminated(event: QueryTerminatedEvent): Unit
}

You can register a StreamingQueryListener using StreamingQueryManager.addListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.addListener(queryListener)

You can remove a StreamingQueryListener using StreamingQueryManager.removeListener method.

val queryListener: StreamingQueryListener = ...
spark.streams.removeListener(queryListener)

onQueryStarted Callback

onQueryStarted(event: QueryStartedEvent): Unit

onQueryStarted handles QueryStartedEvents.

Figure 1. StreamingQueryListener Notified about Query’s Start (onQueryStarted)

Note	Used internally to unblock the starting thread (of StreamExecution).

Note	onQueryStarted is triggered right after StreamExecution has started running batches.

onQueryProgress Callback

onQueryProgress(event: QueryProgressEvent): Unit

onQueryProgress handles QueryProgressEvents.

Figure 2. StreamingQueryListener Notified about Query’s Progress (onQueryProgress)

Note	onQueryProgress is triggered when ProgressReporter updates query progress (which is when StreamExecution runs batches and a trigger has finished).

onQueryTerminated Callback

onQueryTerminated(event: QueryTerminatedEvent): Unit

onQueryTerminated handles QueryTerminatedEvents.

Figure 3. StreamingQueryListener Notified about Query’s Termination (onQueryTerminated)

Note	onQueryTerminated is triggered right before StreamExecution finishes running batches (regardless of whether an exception was reported or not).