[SPARK-41895][SS][UI] Add tests for streaming UI with RocksDB backend

[gengliangwang]

What changes were proposed in this pull request?

Add tests for streaming UI with RocksDB backend

Why are the changes needed?

Better test coverage when live UI with RocksDB backend is enabled.

Does this PR introduce any user-facing change?

No

How was this patch tested?

GA tests

[gengliangwang]

cc @LuciferYang

[gengliangwang]

I am rerunning tests since #39416 is merged.

[LuciferYang]

13:45:13.014 ERROR org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus: Listener StreamingQueryStatusListener threw an exception
java.lang.NullPointerException
	at org.apache.spark.status.protobuf.StoreTypes$StreamingQueryData$Builder.setName(StoreTypes.java:57107)
	at org.apache.spark.status.protobuf.sql.StreamingQueryDataSerializer.serialize(StreamingQueryDataSerializer.scala:33)
	at org.apache.spark.status.protobuf.KVStoreProtobufSerializer.serialize(KVStoreProtobufSerializer.scala:29)
	at org.apache.spark.util.kvstore.RocksDB.write(RocksDB.java:188)
	at org.apache.spark.status.ElementTrackingStore.write(ElementTrackingStore.scala:123)
	at org.apache.spark.status.ElementTrackingStore.write(ElementTrackingStore.scala:127)
	at org.apache.spark.sql.streaming.ui.StreamingQueryStatusListener.onQueryStarted(StreamingQueryStatusListener.scala:82)
	at org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus.doPostEvent(StreamingQueryListenerBus.scala:131)
	at org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus.doPostEvent(StreamingQueryListenerBus.scala:43)
	at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
	at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
	at org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus.postToAll(StreamingQueryListenerBus.scala:88)
	at org.apache.spark.sql.execution.streaming.StreamingQueryListenerBus.post(StreamingQueryListenerBus.scala:77)
	at org.apache.spark.sql.streaming.StreamingQueryManager.postListenerEvent(StreamingQueryManager.scala:231)
	at org.apache.spark.sql.execution.streaming.StreamExecution.postEvent(StreamExecution.scala:402)
	at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:278)
	at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:207)

spark/sql/core/src/main/scala/org/apache/spark/status/protobuf/sql/StreamingQueryDataSerializer.scala

Line 33 in 99739ae

.setName(data.name)

data.name still needs to be protected to avoid NPE

[LuciferYang]

+1, LGTM

[gengliangwang]

Merging to master

[dongjoon-hyun]

+1, LGTM.

[dongjoon-hyun]

Hi, @gengliangwang .
This might cause test flakiness. I saw several failures in these days. I filed a JIRA, SPARK-41969 for flaky tests. Could you take a look at this, please?

Also, cc @xinrong-meng as Apache Spark 3.4.0 release manager.

[gengliangwang]

@dongjoon-hyun yes I will try fixing it today