[SPARK-38234] [SQL] [SS] Added structured streaming monitoring APIs.

[yeskarthik]

What changes were proposed in this pull request?

Add two monitoring REST API endpoints under SQL that provides details about the structured streaming queries. Even though a store exists for it and the data is presented in the UI under the "Structured Streaming" tab, this data is not exposed as a REST API.

Summary API

Returns the summary of all existing streaming queries.

GET /{appId}/sql/streamingqueries

Response is list of StreamingQueryData.

Progress API

Returns the progress events of a specific streaming query by runId. User can also specify how many of the most recent events needs to be retrieved by using the last query parameter. By default, we return the most recent progress event i.e. last is set to 1.

GET /{appId}/sql/streamingqueries/{runId}/progress?last={N}

Response is list of StreamingQueryProgress.

Note: We are not introducing new object definitions for the response since we are just returning the data from the store without aggregation, these are existing event structures - StreamingQueryData and StreamingQueryProgress.

Why are the changes needed?

This data can be used for monitoring, detecting streaming and to build custom dashboards. This monitoring API will be similar to the monitoring APIs that are present for DStreams - refer SPARK-18470.

Does this PR introduce any user-facing change?

No

How was this patch tested?

• Added 4 tests under SqlResourceWithActualMetricsSuite to test the endpoints when streaming code is executed. Tested various cases, parameters, boundary conditions
  

• Added 2 tests under SqlResourceInvalidEndpointSuite.scala to test the endpoints when no streaming code is running.
  

• As mentioned earlier, since we are just returning the objects from the store, there is no need to test them individually.

[yeskarthik]

@HeartSaVioR @uncleGen @dongjoon-hyun @xuanyuanking @zsxwing can you please take a look at this PR? Thanks

[AmplabJenkins]

Can one of the admins verify this patch?

[dongjoon-hyun]

Thank you for making a PR, @yeskarthik .

However, Apache Spark has a versioning policy which doesn't allow a new feature or improvement on the released versions.

• https://spark.apache.org/versioning-policy.html

You can open a PR to master branch for Apache Spark 3.3. Thanks!

[yeskarthik]

@dongjoon-hyun thank you! I will make a PR against master.

[dongjoon-hyun]

Thank you, @yeskarthik !