[SPARK-27142][SQL] Provide REST API for SQL information #24076

ajithme · 2019-03-13T03:57:57Z

What changes were proposed in this pull request?

Currently for Monitoring Spark application SQL information is not available from REST but only via UI. REST provides only applications,jobs,stages,environment. This Jira is targeted to provide a REST API so that SQL level information can be found
A single SQL query can result into multiple jobs. So for end user who is using STS or spark-sql, the intended highest level of probe is the SQL which he has executed. This information can be seen from SQL tab. Attaching a sample.

But same information he cannot access using the REST API exposed by spark and he always have to rely on jobs API which may be difficult. So i intend to expose the information seen in SQL tab in UI via REST API

Mainly:

Id : Long - execution id of the sql
status : String - possible values COMPLETED/RUNNING/FAILED
description : String - executed SQL string
planDescription : String - Plan representation
metrics : Seq[Metrics] - Metrics contain metricName: String, metricValue: String
submissionTime : String - formatted Date time of SQL submission
duration : Long - total run time in milliseconds
runningJobIds : Seq[Int] - sequence of running job ids
failedJobIds : Seq[Int] - sequence of failed job ids
successJobIds : Seq[Int] - sequence of success job ids

To fetch sql executions: /sql?details=boolean&offset=integer&length=integer
To fetch single execution: /sql/{executionID}?details=boolean

parameter	type	remarks
details	boolean	Optional. Set true to get plan description and metrics information, defaults to false
offset	integer	Optional. offset to fetch the executions, defaults to 0
length	integer	Optional. total number of executions to be fetched, defaults to 20

Why are the changes needed?

To support users query SQL information via REST API

Does this PR introduce any user-facing change?

Yes. It provides a new monitoring URL for SQL

How was this patch tested?

Tested manually

ajithme · 2019-03-13T04:01:36Z

cc @srowen @cloud-fan @dongjoon-hyun @gengliangwang

sql/core/src/main/scala/org/apache/spark/status/api/v1/SqlListResource.scala

sql/core/src/main/scala/org/apache/spark/status/api/v1/ApiSqlRootResource.scala

sql/core/src/main/scala/org/apache/spark/status/api/v1/SqlListResource.scala

gengliangwang · 2019-03-13T09:17:07Z

@ajithme Thanks for the work.
To make it consistent with other API, I think we need to have at least two APIs

@Path("sql")
@Path("sql/{sqlExecutionId: \\d+}")

ajithme · 2019-03-13T13:06:00Z

@ajithme Thanks for the work.
To make it consistent with other API, I think we need to have at least two APIs
@Path("sql")
@Path("sql/{sqlExecutionId: \\d+}")

@gengliangwang So the full api looks like

http://localhost:4040/api/v1/applications/{applicationId}/sql
http://localhost:4040/api/v1/applications/{applicationId}/sql/{executionId}

is this the expectation.?

gengliangwang · 2019-03-13T13:57:29Z

http://localhost:4040/api/v1/applications/{applicationId}/sql
http://localhost:4040/api/v1/applications/{applicationId}/sql/{executionId}

Yes, it is consistent with

http://localhost:4040/api/v1/applications/{applicationId}/jobs/
http://localhost:4040/api/v1/applications/{applicationId}/jobs/{jobId}

ajithme · 2019-03-13T14:06:07Z

Ok i have updated the PR accordingly. Please review

sql/core/src/main/scala/org/apache/spark/status/api/v1/SqlListResource.scala

dongjoon-hyun · 2019-03-13T18:44:45Z

ok to test

SparkQA · 2019-03-13T18:50:14Z

Test build #103450 has started for PR 24076 at commit 8827301.

ajithme · 2019-03-14T15:49:04Z

gentle ping @gengliangwang @dongjoon-hyun @srowen

srowen · 2019-03-14T19:14:06Z

As I say I'm not sure this is something we should expose this way. See prior comments on JIRA

ajithme · 2019-03-17T14:07:51Z

@srowen i have updated the use case in the JIRA already for prior comments. Please have a look

ajithme · 2019-03-19T18:03:06Z

gentle ping @cloud-fan @srowen @dongjoon-hyun @gengliangwang Any further suggestions of this feature.?

gengliangwang · 2019-03-20T16:46:52Z

@ajithme I am +1 for this.
Some comments are not addressed correctly(e.g. #24076 (comment)), so I create a PR(ajithme#1) in your repo. Please review it.

ajithme · 2019-03-26T03:53:06Z

@ajithme I am +1 for this.
Some comments are not addressed correctly(e.g. #24076 (comment)), so I create a PR(ajithme#1) in your repo. Please review it.

Thanks @gengliangwang I got confused between core/src/main/scala/org/apache/spark/status/api/v1/api.scala and sql/core/src/main/scala/org/apache/spark/status/api/v1/api.scala as file names are same :) . i will merge your comments shortly here.

ajithme · 2019-03-26T05:51:21Z

Updated with latest comments fixed. Please review

SparkQA · 2019-03-26T07:05:02Z

Test build #103950 has finished for PR 24076 at commit b1a8deb.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2019-03-26T08:10:39Z

retest this please.

SparkQA · 2019-03-26T12:29:12Z

Test build #103965 has finished for PR 24076 at commit b1a8deb.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gengliangwang · 2019-03-26T18:07:03Z

@vanzin

ajithme · 2019-05-06T04:47:38Z

@vanzin @srowen @dongjoon-hyun gentle ping. Can we re-look into this.? its been open for quite sometime

vanzin

This seems to be missing the plan description and metrics. Any specific reason not to expose those?

sql/core/src/main/scala/org/apache/spark/status/api/v1/SqlResource.scala

sql/core/src/main/scala/org/apache/spark/status/api/v1/api.scala

sql/core/src/main/scala/org/apache/spark/status/api/v1/SqlResource.scala

ajithme · 2019-08-09T09:37:56Z

This seems to be missing the plan description and metrics. Any specific reason not to expose those?

@vanzin Wanted this API to be light weight for monitoring. Returning plan and metrics can significantly increase the payload. Please share your thoughts

vanzin · 2019-08-15T00:35:31Z

Returning plan and metrics can significantly increase the payload.

Yes, but it's also useful information and there would be no way to retrieve it from the API...

In our APIs we generally have a query parameter that tells you what level of detail you have in the response, so you can have a small summary or a full view of an object, for example. That sounds like something we could add here.

I even did something similar in the StagesResource.oneAttemptData method with the details parameter. You could add that parameter here with a default value of false, for example, so only those who really want the extra data would get it.

ajithme · 2019-08-15T15:13:51Z

Returning plan and metrics can significantly increase the payload.

Yes, but it's also useful information and there would be no way to retrieve it from the API...

In our APIs we generally have a query parameter that tells you what level of detail you have in the response, so you can have a small summary or a full view of an object, for example. That sounds like something we could add here.

I even did something similar in the StagesResource.oneAttemptData method with the details parameter. You could add that parameter here with a default value of false, for example, so only those who really want the extra data would get it.

Thanks @vanzin for inputs. I agree with your opinion and will update the PR

ajithme · 2020-01-08T04:34:01Z

After handling review comments

ajithme · 2020-01-08T04:34:21Z

@vanzin Thank you for review and i apologise for indent issues. Now i have run ./dev/scalafmt and updated the PR with other review comments. Please review

vanzin

Few minor things.

sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLAppStatusStore.scala

sql/core/src/main/scala/org/apache/spark/status/api/v1/api.scala

…review comment

ajithme · 2020-01-10T22:17:55Z

@vanzin please review, updated as per the comments

vanzin · 2020-01-10T22:37:11Z

Weird tests aren't running.

vanzin · 2020-01-10T22:37:19Z

ok to test

SparkQA · 2020-01-11T02:40:31Z

Test build #116509 has finished for PR 24076 at commit 4560df9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

ajithme · 2020-01-14T15:38:18Z

@vanzin gentle ping

vanzin · 2020-01-14T18:04:39Z

Merging to master.

gengliangwang · 2020-05-20T04:21:17Z

@ajithme I think we have to revert this one in branch-3.0. See my comments in #28588

### What changes were proposed in this pull request? Revert #28208 and #24076 in branch 3.0 ### Why are the changes needed? Unfortunately, the PR #28208 is merged after Spark 3.0 RC 2 cut. Although the improvement is great, we can't break the policy to add new improvement commits into branch 3.0 now. Also, if we are going to adopt the improvement in a future release, we should not release 3.0 with #24076, since the API result will be changed. After discuss with cloud-fan and gatorsmile offline, we think the best choice is to revert both commits and follow community release policy. ### Does this PR introduce _any_ user-facing change? Yes, let's hold the SQL rest API until next release. ### How was this patch tested? Jenkins unit tests. Closes #28588 from gengliangwang/revertSQLRestAPI. Authored-by: Gengliang Wang <gengliang.wang@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

ajithme force-pushed the restapi branch from 51b1565 to 345c3f1 Compare March 13, 2019 04:09

gengliangwang reviewed Mar 13, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/status/api/v1/SqlListResource.scala Outdated Show resolved Hide resolved

gengliangwang reviewed Mar 13, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/status/api/v1/ApiSqlRootResource.scala Outdated Show resolved Hide resolved

gengliangwang reviewed Mar 13, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/status/api/v1/SqlListResource.scala Outdated Show resolved Hide resolved

gengliangwang requested changes Mar 13, 2019

View reviewed changes

vanzin reviewed May 6, 2019

View reviewed changes

dongjoon-hyun added the SQL label Jun 14, 2019

ajithme force-pushed the restapi branch from b1a8deb to 7b17c25 Compare August 9, 2019 09:34

ajithme and others added 11 commits January 8, 2020 08:20

update

89905fc

refactor

9f085c3

refactored

566e551

update to introduce induvidual api for sql

8cff44f

update as per review comments

e1893ad

update

0cc87c0

minor refactor

f95b00b

update

ae5413c

Updated as per review comments

640e1a7

Add option to get the planDescription and Metrics details

31bba41

Updated as per review comments

1c2419a

ajithme force-pushed the restapi branch from 62b9716 to 1c2419a Compare January 8, 2020 04:26

ajithme requested a review from vanzin January 10, 2020 06:16

vanzin reviewed Jan 10, 2020

View reviewed changes

ajithme added 3 commits January 11, 2020 02:34

run scalafmt on modifed block only and move to sql package to handle …

7d0dd19

…review comment

Update

a8e2525

Import

4560df9

vanzin closed this in 0c6bd3b Jan 14, 2020

gengliangwang mentioned this pull request May 20, 2020

Revert [SPARK-27142][SPARK-31440] SQL rest API in branch 3.0 #28588

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-27142][SQL] Provide REST API for SQL information #24076

[SPARK-27142][SQL] Provide REST API for SQL information #24076

ajithme commented Mar 13, 2019 •

edited

Loading

ajithme commented Mar 13, 2019

gengliangwang commented Mar 13, 2019

ajithme commented Mar 13, 2019

gengliangwang commented Mar 13, 2019

ajithme commented Mar 13, 2019

dongjoon-hyun commented Mar 13, 2019

SparkQA commented Mar 13, 2019

ajithme commented Mar 14, 2019

srowen commented Mar 14, 2019

ajithme commented Mar 17, 2019

ajithme commented Mar 19, 2019

gengliangwang commented Mar 20, 2019 •

edited

Loading

ajithme commented Mar 26, 2019 •

edited

Loading

ajithme commented Mar 26, 2019

SparkQA commented Mar 26, 2019

gengliangwang commented Mar 26, 2019

SparkQA commented Mar 26, 2019

gengliangwang commented Mar 26, 2019

ajithme commented May 6, 2019 •

edited

Loading

vanzin left a comment

ajithme commented Aug 9, 2019

vanzin commented Aug 15, 2019

ajithme commented Aug 15, 2019

ajithme commented Jan 8, 2020

ajithme commented Jan 8, 2020

vanzin left a comment

ajithme commented Jan 10, 2020

vanzin commented Jan 10, 2020

vanzin commented Jan 10, 2020

SparkQA commented Jan 11, 2020

ajithme commented Jan 14, 2020

vanzin commented Jan 14, 2020

gengliangwang commented May 20, 2020

[SPARK-27142][SQL] Provide REST API for SQL information #24076

[SPARK-27142][SQL] Provide REST API for SQL information #24076

Conversation

ajithme commented Mar 13, 2019 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

ajithme commented Mar 13, 2019

gengliangwang commented Mar 13, 2019

ajithme commented Mar 13, 2019

gengliangwang commented Mar 13, 2019

ajithme commented Mar 13, 2019

dongjoon-hyun commented Mar 13, 2019

SparkQA commented Mar 13, 2019

ajithme commented Mar 14, 2019

srowen commented Mar 14, 2019

ajithme commented Mar 17, 2019

ajithme commented Mar 19, 2019

gengliangwang commented Mar 20, 2019 • edited Loading

ajithme commented Mar 26, 2019 • edited Loading

ajithme commented Mar 26, 2019

SparkQA commented Mar 26, 2019

gengliangwang commented Mar 26, 2019

SparkQA commented Mar 26, 2019

gengliangwang commented Mar 26, 2019

ajithme commented May 6, 2019 • edited Loading

vanzin left a comment

Choose a reason for hiding this comment

ajithme commented Aug 9, 2019

vanzin commented Aug 15, 2019

ajithme commented Aug 15, 2019

ajithme commented Jan 8, 2020

ajithme commented Jan 8, 2020

vanzin left a comment

Choose a reason for hiding this comment

ajithme commented Jan 10, 2020

vanzin commented Jan 10, 2020

vanzin commented Jan 10, 2020

SparkQA commented Jan 11, 2020

ajithme commented Jan 14, 2020

vanzin commented Jan 14, 2020

gengliangwang commented May 20, 2020

ajithme commented Mar 13, 2019 •

edited

Loading

gengliangwang commented Mar 20, 2019 •

edited

Loading

ajithme commented Mar 26, 2019 •

edited

Loading

ajithme commented May 6, 2019 •

edited

Loading