[SPARK-28556][SQL] QueryExecutionListener should also notify Error by zsxwing · Pull Request #25292 · apache/spark

zsxwing · 2019-07-29T20:34:37Z

What changes were proposed in this pull request?

Right now Error is not sent to QueryExecutionListener.onFailure. If there is any Error (such as AssertionError) when running a query, QueryExecutionListener.onFailure cannot be triggered.

This PR changes onFailure to accept a Throwable instead.

How was this patch tested?

Jenkins

zsxwing · 2019-07-29T20:35:18Z

sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala

   */
  @DeveloperApi
-  def onFailure(funcName: String, qe: QueryExecution, exception: Exception): Unit
+  def onFailure(funcName: String, qe: QueryExecution, error: Throwable): Unit


This will need a release note.

There is another issue in this API: It passes a private class QueryExecution to the user in a public API. Didn't fix it as it will require a re-design of this API.

Yea, we should sort those problems out. There's another occurrence at Dataset.queryExecution as an API.

I'm a bit concerned about this change. This broke one of my project that uses this API. Instead of changing the signature here, did we consider just wrapping the exception?

RE: QueryExecution I think that has become a developer API at this point. I think a lot of developers use it to debug things.

wrapping the exception?

This is a good suggestion. This can avoid api changes, and we can also fix this bug that errors are not sent to the listener in 2.4.

SparkQA · 2019-07-29T20:47:10Z

Test build #108345 has finished for PR 25292 at commit 82e1800.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala

SparkQA · 2019-07-29T21:16:50Z

Test build #108348 has finished for PR 25292 at commit 8fb2c5f.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-07-29T23:43:00Z

Test build #108349 has finished for PR 25292 at commit bdc2c26.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun

+1, LGTM.

HyukjinKwon

LGTM too

HyukjinKwon · 2019-07-30T02:47:18Z

Merged to master.

dongjoon-hyun · 2020-01-29T06:42:49Z

Hi, @gatorsmile , @zsxwing , @HyukjinKwon .
If we need to mention this at 2.4.5 release-note, please let me know.
2.4.5 is the last version before 3.0.0.

dongjoon-hyun · 2020-03-11T23:58:56Z

For the following, cc @rxin since he is a release manager for 3.0.0. (Also, cc @gatorsmile ).

[SPARK-28556][SQL] QueryExecutionListener should also notify Error #25292 (comment)

dongjoon-hyun · 2020-03-12T00:01:16Z

Hi, @marmbrus .
This was discussed legitimately at that time here and on JIRA.
If you want to revert this, please make a new legitimate PR. We will review that carefully according to the community policy (AS-IS).

marmbrus · 2020-03-12T00:47:51Z

I don't see any discussion of alternatives to breaking the API here or on JIRA. I'm just asking that we discuss. I don't see anything "illegitimate" about that... We can open a JIRA after others have a chance to chime in.

zsxwing · 2020-03-13T06:31:18Z

I didn't think about wrapping the exception. Totally agree that this is actually a better solution. It can avoid breaking APIs , and in addition, the same fix can be applied to maintenance branches (such as branch-2.4). I will create a new JIRA ticket for this alternative solution.

zsxwing · 2020-03-13T06:36:49Z

Created https://issues.apache.org/jira/browse/SPARK-31144

…QueryExecutionListener ### What changes were proposed in this pull request? This PR manually reverts changes in #25292 and then wraps java.lang.Error with `QueryExecutionException` to notify `QueryExecutionListener` to send it to `QueryExecutionListener.onFailure` which only accepts `Exception`. The bug fix PR for 2.4 is #27904. It needs a separate PR because the touched codes were changed a lot. ### Why are the changes needed? Avoid API changes and fix a bug. ### Does this PR introduce any user-facing change? Yes. Reverting an API change happening in 3.0. QueryExecutionListener APIs will be the same as 2.4. ### How was this patch tested? The new added test. Closes #27907 from zsxwing/SPARK-31144. Authored-by: Shixiong Zhu <zsxwing@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 1ddf44d) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…QueryExecutionListener ### What changes were proposed in this pull request? This PR manually reverts changes in #25292 and then wraps java.lang.Error with `QueryExecutionException` to notify `QueryExecutionListener` to send it to `QueryExecutionListener.onFailure` which only accepts `Exception`. The bug fix PR for 2.4 is #27904. It needs a separate PR because the touched codes were changed a lot. ### Why are the changes needed? Avoid API changes and fix a bug. ### Does this PR introduce any user-facing change? Yes. Reverting an API change happening in 3.0. QueryExecutionListener APIs will be the same as 2.4. ### How was this patch tested? The new added test. Closes #27907 from zsxwing/SPARK-31144. Authored-by: Shixiong Zhu <zsxwing@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…QueryExecutionListener ### What changes were proposed in this pull request? This PR manually reverts changes in apache#25292 and then wraps java.lang.Error with `QueryExecutionException` to notify `QueryExecutionListener` to send it to `QueryExecutionListener.onFailure` which only accepts `Exception`. The bug fix PR for 2.4 is apache#27904. It needs a separate PR because the touched codes were changed a lot. ### Why are the changes needed? Avoid API changes and fix a bug. ### Does this PR introduce any user-facing change? Yes. Reverting an API change happening in 3.0. QueryExecutionListener APIs will be the same as 2.4. ### How was this patch tested? The new added test. Closes apache#27907 from zsxwing/SPARK-31144. Authored-by: Shixiong Zhu <zsxwing@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

QueryExecutionListener should also notify Error

82e1800

zsxwing added the SQL label Jul 29, 2019

zsxwing requested a review from cloud-fan July 29, 2019 20:34

zsxwing commented Jul 29, 2019

View reviewed changes

dongjoon-hyun changed the title ~~[SPARK-28556][SQL]QueryExecutionListener should also notify Error~~ [SPARK-28556][SQL] QueryExecutionListener should also notify Error Jul 29, 2019

dongjoon-hyun reviewed Jul 29, 2019

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/util/QueryExecutionListener.scala Outdated Show resolved Hide resolved

fix-tests

8fb2c5f

fix mima

bdc2c26

dongjoon-hyun approved these changes Jul 30, 2019

View reviewed changes

HyukjinKwon approved these changes Jul 30, 2019

View reviewed changes

HyukjinKwon closed this in 196a4d7 Jul 30, 2019

zsxwing deleted the fix-QueryExecutionListener branch July 30, 2019 05:43

zsxwing mentioned this pull request Mar 13, 2020

[SPARK-31144][SQL]Wrap Error with QueryExecutionException to notify QueryExecutionListener #27907

Closed

Conversation

zsxwing commented Jul 29, 2019

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

zsxwing Jul 29, 2019

Choose a reason for hiding this comment

Uh oh!

zsxwing Jul 29, 2019

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon Jul 30, 2019

Choose a reason for hiding this comment

Uh oh!

marmbrus Mar 11, 2020

Choose a reason for hiding this comment

Uh oh!

marmbrus Mar 11, 2020

Choose a reason for hiding this comment

Uh oh!

zsxwing Mar 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Jul 29, 2019

Uh oh!

Uh oh!

SparkQA commented Jul 29, 2019

Uh oh!

SparkQA commented Jul 29, 2019

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon left a comment

Choose a reason for hiding this comment

Uh oh!

HyukjinKwon commented Jul 30, 2019

Uh oh!

dongjoon-hyun commented Jan 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Mar 11, 2020

Uh oh!

dongjoon-hyun commented Mar 12, 2020

Uh oh!

marmbrus commented Mar 12, 2020

Uh oh!

zsxwing commented Mar 13, 2020

Uh oh!

zsxwing commented Mar 13, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zsxwing Mar 12, 2020 •

edited

Loading

dongjoon-hyun commented Jan 29, 2020 •

edited

Loading