[SPARK-40098][SQL] Format error messages in the Thrift Server #37520

MaxGekk · 2022-08-15T08:20:56Z

What changes were proposed in this pull request?

In the PR, I propose to introduce new SQL config spark.sql.error.messageFormat to control the format of Spark SQL error messages. The config can have at least the following string values:

PRETTY - textual form of error messages including error class/sub-class, message itself + query context in text form.
MINIMAL - JSON representation of SparkThrowable. Compact form w/ minimal set of fields. Aims to be parsed machinery.
STANDARD - JSON form contains the same set of fields as MINIMAL + message.

Added new method getMessage() to SparkThrowableHelper which formats SparkThrowable + Throwable according to the requested format (see the SQL config). When the format is MINIMAL or STANDARD, the JSON record includes the following fields:

errorClass - human-readable, unique, and consistent representation of the error category. It is a string.
errorSubClass - optional sub-class of the error class as a string. It may be missing in the result string.
sqlState - optional string w/ error identifier across SQL engines. It may be missing in the result string.
message - the message template from error-classes.json. It is included to JSON when the format is STANDARD.
messageParameters - a map of message parameter names to its values.
queryContext - an array of query contexts of the error message which the following JSON fields. If it is empty, the array is not outputted:
- objectType - the string represents the object type of the query which throws the exception. If the exception is directly from the main query, it should be an empty string.
- objectName - the string represents the object name of the query which throws the exception. If the exception is directly from the main query, it should be an empty string.
- startIndex - the integer represents the starting index in the query text which throws the exception. The index starts from 1. If the index is unknown, the field is not outputted.
- stopIndex - the integer represents the stopping index in the query which throws the exception. The index starts from 1. If the index is unknown, the field is not outputted.
- fragment - the string represents the corresponding fragment of the query which throws the exception. If the query fragment cannot be identified, the string is empty.

For example:

MINIMAL

{"errorClass":"DIVIDE_BY_ZERO","sqlState":"22012","messageParameters":{"config":"\"spark.sql.ansi.enabled\""} ...

STANDARD

{
  "errorClass" : "DIVIDE_BY_ZERO",
  "message" : "Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set <config> to \"false\" to bypass this error.",
  "sqlState" : "22012",
  "messageParameters" : {
    "config" : "\"spark.sql.ansi.enabled\""
  },
...

When errorClass is null, Spark outputs the error message in the legacy form where:

errorClass is LEGACY
messageParameters is an array w/ only one elements as a string. The element contains textual error message.

In the case if the specified format is PRETTY, Spark follows existing behavior and outputs error message as plain text, for example:

[DIVIDE_BY_ZERO] Division by zero. Use `try_divide` to tolerate divisor being 0 and return NULL instead. If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
== SQL(line 1, position 8) ==
select 1 / 0
       ^^^^^

Also, this PR modifies the Thrift server to output the error messages according to current value of the SQL config spark.sql.error.messageFormat.

Why are the changes needed?

Spark SQL client tools need to parse out relevant information like parameters and positions in order to render messages to their liking. This makes the API fickle, hard to change, and limits degrees of freedom for UI.
Updates to error message texts result in a need to discover and update QA tests being affected. This makes it expensive to improve error-messages and prevents e.g. tech writers from taking ownership down the road.

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

By running new tests:

$ build/sbt "core/testOnly *SparkThrowableSuite"
$ build/sbt -Phive -Phive-thriftserver "test:testOnly *ThriftServerWithSparkContextInBinarySuite"
$ build/sbt -Phive -Phive-thriftserver "test:testOnly *ThriftServerWithSparkContextInHttpSuite"

MaxGekk · 2022-08-16T06:48:24Z

@srielau @anchovYu Please, take a look at the PR.

MaxGekk · 2022-08-16T13:47:05Z

@cloud-fan @gengliangwang @HyukjinKwon Could you review this PR, please.

core/src/main/scala/org/apache/spark/SparkThrowableHelper.scala

srielau · 2022-08-16T16:00:41Z

I want to point out that I'm not married to the labels MINIMAL, STANDARD, and PRETTY.... If someone has more appropriate names ..

srielau · 2022-08-16T16:03:20Z

"startIndex - the integer represents the starting index in the query text which throws the exception. The index starts from 0, but it can be -1 if the index is unknown.
stopIndex - the integer represents the stopping index in the query which throws the exception. The index starts from 0, but it can be -1 if the index is unknown."
Didn't we fix this recently to be 1/1 based?

srielau · 2022-08-16T16:04:46Z

Also this is a map:
messageParameters - an array of message parameters as a strings.

…rors

MaxGekk · 2022-08-16T18:34:12Z

Didn't we fix this recently to be 1/1 based?

@srielau If you mean #36651 but it increased the position in textual form of query context not in QueryContext itself. @gengliangwang Am I right?

gengliangwang · 2022-08-16T20:06:51Z

@srielau If you mean #36651 but it increased the position in textual form of query context not in QueryContext itself. @gengliangwang Am I right?

Yes I changed the position in the text summary.
For Json format output, I am ok with 0-based or 1-based. Either way makes sense...

MaxGekk · 2022-08-16T20:38:19Z

For Json format output, I am ok with 0-based or 1-based. Either way makes sense...

If we assume that the messages in the JSON format will be parsed machinery, it would be more natural to begin the indexes from 0, IMHO.

MaxGekk · 2022-08-17T06:29:56Z

After offline discussion with @srielau @cloud-fan , I am going to change startIndex and stopIndex to 1-based. And If the index is unknown, just don't output it as we do in the textual form.

core/src/main/scala/org/apache/spark/SparkThrowableHelper.scala

core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala

…rors

core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

…onf.scala Co-authored-by: Serge Rielau <srielau@users.noreply.github.com>

cloud-fan · 2022-08-18T04:34:03Z

core/src/main/scala/org/apache/spark/SparkThrowableHelper.scala

+            g.writeStringField(name, value)
+          }
+          g.writeEndObject()
+          g.writeArrayFieldStart("queryContext")


do we still need the queryContext field if there is no query context in the error?

I will remove it.

cloud-fan · 2022-08-18T04:34:32Z

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

+  val ERROR_MESSAGE_FORMAT = buildConf("spark.sql.error.messageFormat")
+    .doc("When PRETTY, the error message consists of textual representation of error class, " +
+      "message and query context. The MINIMAL and STANDARD formats are pretty JSON formats where " +
+      "STANDARD includes an additional JSON field `message`.")


let's also mention where this config applies: thriftserver and sql shell

In this logic, we should mention where every SQL config is applied for. I could mention the Thrift server as one of applications and of the config, only as an example.

cloud-fan · 2022-08-18T04:35:36Z

...rc/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerWithSparkContextSuite.scala

+          |    "objectName" : "",
+          |    "startIndex" : 8,
+          |    "stopIndex" : 12,
+          |    "fragment" : "1 / "


is it a bug? it should be 1 / 0

is this bug thrift-server only? if yes we can probably create a ticket and ask someone who knows thriftserver better to fix it.

It is not thrift-server specific:

withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") { val e = intercept[SparkArithmeticException] { sql("select 1 / 0").collect() } println("'" + e.getQueryContext()(0).fragment() + "'") } '1 / '

Seems like the issue in:

override lazy val fragment: String = { if (!isValid) { "" } else { sqlText.get.substring(originStartIndex.get, originStopIndex.get) } }

originStopIndex.get points to the last char of the fragment but substring() takes a sub-string up to endIndex - 1.

Let me open a PR and fix this separately.

Here is the JIRA for the issue: https://issues.apache.org/jira/browse/SPARK-40136

I have opened the PR w/ a bug fix: #37566

srielau · 2022-08-18T05:17:51Z

What is the reason for the limit? Will it break notebook and query editor? I think the point is that we want tools to start using it.

Sent from my iPhone On Aug 17, 2022, at 9:34 PM, Wenchen Fan ***@***.***> wrote: @cloud-fan commented on this pull request.

________________________________ In sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala<#37520 (comment)>:

@@ -3876,6 +3876,15 @@ object SQLConf {

.booleanConf .createWithDefault(false) + val ERROR_MESSAGE_FORMAT = buildConf("spark.sql.error.messageFormat") + .doc("When PRETTY, the error message consists of textual representation of error class, " + + "message and query context. The MINIMAL and STANDARD formats are pretty JSON formats where " + + "STANDARD includes an additional JSON field `message`.") let's also mention where this config applies: thriftserver and sql shell — Reply to this email directly, view it on GitHub<#37520 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA22CFGHCBQXN6WA4WU5XEDVZW4OLANCNFSM56RPFPYA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

…rors

cloud-fan · 2022-08-19T01:28:27Z

core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala

+      override val stopIndex = -1
+      override val fragment = "1 / 0"
+    }
+    val e = new SparkArithmeticException(


let's also test an exception with error class but without query context.

I will add a test w/ a sub-class but w/o query context. Thanks.

ah, do you mean before this PR thriftserver print error class twice?

…rors

MaxGekk · 2022-08-19T09:02:22Z

ImageFileFormatSuite failed again. I re-ran it locally. Seems like it is just a flaky test. Merging to master.
Thank you, @cloud-fan @srielau for review.

### What changes were proposed in this pull request? 1. Respect the SQL config `spark.sql.error.messageFormat` introduced by #37520, and output error messages in the one of format: `PRETTY` (by default), `MINIMAL` or `STANDARD`. 2. In the `PRETTY` format, output the error message of the `AnalysisException` exception in the same way as for other exceptions, i. e. w/o `Error in query:`. 3. Take into account the `silent` CLI option, and output the call stack only when it is `false`. In the `MINIMAL` and `STANDARD` formats don't print the call stack independently from the `silent` mode. ### Why are the changes needed? To respect the SQL config `spark.sql.error.messageFormat` and to be consistent to error outputs of the Thrift Server. ### Does this PR introduce _any_ user-facing change? Yes. The PR changes the behavior for `AnalysisException`. In that case, `spark-sql` does not output the prefix: `Error in query:` by default (format is PRETTY). Before: ```sql spark-sql> DROP TABLE ajhkdha; Error in query: Table or view not found: ajhkdha; line 1 pos 11; ``` After: ```sql spark-sql> DROP TABLE ajhkdha; Table or view not found: ajhkdha; line 1 pos 11; ``` ### How was this patch tested? By running the modified test suites: ``` $ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly org.apache.spark.sql.hive.thriftserver.CliSuite" ``` Closes #37590 from MaxGekk/spark-sql-error-json. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

cloud-fan · 2022-08-25T07:31:21Z

...iftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServerErrors.scala

@@ -36,11 +36,10 @@ object HiveThriftServerErrors {
      " new task for execution, please retry the operation", rejected)
  }

-  def runningQueryError(e: Throwable): Throwable = e match {
-    case st: SparkThrowable =>
-      val errorClassPrefix = Option(st.getErrorClass).map(e => s"[$e] ").getOrElse("")


I noticed that thriftserver already print the error class. Can we keep it when the format is PRETTY?

The PRETTY message includes the error class, see the test
https://github.com/apache/spark/pull/37520/files#diff-44a4c34050c5ff3cc3488e7e800ad41d791936c97c2f9535371931e43868741bR165

ah, do you mean before this PR thriftserver print error class twice?

Yep, it did.

cloud-fan · 2022-09-01T02:38:58Z

...iftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServerErrors.scala

-    case st: SparkThrowable =>
-      val errorClassPrefix = Option(st.getErrorClass).map(e => s"[$e] ").getOrElse("")
-      new HiveSQLException(
-        s"Error running query: ${errorClassPrefix}${st.toString}", st.getSqlState, st)


found a behavior change here: previously we call st.toString, but now SparkThrowableHelper.getMessage will call st.getMessage at the end.

Can we restore the previous behavior?

…es in the Thrift Server ### What changes were proposed in this pull request? In the PR, I propose: 1. Output errors in the PRETTY format in the same way as before the PR #37520. 2. Do not output non-JSON elements in the MINIMAL and STANDARD formats. ### Why are the changes needed? 1. To not break existing apps that might expect text errors in particular format. 3. Do not output extra text when the Thrift Server outputs errors in an JSON format. ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? By running the modified tests: ``` $ build/sbt -Phive -Phive-thriftserver "test:testOnly *ThriftServerWithSparkContextInBinarySuite" ``` Closes #37773 from MaxGekk/thrift-serv-json-errors-followup. Authored-by: Max Gekk <max.gekk@gmail.com> Signed-off-by: Max Gekk <max.gekk@gmail.com>

srielau · 2022-10-11T08:58:45Z

Please stick with 1. we should not change values based on formatting of the error.

…

Sent from my iPhone On Aug 16, 2022, at 1:38 PM, Maxim Gekk ***@***.***> wrote: For Json format output, I am ok with 0-based or 1-based. Either way makes sense... If we assume that the messages in the JSON format will be parsed machinery, it would be more natural to begin the indexes from 0, IMHO. — Reply to this email directly, view it on GitHub<#37520 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA22CFGO7PHMRABGOZVH6M3VZP34PANCNFSM56RPFPYA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

Add ErrorMessageFormat

2d3b073

github-actions bot added the CORE label Aug 15, 2022

MaxGekk added 5 commits August 15, 2022 12:11

Rename the file ErrorInfo.scala to SparkThrowableHelper.scala

f15dc89

Move ErrorMessageFormat out of the SparkThrowableHelper object

69e478b

Add a test

f19710d

Add the SQL config spark.sql.error.messageFormat

363323d

Modify message in runningQueryError

fd2ca73

github-actions bot added the SQL label Aug 15, 2022

Add a test for the Thrift server

3011fc1

MaxGekk changed the title ~~[WIP][SQL] Format error messages in the Thrift Server~~ [WIP][SPARK-40098][SQL] Format error messages in the Thrift Server Aug 16, 2022

MaxGekk changed the title ~~[WIP][SPARK-40098][SQL] Format error messages in the Thrift Server~~ [SPARK-40098][SQL] Format error messages in the Thrift Server Aug 16, 2022

MaxGekk marked this pull request as ready for review August 16, 2022 06:47

MaxGekk requested review from cloud-fan, gengliangwang and HyukjinKwon August 16, 2022 06:47

cloud-fan reviewed Aug 16, 2022

View reviewed changes

core/src/main/scala/org/apache/spark/SparkThrowableHelper.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Aug 16, 2022

View reviewed changes

core/src/main/scala/org/apache/spark/SparkThrowableHelper.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Aug 16, 2022

View reviewed changes

core/src/main/scala/org/apache/spark/SparkThrowableHelper.scala Outdated Show resolved Hide resolved

MaxGekk added 2 commits August 16, 2022 20:11

Merge remote-tracking branch 'origin/master' into thrift-serv-json-er…

5bc104c

…rors

Use SparkException.internalError

e8c7182

Use Jackson instead of json4s

247b7e2

1-based indexes

254aa75

cloud-fan reviewed Aug 17, 2022

View reviewed changes

core/src/main/scala/org/apache/spark/SparkThrowableHelper.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Aug 17, 2022

View reviewed changes

core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala Outdated Show resolved Hide resolved

MaxGekk added 2 commits August 17, 2022 13:01

Print MINIMAL as pretty JSON

671e49d

Merge remote-tracking branch 'origin/master' into thrift-serv-json-er…

abbaa7b

…rors

srielau approved these changes Aug 17, 2022

View reviewed changes

core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala Outdated Show resolved Hide resolved

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala Outdated Show resolved Hide resolved

MaxGekk and others added 2 commits August 17, 2022 19:49

Update sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLC…

3c26416

…onf.scala Co-authored-by: Serge Rielau <srielau@users.noreply.github.com>

legacy -> LEGACY

e89e355

cloud-fan reviewed Aug 18, 2022

View reviewed changes

MaxGekk added 3 commits August 18, 2022 09:52

Address Wenchen's comments

c871fdd

Merge remote-tracking branch 'origin/master' into thrift-serv-json-er…

d3f03ca

…rors

Fix ThriftServerWithSparkContextSuite

9bf3cb4

cloud-fan reviewed Aug 19, 2022

View reviewed changes

cloud-fan approved these changes Aug 19, 2022

View reviewed changes

MaxGekk added 2 commits August 19, 2022 08:34

Test w/o query context

a5724ea

Merge remote-tracking branch 'origin/master' into thrift-serv-json-er…

f4ee4f5

…rors

MaxGekk closed this in abb4345 Aug 19, 2022

MaxGekk mentioned this pull request Aug 22, 2022

[SPARK-40180][SQL] Format error messages by spark-sql #37590

Closed

cloud-fan reviewed Aug 25, 2022

View reviewed changes

cloud-fan reviewed Sep 1, 2022

View reviewed changes

MaxGekk mentioned this pull request Sep 2, 2022

[SPARK-40098][SQL][FOLLOWUP] Revert the pretty format of error messages in the Thrift Server #37773

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-40098][SQL] Format error messages in the Thrift Server #37520

[SPARK-40098][SQL] Format error messages in the Thrift Server #37520

MaxGekk commented Aug 15, 2022 •

edited

MaxGekk commented Aug 16, 2022

MaxGekk commented Aug 16, 2022

srielau commented Aug 16, 2022

srielau commented Aug 16, 2022

srielau commented Aug 16, 2022

MaxGekk commented Aug 16, 2022

gengliangwang commented Aug 16, 2022

MaxGekk commented Aug 16, 2022

MaxGekk commented Aug 17, 2022

cloud-fan Aug 18, 2022

MaxGekk Aug 18, 2022

cloud-fan Aug 18, 2022

MaxGekk Aug 18, 2022

cloud-fan Aug 18, 2022

cloud-fan Aug 18, 2022

MaxGekk Aug 18, 2022

cloud-fan Aug 18, 2022

MaxGekk Aug 18, 2022

MaxGekk Aug 18, 2022

MaxGekk Aug 18, 2022

MaxGekk Aug 18, 2022

srielau commented Aug 18, 2022 via email

cloud-fan Aug 19, 2022

MaxGekk Aug 19, 2022

cloud-fan Aug 25, 2022

MaxGekk commented Aug 19, 2022 •

edited

cloud-fan Aug 25, 2022 •

edited

MaxGekk Aug 25, 2022

MaxGekk Aug 25, 2022

cloud-fan Sep 1, 2022

srielau commented Oct 11, 2022 via email

[SPARK-40098][SQL] Format error messages in the Thrift Server #37520

[SPARK-40098][SQL] Format error messages in the Thrift Server #37520

Conversation

MaxGekk commented Aug 15, 2022 • edited

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

MaxGekk commented Aug 16, 2022

MaxGekk commented Aug 16, 2022

srielau commented Aug 16, 2022

srielau commented Aug 16, 2022

srielau commented Aug 16, 2022

MaxGekk commented Aug 16, 2022

gengliangwang commented Aug 16, 2022

MaxGekk commented Aug 16, 2022

MaxGekk commented Aug 17, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srielau commented Aug 18, 2022 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MaxGekk commented Aug 19, 2022 • edited

cloud-fan Aug 25, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srielau commented Oct 11, 2022 via email

MaxGekk commented Aug 15, 2022 •

edited

MaxGekk commented Aug 19, 2022 •

edited

cloud-fan Aug 25, 2022 •

edited