[SPARK-36292][SQL] Refactor third set of 20 in QueryExecutionErrors to use error classes by dgd-contributor · Pull Request #33863 · apache/spark

dgd-contributor · 2021-08-29T07:17:09Z

What changes were proposed in this pull request?

Refactor some exceptions in QueryExecutionErrors to use error classes.

cannotGenerateCodeForUnsupportedTypeError
cannotInterpolateClassIntoCodeBlockError
customCollectionClsNotResolvedError
classUnsupportedByMapObjectsError
nullAsMapKeyNotAllowedError
methodNotDeclaredError
constructorNotFoundError
primaryConstructorNotFoundError
unsupportedNaturalJoinTypeError
notExpectedUnresolvedEncoderError
unsupportedEncoderError
notOverrideExpectedMethodsError
failToConvertValueToJsonError
unexpectedOperatorInCorrelatedSubquery
unreachableError
unsupportedRoundingMode
resolveCannotHandleNestedSchema
inputExternalRowCannotBeNullError
fieldCannotBeNullMsg
fieldCannotBeNullError

Why are the changes needed?

There are currently ~350 exceptions in this file; so this PR only focuses on the third set of 20.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Existed UT Testcase

… error classes

AmplabJenkins · 2021-08-29T07:17:59Z

Can one of the admins verify this patch?

HyukjinKwon · 2021-08-30T01:10:29Z

cc @karenfeng FYI

karenfeng

Thanks for working on this @dgd-contributor! I left some comments.

karenfeng · 2021-09-07T21:19:14Z

core/src/main/resources/error/error-classes.json

+  },
+  "CLASS_UNSUPPORTED_BY_MAP_OBJECTS" : {
+    "message" : [ "Class `%s` is not supported by `MapObjects` as resulting collection." ],
+    "sqlState" : "42000"


0A000 may be a better fit here.

karenfeng · 2021-09-07T21:20:19Z

core/src/main/resources/error/error-classes.json

  "FAILED_SET_ORIGINAL_PERMISSION_BACK" : {
    "message" : [ "Failed to set original permission %s back to the created path: %s. Exception: %s" ]
  },
+  "FAIL_TO_CONVERT_VALUE_TO_JSON" : {


For consistency: FAIL -> FAILED?

karenfeng · 2021-09-07T21:21:12Z

core/src/main/resources/error/error-classes.json

    "sqlState" : "22023"
  },
+  "RESOLVE_CANNOT_HANDLE_NESTED_SCHEMA" : {
+    "message" : [ "Can not handle nested schema yet...  plan %s" ],


We should avoid making promises about future support. Can you remove yet?

karenfeng · 2021-09-07T21:21:46Z

core/src/main/resources/error/error-classes.json

    "message" : [ "Unable to acquire %s bytes of memory, got %s" ]
  },
+  "UNEXPECTED_OPERATOR_IN_CORRELATED_SUBQUERY" : {
+    "message" : [ "Unexpected operator %s in correlated subquery%s" ],


Nit: there should be a space between subquery and %s

karenfeng · 2021-09-07T21:21:57Z

core/src/main/resources/error/error-classes.json

+    "sqlState" : "42000"
+  },
+  "UNREACHABLE" : {
+    "message" : [ "This line should be unreachable%s" ],


Nit: there should be a space between unreachable and %s

karenfeng · 2021-09-07T21:22:25Z

core/src/main/resources/error/error-classes.json

    "sqlState" : "42000"
  },
+  "UNSUPPORTED_ENCODER" : {
+    "message" : [ "Only expression encoders are supported for now." ],


We should avoid making promises about future support. Can you remove for now?

karenfeng · 2021-09-07T21:23:03Z

core/src/main/resources/error/error-classes.json

  },
+  "UNSUPPORTED_ENCODER" : {
+    "message" : [ "Only expression encoders are supported for now." ],
+    "sqlState" : "42000"


0A000 may be a better fit for all "unsupported" errors.

karenfeng · 2021-09-07T21:23:06Z

core/src/main/resources/error/error-classes.json

  },
+  "UNSUPPORTED_NATURAL_JOIN_TYPE" : {
+    "message" : [ "Unsupported natural join type %s" ],
+    "sqlState" : "42000"


0A000 may be a better fit for all "unsupported" errors.

karenfeng · 2021-09-07T21:23:48Z

core/src/main/scala/org/apache/spark/SparkException.scala

+/**
+ * Unsupported operation exception thrown from Spark with an error class.
+ */
+class SparkUnsupportedOperationException(errorClass: String, messageParameters: Array[String])


We should make these classes package-private, eg.

spark/core/src/main/scala/org/apache/spark/SparkException.scala

Line 234 in f78d839

private[spark] class SparkSQLFeatureNotSupportedException(

karenfeng

Thanks for working on this @dgd-contributor! I left some comments, please tag me when you address them.

karenfeng · 2021-09-16T22:59:17Z

core/src/main/resources/error/error-classes.json

  },
+  "FAILED_TO_CONVERT_VALUE_TO_JSON" : {
+    "message" : [ "Failed to convert value %s (class of %s) with the type of %s to JSON." ],
+    "sqlState" : "42000"


22005 may be a better fit here.

karenfeng · 2021-09-16T23:00:16Z

core/src/main/resources/error/error-classes.json

@@ -55,7 +83,12 @@
    "sqlState" : "22023"
  },
  "INVALID_JSON_SCHEMA_MAPTYPE" : {


While we're here, can you also update: INVALID_JSON_SCHEMA_MAPTYPE -> INVALID_JSON_MAP_KEY_TYPE? The original class name is not very descriptive.

karenfeng · 2021-09-16T23:04:09Z

core/src/main/resources/error/error-classes.json

+    "message" : [ "Input schema %s can only contain StringType as a key type for a MapType." ],
+    "sqlState" : "42000"
+  },
+  "METHOD_NOT_DECLARED" : {


I don't think this is used? It's also already defined as MISSING_METHOD.

karenfeng · 2021-09-16T23:05:13Z

core/src/main/resources/error/error-classes.json

    "message" : [ "PARTITION clause cannot contain a non-partition column name: %s" ],
    "sqlState" : "42000"
  },
+  "NOT_EXPECTED_UNRESOLVED_ENCODER" : {


NOT_EXPECTED_UNRESOLVED_ENCODER -> UNEXPECTED_UNRESOLVED_ENCODER

karenfeng · 2021-09-16T23:05:22Z

core/src/main/resources/error/error-classes.json

  },
+  "NOT_EXPECTED_UNRESOLVED_ENCODER" : {
+    "message" : [ "Unresolved encoder expected, but %s was found." ],
+    "sqlState" : "42000"


This looks like an internal error, let's remove the SQLSTATE.

karenfeng · 2021-09-16T23:07:28Z

core/src/main/resources/error/error-classes.json

  },
+  "RESOLVE_CANNOT_HANDLE_NESTED_SCHEMA" : {
+    "message" : [ "Can not handle nested schema...  plan %s" ],
+    "sqlState" : "42000"


This looks like an internal error, let's remove the SQLSTATE.

karenfeng · 2021-09-16T23:07:58Z

core/src/main/resources/error/error-classes.json

    "message" : [ "Failed to rename as %s was not found" ],
    "sqlState" : "22023"
  },
+  "RESOLVE_CANNOT_HANDLE_NESTED_SCHEMA" : {


RESOLVE_CANNOT_HANDLE_NESTED_SCHEMA -> CANNOT_RESOLVE_NESTED_SCHEMA

karenfeng · 2021-09-16T23:08:39Z

core/src/main/resources/error/error-classes.json

+  },
+  "UNREACHABLE" : {
+    "message" : [ "This line should be unreachable %s" ],
+    "sqlState" : "42000"


This looks like an internal error, let's remove the SQLSTATE.

karenfeng · 2021-09-16T23:09:07Z

core/src/main/resources/error/error-classes.json

+    "message" : [ "Unexpected operator %s in correlated subquery %s" ],
+    "sqlState" : "42000"
+  },
+  "UNREACHABLE" : {


UNREACHABLE -> REACHED_UNREACHABLE_LINE - we should provide more detail

karenfeng · 2021-09-16T23:10:57Z

core/src/main/resources/error/error-classes.json

+    "message" : [ "Unsupported natural join type %s" ],
+    "sqlState" : "0A000"
+  },
+  "UNSUPPORTED_OPERATION_NOT_RESOLVED" : {


UNSUPPORTED_OPERATION_NOT_RESOLVED -> NOT_RESOLVED

…e error classes

dgd-contributor · 2021-09-28T04:53:23Z

cc @karenfeng, can you pls review it again?
Thank you.

karenfeng

I left a few more comments. Thanks for working on this @dgd-contributor, please ping me when you've addressed them.

karenfeng · 2021-09-30T18:47:15Z

core/src/main/resources/error/error-classes.json

  },
+  "CANNOT_GENERATE_CODE_FOR_UNSUPPORTED_TYPE" : {
+    "message" : [ "Cannot generate code for unsupported type: %s" ],
+    "sqlState" : "42000"


This should be 0A000 (unsupported)

karenfeng · 2021-09-30T18:47:46Z

core/src/main/resources/error/error-classes.json

  },
+  "PRIMARY_CONSTRUCTOR_NOT_FOUND" : {
+    "message" : [ "Couldn't find a primary constructor on %s" ],
+    "sqlState" : "42000"


karenfeng · 2021-09-30T18:47:54Z

core/src/main/resources/error/error-classes.json

    "message" : [ "Failed to rename as %s was not found" ],
    "sqlState" : "22023"
  },
+  "RESOLVE_CANNOT_HANDLE_NESTED_SCHEMA" : {


karenfeng · 2021-09-30T18:49:07Z

core/src/main/resources/error/error-classes.json

+    "message" : [ "Unsupported natural join type %s" ],
+    "sqlState" : "0A000"
+  },
+  "UNSUPPORTED_ROUNDING" : {


UNSUPPORTED_ROUNDING -> UNSUPPORTED_ROUNDING_MODE

karenfeng · 2021-09-30T18:50:20Z

core/src/main/resources/error/error-classes.json

+    "message" : [ "Unexpected operator %s in correlated subquery %s" ],
+    "sqlState" : "42000"
+  },
+  "UNEXPECTED_UNRESOLVED_ENCODER" : {


Whoops, I meant: UNEXPECTED_UNRESOLVED_ENCODER -> EXPECTED_UNRESOLVED_ENCODER

karenfeng · 2021-09-30T18:51:49Z

core/src/main/resources/error/error-classes.json

    "message" : [ "PARTITION clause cannot contain a non-partition column name: %s" ],
    "sqlState" : "42000"
  },
+  "NOT_RESOLVED" : {


In retrospect, this is specific to custom collection class resolution. Can we make this class name more specific? Eg. CUSTOM_COLLECTION_CLASS_NOT_RESOLVED

karenfeng · 2021-09-30T18:52:20Z

core/src/main/resources/error/error-classes.json

+    "message" : [ "Failed to convert value %s (class of %s) with the type of %s to JSON." ],
+    "sqlState" : "22005"
+  },
+  "FIELD_CANNOT_BE_NULL" : {


Let's make this more specific: ROW_FIELD_CANNOT_BE_NULL

dgd-contributor · 2021-10-04T03:48:49Z

Thank you for your review, @karenfeng. As we and @HyukjinKwon discussed contributing by sharing accounts, we will close this PR, and recreate this one with an individual account. We will also update the code with your comments.

dchvn added 2 commits August 27, 2021 16:54

[SPARK-36292]: Refactor third set of 20 query execution errors to use…

2037524

… error classes

[SPARK-36292]: Fix error output for test sql query

a823b9c

github-actions bot added CORE SQL labels Aug 29, 2021

karenfeng reviewed Sep 7, 2021

View reviewed changes

dchvn added 2 commits September 8, 2021 08:55

Merge branch 'master' into SPARK-36292-v2

072cd28

[SPARK-36292][SQL]: Correct some mistakes

449fa57

karenfeng reviewed Sep 16, 2021

View reviewed changes

dchvn added 3 commits September 23, 2021 11:09

[SPARK-36292]: Refactor third set of 20 in QueryExecutionErrors to us…

72800b5

…e error classes

[SPARK-36292][SQL] Fix error-classes.json format

7f94781

fix

65ad607

karenfeng reviewed Sep 30, 2021

View reviewed changes

phamhuyhoang97 and others added 2 commits October 1, 2021 11:17

fix error-classes

3e47530

fix conflict

0b83342

dgd-contributor closed this Oct 4, 2021

Conversation

dgd-contributor commented Aug 29, 2021

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

AmplabJenkins commented Aug 29, 2021

Uh oh!

HyukjinKwon commented Aug 30, 2021

Uh oh!

karenfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karenfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgd-contributor commented Sep 28, 2021

Uh oh!

karenfeng left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgd-contributor commented Oct 4, 2021

Uh oh!

Reviewers

Assignees