[SPARK-45710][SQL] Assign names to error _LEGACY_ERROR_TEMP_21[59,60,61,62] #43567
Conversation
|
The use of the 4 similar errors is a bit confusing, we have separate errors for flatten/concat/union, but we are using |
dengziming
force-pushed
the
SPARK-45710
branch
from
6c3cfba
to
921b9f1
Compare
There was a problem hiding this comment.
@dengziming There is another similar error class TOO_MANY_ARRAY_ELEMENTS. I would create new error class like COLLECTION_SIZE_LIMIT_EXCEEDED with sub-classes.
dengziming
force-pushed
the
SPARK-45710
branch
from
921b9f1
to
9827199
Compare
There was a problem hiding this comment.
LGTM except of some comments. @dengziming Please, update PR's description according to your recent changes.
| }, | ||
| "INITIALIZE" : { | ||
| "message" : [ | ||
| "cannot initialize array with specified parameters." |
There was a problem hiding this comment.
| "cannot initialize array with specified parameters." | |
| "cannot initialize an array with specified parameters." |
| "subClass" : { | ||
| "FUNCTION" : { | ||
| "message" : [ | ||
| "unsuccessful try to create arrays in function <functionName>." |
There was a problem hiding this comment.
| "unsuccessful try to create arrays in function <functionName>." | |
| "unsuccessful try to create arrays in the function <functionName>." |
| }, | ||
| "PARAMETER" : { | ||
| "message" : [ | ||
| "the value of parameter(s) <parameter> in <functionName> is invalid." |
There was a problem hiding this comment.
| "the value of parameter(s) <parameter> in <functionName> is invalid." | |
| "the value of parameter(s) <parameter> in the function <functionName> is invalid." |
sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala
Outdated
Show resolved
Hide resolved
| new SparkIllegalArgumentException( | ||
| errorClass = "TOO_MANY_ARRAY_ELEMENTS", | ||
| errorClass = "COLLECTION_SIZE_LIMIT_EXCEEDED", |
|
The failed python GA is not related to the changes, and it passed a couple commits above. |
What changes were proposed in this pull request?
This PR are removing
_LEGACY_ERROR_TEMP_21[59,60,61,62]andTOO_MANY_ARRAY_ELEMENTS:_LEGACY_ERROR_TEMP_2159is used in concat/array_insert;_LEGACY_ERROR_TEMP_2160is only used in flatten;_LEGACY_ERROR_TEMP_2161is used in array_repeat/array_insert/array_distinct/array_union/array_intersect/array_remove;_LEGACY_ERROR_TEMP_2162is used in array_union/array_distinct;TOO_MANY_ARRAY_ELEMENTSwhich are used inUnsafeArrayWriter.java.I removed these 5 similar error classes and create a new error class
COLLECTION_SIZE_LIMIT_EXCEEDEDwith 3 sub-classes:PARAMETERis used when the parameter exceed size limit, such asarray_repeatwith count too large;FUNCTIONis used when trying to create an array exceeding size limit in a function, for example, flatten 2 arrays to a larger array;INITIALIZEis used inUnsafeArrayWriter.javawhen trying to initialize an array exceeding size limit.Why are the changes needed?
To assign proper name as a part of activity in SPARK-37935.
Does this PR introduce any user-facing change?
Yes, the error message will include the error class name.
How was this patch tested?
COLLECTION_SIZE_LIMIT_EXCEEDED.PARAMETERcan be tested from use code;COLLECTION_SIZE_LIMIT_EXCEEDED.FUNCTIONis tested using aColumnarArrayinconcat/flatten, but can't be tested inarray_insert/array_distinct/array_union/array_intersect/array_removesince we need to deduplicate the data and create an array which will cause OOM.INITIALIZEis already tested in a existing case.Was this patch authored or co-authored using generative AI tooling?
No.