[WIP][SQL] Replace require() by an internal error in catalyst#38896
[WIP][SQL] Replace require() by an internal error in catalyst#38896MaxGekk wants to merge 9 commits intoapache:masterfrom
require() by an internal error in catalyst#38896Conversation
| left: Expression, | ||
| right: Expression, | ||
| override val dataType: DataType) extends BinaryOperator { | ||
| require(dataType.isInstanceOf[DecimalType]) |
There was a problem hiding this comment.
This is not needed since we require the type below in the method inputType()
| * Tests an condition, throwing an `SparkException` with | ||
| * the error class `INTERNAL_ERROR` if false. | ||
| */ | ||
| def checkInternalError(condition: Boolean, msg: String): Unit = { |
There was a problem hiding this comment.
Is this tiny utility method worth it? not sure
There was a problem hiding this comment.
I want to be as close as possible to the existing require and don't introduce unnecessary boilerplate code.
| */ | ||
| class ArrayBasedMapData(val keyArray: ArrayData, val valueArray: ArrayData) extends MapData { | ||
| require(keyArray.numElements() == valueArray.numElements()) | ||
| checkInternalError( |
There was a problem hiding this comment.
This one is an user-facing error, actually. See the failed test:
JsonFunctionsSuite.SPARK-33134: return partial results only for root JSON objects
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 186.0 failed 1 times, most recent failure: Lost task 0.0 in stage 186.0 (TID 227) (localhost executor driver): org.apache.spark.SparkException: [INTERNAL_ERROR] The number of key elements 1 and value elements 0 must be the same.
at org.apache.spark.SparkException$.internalError(SparkException.scala:77)
at org.apache.spark.SparkException$.internalError(SparkException.scala:81)
at org.apache.spark.SparkException$.checkInternalError(SparkException.scala:97)
| * are different. | ||
| */ | ||
| class ArrayBasedMapData(val keyArray: ArrayData, val valueArray: ArrayData) extends MapData { | ||
| require(keyArray.numElements() == valueArray.numElements()) |
There was a problem hiding this comment.
I have to leave it as is because JSON datasource expects IllegalArgumentException otherwise it propagates SparkException w/ INTERNAL_ERROR to users.
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
2 participants
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?