Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-45891][SQL][FOLLOW-UP] Added length check to the is_variant_null expression #46311

Closed
wants to merge 26 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
81f4866
Implemented is_variant_null expression
harshmotw-db Apr 11, 2024
3673ef7
Merge branch 'apache:master' into master
harshmotw-db Apr 11, 2024
7acb773
minor change
harshmotw-db Apr 11, 2024
e1e561d
Regenerated golden files
harshmotw-db Apr 12, 2024
c2329f2
Fixed some comments left by Chenhao (one still remaining)
harshmotw-db Apr 12, 2024
cb973a9
Merge branch 'apache:master' into master
harshmotw-db Apr 12, 2024
f82c11a
Fixed major bug in the StaticInvoke call in IsVariantNull
harshmotw-db Apr 12, 2024
85afe57
Added is_variant_null tests in VariantEndToEndSuite where it is worki…
harshmotw-db Apr 12, 2024
506dd20
Adding one more test to re trigger CI/CD tests. Currently, there are …
harshmotw-db Apr 12, 2024
c0607f0
regenerated golden files again
harshmotw-db Apr 12, 2024
7767e55
Merge branch 'master' into master
harshmotw-db Apr 15, 2024
b32b78d
Merge branch 'master' into master
harshmotw-db Apr 16, 2024
f53ee04
Resolved merge conflicts
harshmotw-db Apr 18, 2024
764e633
Merge branch 'apache:master' into master
harshmotw-db Apr 19, 2024
a3ceee9
Implemented try_parse_json
harshmotw-db Apr 19, 2024
731f6b5
Merge branch 'apache:master' into master
harshmotw-db Apr 20, 2024
ee98221
Merge branch 'master' of https://github.com/harshmotw-db/spark
harshmotw-db Apr 20, 2024
7b25cb7
Merge branch 'apache:master' into master
harshmotw-db Apr 22, 2024
5534627
Merge branch 'apache:master' into master
harshmotw-db Apr 25, 2024
a86908a
Merge branch 'apache:master' into master
harshmotw-db Apr 30, 2024
c42372f
Merge branch 'apache:master' into master
harshmotw-db Apr 30, 2024
8be9630
Added length check in is_variant_null
harshmotw-db Apr 30, 2024
bdf1aca
Minor changes
harshmotw-db Apr 30, 2024
c90e39e
minor changes
harshmotw-db Apr 30, 2024
e07be61
minor changes
harshmotw-db Apr 30, 2024
d526ca6
Resolved comments raised by Dongjoon
harshmotw-db May 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,13 @@ object VariantExpressionEvalUtils {
false
} else {
val variantValue = input.getValue
// Variant NULL is denoted by basic_type == 0 and val_header == 0
variantValue(0) == 0
}
if (variantValue.isEmpty) {
throw QueryExecutionErrors.malformedVariant()
} else {
// Variant NULL is denoted by basic_type == 0 and val_header == 0
variantValue(0) == 0
}
}
}

/** Cast a Spark value from `dataType` into the variant type. */
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2727,6 +2727,11 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
messageParameters = Map("path" -> path, "functionName" -> toSQLId(functionName)))
}

def malformedVariant(): Throwable = new SparkRuntimeException(
"MALFORMED_VARIANT",
Map.empty
)

def invalidCharsetError(functionName: String, charset: String): RuntimeException = {
new SparkIllegalArgumentException(
errorClass = "INVALID_PARAMETER_VALUE.CHARSET",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,13 @@ class VariantExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
check(expectedResult4, smallObject, smallMetadata)
}

test("is_variant_null invalid input") {
checkErrorInExpression[SparkRuntimeException](
IsVariantNull(Literal(new VariantVal(Array(), Array(1, 2, 3)))),
"MALFORMED_VARIANT"
)
}

private def parseJson(input: String): VariantVal =
VariantExpressionEvalUtils.parseJson(UTF8String.fromString(input))

Expand Down