[SPARK-47413][SQL] - add support to substr/left/right for collations #46040

GideonPotok · 2024-04-13T09:38:26Z

https://issues.apache.org/jira/browse/SPARK-46830

What changes were proposed in this pull request?

Add collation support to types of return values for calls to substr, left, right, when passed in arguments of an explicit, implicit, or session-specified collations. Add tests to validate behavior.

Why are the changes needed?

We are incrementally adding collation support to built-in string functions in Spark. These functions are intended to be supported for collated types.

Does this PR introduce any user-facing change?

these sql functions will now not throw errors when passed in collated types. Instead, they will return the right value, of the passed in type. Or of the default collation.

How was this patch tested?

Unit testing + ad-hoc spark shell and pyspark shell interactions.

Was this patch authored or co-authored using generative AI tooling?

No.

GideonPotok · 2024-04-14T08:15:41Z

@uros-db please review

HyukjinKwon · 2024-04-15T00:05:45Z

What's idff w/ #46039?

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala

GideonPotok · 2024-04-15T08:54:01Z

@uros-db I made all suggested changes. Please re-review. Thanks!

uros-db · 2024-04-15T09:00:15Z

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala

+        SubstringTestCase("select left('abc' collate " + c + ", 1)", c, Row("a")),
+        SubstringTestCase("select right('def' collate " + c + ", 1)", c, Row("f")),
+        SubstringTestCase("select substr('abc' collate " + c + ", 2)", c, Row("bc")),
+        SubstringTestCase("select substr('example' collate " + c + ", 0, 2)", c, Row("ex")),
+        SubstringTestCase("select substr('example' collate " + c + ", 1, 2)", c, Row("ex")),
+        SubstringTestCase("select substr('example' collate " + c + ", 0, 7)", c, Row("example")),
+        SubstringTestCase("select substr('example' collate " + c + ", 1, 7)", c, Row("example")),
+        SubstringTestCase("select substr('example' collate " + c + ", 0, 100)", c, Row("example")),
+        SubstringTestCase("select substr('example' collate " + c + ", 1, 100)", c, Row("example")),
+        SubstringTestCase("select substr('example' collate " + c + ", 2, 2)", c, Row("xa")),
+        SubstringTestCase("select substr('example' collate " + c + ", 1, 6)", c, Row("exampl")),
+        SubstringTestCase("select substr('example' collate " + c + ", 2, 100)", c, Row("xample")),
+        SubstringTestCase("select substr('example' collate " + c + ", 0, 0)", c, Row("")),
+        SubstringTestCase("select substr('example' collate " + c + ", 100, 4)", c, Row("")),
+        SubstringTestCase("select substr('example' collate " + c + ", 0, 100)", c, Row("example")),
+        SubstringTestCase("select substr('example' collate " + c + ", 1, 100)", c, Row("example")),
+        SubstringTestCase("select substr('example' collate " + c + ", 2, 100)", c, Row("xample")),
+        SubstringTestCase("select substr('example' collate " + c + ", -3, 2)", c, Row("pl")),
+        SubstringTestCase("select substr('example' collate " + c + ", -100, 4)", c, Row("")),
+        SubstringTestCase("select substr('example' collate " + c + ", -2147483648, 6)", c, Row("")),
+        SubstringTestCase("select substr(' a世a ' collate " + c + ", 2, 3)", c, Row("a世a")), // scalastyle:ignore
+        SubstringTestCase("select left(' a世a ' collate " + c + ", 3)", c, Row(" a世")), // scalastyle:ignore
+        SubstringTestCase("select right(' a世a ' collate " + c + ", 3)", c, Row("世a ")), // scalastyle:ignore
+        SubstringTestCase("select substr('AaAaAaAa000000' collate " + c + ", 2, 3)", c, Row("aAa")),
+        SubstringTestCase("select left('AaAaAaAa000000' collate " + c + ", 3)", c, Row("AaA")),
+        SubstringTestCase("select right('AaAaAaAa000000' collate " + c + ", 3)", c, Row("000")),
+        SubstringTestCase("select substr('' collate " + c + ", 1, 1)", c, Row("")),
+        SubstringTestCase("select left('' collate " + c + ", 1)", c, Row("")),
+        SubstringTestCase("select right('' collate " + c + ", 1)", c, Row("")),
+        SubstringTestCase("select left('ghi' collate " + c + ", 1)", c, Row("g"))


I don't think we need this many test cases here, if you didn't modify the way Substring/Left/Right expressions behave when given collated strings (i.e. you didn't introduce any collation awareness to nullSafeEval/doCodeGen), then there should be no need to go this deep - a couple of test cases should do the trick just fine

also, I think these tests can be combined with the one above to make:
test("Support Left/Right/Substr with collation") {

so that we could have something like:

checks.foreach { check => // Result & data type (explicit collation) ... // Result & data type (implicit collation) ...

Will do (But without implicit collation at all, as per your other review comment.)

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala

GideonPotok · 2024-04-15T10:12:06Z

What's idff w/ #46039?

@HyukjinKwon Sorry for confusion. I closed out the other one to clear up which one to review.

(To answer your question, that other one did not have the unit test over a struct field, because in the past I have dealt with GHA Test flakiness over using withTable expressions. )

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala

GideonPotok · 2024-04-16T06:35:54Z

@uros-db please re-review!

GideonPotok · 2024-04-17T12:00:07Z

@uros-db please re-review this one too.

uros-db · 2024-04-17T12:15:56Z

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala

+        SubstringTestCase("select substr('example' collate " + c + ", 1, 100)", c, Row("example")),
+        SubstringTestCase("select substr('example' collate " + c + ", 2, 2)", c, Row("xa")),
+        SubstringTestCase("select substr('example' collate " + c + ", 0, 0)", c, Row("")),
+        SubstringTestCase("select substr('example' collate " + c + ", -3, 2)", c, Row("pl")),
+        SubstringTestCase("select substr(' a世a ' collate " + c + ", 2, 3)", c, Row("a世a")), // scalastyle:ignore
+        SubstringTestCase("select left(' a世a ' collate " + c + ", 3)", c, Row(" a世")), // scalastyle:ignore
+        SubstringTestCase("select right(' a世a ' collate " + c + ", 3)", c, Row("世a ")), // scalastyle:ignore
+        SubstringTestCase("select left('AaAaAaAa000000' collate " + c + ", 3)", c, Row("AaA")),
+        SubstringTestCase("select right('AaAaAaAa000000' collate " + c + ", 3)", c, Row("000")),
+        SubstringTestCase("select substr('' collate " + c + ", 1, 1)", c, Row("")),
+        SubstringTestCase("select left('' collate " + c + ", 1)", c, Row("")),
+        SubstringTestCase("select right('' collate " + c + ", 1)", c, Row("")),
+        // improper values
+        SubstringTestCase("select left(null collate " + c + ", 1)", c, Row(null)),
+        SubstringTestCase("select right(null collate " + c + ", 1)", c, Row(null)),
+        SubstringTestCase("select substr(null collate " + c + ", 1)", c, Row(null)),
+        SubstringTestCase("select substr(null collate " + c + ", 1, 1)", c, Row(null)),
+        SubstringTestCase("select left(null collate " + c + ", null)", c, Row(null)),
+        SubstringTestCase("select right(null collate " + c + ", null)", c, Row(null)),
+        SubstringTestCase("select substr(null collate " + c + ", null)", c, Row(null)),
+        SubstringTestCase("select substr(null collate " + c + ", null, null)", c, Row(null)),
+        SubstringTestCase("select left('AaAaAaAa000000' collate " + c + ", null)", c, Row(null)),
+        SubstringTestCase("select right('AaAaAaAa000000' collate " + c + ", null)", c, Row(null)),
+        SubstringTestCase("select substr('AaAaAaAa000000' collate " + c + ", null)", c, Row(null)),
+        SubstringTestCase("select substr('AaAaAaAa0' collate " + c + ", null, null)", c, Row(null)),
+        SubstringTestCase("select right('' collate " + c + ", null)", c, Row(null)),
+        SubstringTestCase("select substr('' collate " + c + ", null)", c, Row(null)),
+        SubstringTestCase("select substr('' collate " + c + ", null, null)", c, Row(null)),
+        SubstringTestCase("select left('' collate " + c + ", null)", c, Row(null))


28 cases * 4 collations = 112 tests

I'd say we don't need that many SQL tests, there's no need to do Seq("utf8_binary_lcase", "utf8_binary", "unicode", "unicode_ci").flatMap, only 4 tests (with valid values) per function (substring/left/right) should be enough

a couple of additional tests for improper values are fine as well, but we don't need to test every possible pair of collation & function

since we don't have unit tests for these functions, let's make sure to use a smaller number of tests here in order to test more things - for example, I don't see any case/accent variation here, as well as a wider variety of variable len characters, etc.

@uros-db done. Please re-review

uros-db

fix tests

uros-db · 2024-04-17T17:37:41Z

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala

+        "select substr('example' collate " + "utf8_binary_lcase" + ", 1, 100)",
+        "utf8_binary_lcase",
+        Row("example")),
+      SubstringTestCase(
+        "select substr('example' collate " + "utf8_binary" + ", 2, 2)",
+        "utf8_binary",
+        Row("xa")),
+      SubstringTestCase(
+        "select right('' collate " + "utf8_binary_lcase" + ", 1)",
+        "utf8_binary_lcase",
+        Row("")),
+      SubstringTestCase(
+        "select substr('example' collate " + "unicode" + ", 0, 0)",
+        "unicode",
+        Row("")),
+      SubstringTestCase(
+        "select substr('example' collate " + "unicode_ci" + ", -3, 2)",
+        "unicode_ci",
+        Row("pl")),
+      SubstringTestCase(
+        "select substr(' a世a ' collate " + "utf8_binary_lcase" + ", 2, 3)", // scalastyle:ignore
+        "utf8_binary_lcase",
+        Row("a世a")), // scalastyle:ignore
+      SubstringTestCase(
+        "select left(' a世a ' collate " + "utf8_binary" + ", 3)", // scalastyle:ignore
+        "utf8_binary",
+        Row(" a世")), // scalastyle:ignore
+      SubstringTestCase(
+        "select right(' a世a ' collate " + "unicode" + ", 3)", // scalastyle:ignore
+        "unicode",
+        Row("世a ")), // scalastyle:ignore
+      SubstringTestCase(
+        "select left('ÀÃÂĀĂȦÄäåäáâãȻȻȻȻȻǢǼÆ' collate " + "unicode_ci" + ", 3)", // scalastyle:ignore
+        "unicode_ci",
+        Row("ÀÃÂ")), // scalastyle:ignore
+      SubstringTestCase(
+        "select right('ÀÃÂĀĂȦÄäåäáâãȻȻȻȻȻǢǼÆ' collate " + "utf8_binary_lcase" + ", 3)", // scalastyle:ignore
+        "utf8_binary_lcase",
+        Row("ǢǼÆ")), // scalastyle:ignore
+      SubstringTestCase(
+        "select substr('' collate " + "utf8_binary_lcase" + ", 1, 1)",
+        "utf8_binary_lcase",
+        Row("")),
+      SubstringTestCase(
+        "select substr('' collate " + "unicode" + ", 1, 1)",
+        "unicode",
+        Row("")),
+      SubstringTestCase(
+        "select left('' collate " + "utf8_binary" + ", 1)",
+        "utf8_binary",
+        Row("")),
+        // improper values
+      SubstringTestCase(
+        "select left(null collate " + "utf8_binary_lcase" + ", 1)",
+        "utf8_binary_lcase",
+        Row(null)),
+      SubstringTestCase(
+        "select right(null collate " + "unicode" + ", 1)",
+        "unicode",
+        Row(null)),
+      SubstringTestCase(
+        "select substr(null collate " + "utf8_binary" + ", 1)",
+        "utf8_binary",
+        Row(null)),
+      SubstringTestCase(
+        "select substr(null collate " + "unicode_ci" + ", 1, 1)",
+        "unicode_ci",
+        Row(null)),
+      SubstringTestCase(
+        "select left(null collate " + "utf8_binary_lcase" + ", null)",
+        "utf8_binary_lcase",
+        Row(null)),
+      SubstringTestCase(
+        "select right(null collate " + "unicode" + ", null)",
+        "unicode",
+        Row(null)),
+      SubstringTestCase(
+        "select substr(null collate " + "utf8_binary" + ", null)",
+        "utf8_binary",
+        Row(null)),
+      SubstringTestCase(
+        "select substr(null collate " + "unicode_ci" + ", null, null)",
+        "unicode_ci",
+        Row(null)),
+      SubstringTestCase(
+        "select left('ÀÃÂĀĂȦÄäåäáâãȻȻȻȻȻǢǼÆ' collate " + "utf8_binary_lcase" + ", null)", // scalastyle:ignore
+        "utf8_binary_lcase",
+        Row(null)),
+      SubstringTestCase(
+        "select right('ÀÃÂĀĂȦÄäåäáâãȻȻȻȻȻǢǼÆ' collate " + "unicode" + ", null)", // scalastyle:ignore
+        "unicode",
+        Row(null)),
+      SubstringTestCase(
+        "select substr('ÀÃÂĀĂȦÄäåäáâãȻȻȻȻȻǢǼÆ' collate " + "utf8_binary" + ", null)", // scalastyle:ignore
+        "utf8_binary",
+        Row(null)),
+      SubstringTestCase(
+        "select substr('' collate " + "unicode_ci" + ", null, null)",
+        "unicode_ci",
+        Row(null))


please follow the format of tests in this suite, queries should be constructed from these parameterized cases, otherwise they don't have a point

@uros-db I will change it accordingly. Please advise - Do you want three case classes, or one case class but with a parameter for function name? If the latter (one case class), how do you want me to handle the third parameter (len), which left and right do not have, and which is optional for substr? Maybe with an Option[String]?

Is the quantity of tests satisfactory? I got it down from 112 tests to 25 tests. Thus 13 for valid values and 12 for invalid values. I can get it down to 12 valid test cases and, say, six invalid values if you prefer.

It's fine if there's a parameter and it's not used in some cases, I don't think that causes any error

otherwise you can always introduce LeftRightTestCase, whatever works

quantity is fine here, what's important is proper coverage without needless repetition, these e2e sql tests are pretty slow, so having hundreds of them for trivial expressions is less than ideal

@uros-db Done.

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala

uros-db

no need to avoid scalastyle guides

GideonPotok · 2024-04-19T06:40:08Z

@uros-db I have made the suggested changes. please re-review.

format format format format fewer test cases now fewer test cases now fewer test cases now fewer test cases now fewer test cases now Fewer test cases and fewer tests for left/right/substr rename t1234 to t1234<index> rename QTestCase to SubstringTestCase remove redundant unit test unify test naming with struct test tests pass locally format test impl to make more in line with refactor. next add struct test, maybe. left impl right impl

GideonPotok · 2024-04-21T21:53:04Z

@uros-db

uros-db

lgtm

cloud-fan · 2024-04-22T07:16:56Z

thanks, merging to master!

https://issues.apache.org/jira/browse/SPARK-46830 ### What changes were proposed in this pull request? Add collation support to types of return values for calls to substr, left, right, when passed in arguments of an explicit, implicit, or session-specified collations. Add tests to validate behavior. ### Why are the changes needed? We are incrementally adding collation support to built-in string functions in Spark. These functions are intended to be supported for collated types. ### Does this PR introduce _any_ user-facing change? these sql functions will now not throw errors when passed in collated types. Instead, they will return the right value, of the passed in type. Or of the default collation. ### How was this patch tested? Unit testing + ad-hoc spark shell and pyspark shell interactions. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#46040 from GideonPotok/spark_collation_47413_5. Authored-by: GideonPotok <g.potok4@gmail.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>

github-actions bot added the SQL label Apr 13, 2024

GideonPotok mentioned this pull request Apr 14, 2024

[SPARK-47413][SQL] - add support to substr/left/right for collations #45738

Closed

GideonPotok marked this pull request as ready for review April 14, 2024 08:15

uros-db reviewed Apr 15, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

uros-db reviewed Apr 15, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

uros-db reviewed Apr 15, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

GideonPotok requested a review from uros-db April 15, 2024 08:53

uros-db reviewed Apr 15, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

uros-db reviewed Apr 15, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

uros-db reviewed Apr 15, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

uros-db reviewed Apr 15, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

uros-db reviewed Apr 15, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

GideonPotok force-pushed the spark_collation_47413_5 branch 2 times, most recently from 62e22c6 to f6efb32 Compare April 16, 2024 06:35

GideonPotok requested a review from uros-db April 16, 2024 06:35

GideonPotok force-pushed the spark_collation_47413_5 branch from f6efb32 to 1d91e95 Compare April 16, 2024 10:39

uros-db reviewed Apr 17, 2024

View reviewed changes

uros-db suggested changes Apr 17, 2024

View reviewed changes

GideonPotok requested a review from uros-db April 17, 2024 13:11

uros-db reviewed Apr 17, 2024

View reviewed changes

GideonPotok force-pushed the spark_collation_47413_5 branch from a4d3c1e to 95be248 Compare April 18, 2024 08:42

GideonPotok requested a review from uros-db April 18, 2024 09:07

GideonPotok changed the title ~~[SPARK-47413][SQL] - add support to substr/left/right for collations [Post-Refactor]~~ [SPARK-47413][SQL] - add support to substr/left/right for collations Apr 18, 2024

uros-db reviewed Apr 18, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

uros-db reviewed Apr 18, 2024

View reviewed changes

sql/core/src/test/scala/org/apache/spark/sql/CollationStringExpressionsSuite.scala Outdated Show resolved Hide resolved

uros-db suggested changes Apr 18, 2024

View reviewed changes

GideonPotok requested a review from uros-db April 19, 2024 06:39

GideonPotok force-pushed the spark_collation_47413_5 branch from 5050cf6 to b0357b7 Compare April 21, 2024 12:46

uros-db approved these changes Apr 22, 2024

View reviewed changes

cloud-fan closed this in e1432ef Apr 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-47413][SQL] - add support to substr/left/right for collations #46040

[SPARK-47413][SQL] - add support to substr/left/right for collations #46040

GideonPotok commented Apr 13, 2024

GideonPotok commented Apr 14, 2024

HyukjinKwon commented Apr 15, 2024

GideonPotok commented Apr 15, 2024

uros-db Apr 15, 2024

GideonPotok Apr 16, 2024

GideonPotok commented Apr 15, 2024

GideonPotok commented Apr 16, 2024

GideonPotok commented Apr 17, 2024

uros-db Apr 17, 2024

uros-db Apr 17, 2024

GideonPotok Apr 17, 2024 •

edited

Loading

uros-db left a comment

uros-db Apr 17, 2024

GideonPotok Apr 18, 2024

uros-db Apr 18, 2024

GideonPotok Apr 18, 2024

uros-db left a comment

GideonPotok commented Apr 19, 2024

GideonPotok commented Apr 21, 2024

uros-db left a comment

cloud-fan commented Apr 22, 2024

[SPARK-47413][SQL] - add support to substr/left/right for collations #46040

[SPARK-47413][SQL] - add support to substr/left/right for collations #46040

Conversation

GideonPotok commented Apr 13, 2024

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

GideonPotok commented Apr 14, 2024

HyukjinKwon commented Apr 15, 2024

GideonPotok commented Apr 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GideonPotok commented Apr 15, 2024

GideonPotok commented Apr 16, 2024

GideonPotok commented Apr 17, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GideonPotok Apr 17, 2024 • edited Loading

Choose a reason for hiding this comment

uros-db left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uros-db left a comment

Choose a reason for hiding this comment

GideonPotok commented Apr 19, 2024

GideonPotok commented Apr 21, 2024

uros-db left a comment

Choose a reason for hiding this comment

cloud-fan commented Apr 22, 2024

GideonPotok Apr 17, 2024 •

edited

Loading