Added Support for Lazy Quantifier [databricks]

[razajafri]

This PR adds the lazy quantifier to the RegexParser

Changes Made:

• RegexParser returns the needed transpiled AST needed
• Test was added to regexp_test.py

Tests:

• Integration test added to regexp_test.py

Integration tests were run locally

fixes #8806

[razajafri]

I will be adding more fuzzer tests

[razajafri]

build

[razajafri]

Databricks cluster failed to start.

[2024-01-17T21:16:34.414Z] subprocess.CalledProcessError: Command 'ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p 2200 -i **** ubuntu@4.155.42.241 'SPARKSRCTGZ=/home/ubuntu/spark-rapids-ci.tgz BASE_SPARK_VERSION=3.2.1 BASE_SPARK_VERSION_TO_INSTALL_DATABRICKS_JARS=3.2.1 MVN_OPT= EXTRA_ENVS= bash /home/ubuntu/build.sh 2>&1 | tee buildout; if [ echo ${PIPESTATUS[0]} -ne 0 ]; then false; else true; fi'' returned non-zero exit status 1.

[razajafri]

build

[andygrove]

LGTM. I ran this locally and manually verified that the results from the integration test looked correct. I also tried some different datagen patterns to see if I could break the test but did not find anything.

[razajafri]

build

[razajafri]

If Databricks keeps failing I will just run the test manually and skip Databricks from the premerge

[razajafri]

build

[razajafri]

build

[sameerz]

build

[NVnavkumar]

Looking at recent blossom failure:

2024-01-22T01:00:54.1200895Z [2024-01-22T00:55:30.814Z] �[31m- AST fuzz test - regexp_find *** FAILED ***�[0m
2024-01-22T01:00:54.1201603Z [2024-01-22T00:55:30.815Z] �[31m  cuDF failed to compile pattern: (2|\u2029??)+? (RegularExpressionTranspilerSuite.scala:954)

It looks like cuDF doesn't support the ?? or ? inside of the choice( |). Will need to handle that case explicitly and fall back to CPU. The +? after the group is fine.

[razajafri]

Looking at recent blossom failure:

2024-01-22T01:00:54.1200895Z [2024-01-22T00:55:30.814Z] �[31m- AST fuzz test - regexp_find *** FAILED ***�[0m
2024-01-22T01:00:54.1201603Z [2024-01-22T00:55:30.815Z] �[31m  cuDF failed to compile pattern: (2|\u2029??)+? (RegularExpressionTranspilerSuite.scala:954)

It looks like cuDF doesn't support the ?? or ? inside of the choice( |). Will need to handle that case explicitly and fall back to CPU. The +? after the group is fine.

Thanks for looking into this. I am struggling to see why the pattern is being generated

[NVnavkumar]

Looking at recent blossom failure:

2024-01-22T01:00:54.1200895Z [2024-01-22T00:55:30.814Z] �[31m- AST fuzz test - regexp_find *** FAILED ***�[0m
2024-01-22T01:00:54.1201603Z [2024-01-22T00:55:30.815Z] �[31m  cuDF failed to compile pattern: (2|\u2029??)+? (RegularExpressionTranspilerSuite.scala:954)

It looks like cuDF doesn't support the ?? or ? inside of the choice( |). Will need to handle that case explicitly and fall back to CPU. The +? after the group is fine.

Thanks for looking into this. I am struggling to see why the pattern is being generated

It's because it's actually a valid pattern in Java (and Python fwiw). CUDF is the only thing throwing an invalid pattern here. Since it's valid in Java, and we have not detected to fallback to CPU, that's why you are getting the test error here.

[andygrove]

I was able to get past the failing test with this diff. Note that this is only checking the right side of the choice and we would need to look at the left as well.

$ git diff
diff --git a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RegexParser.scala b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RegexParser.scala
index 2380eb623..7666f1ab9 100644
--- a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RegexParser.scala
+++ b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RegexParser.scala
@@ -1565,6 +1565,14 @@ class CudfRegexTranspiler(mode: RegexMode) {
             r.position)
         }
 
+        rr match {
+          case RegexSequence(ListBuffer(RegexRepetition(
+              RegexRepetition(_,SimpleQuantifier('?')),SimpleQuantifier('?')))) =>
+            throw new RegexUnsupportedException(
+              "cuDF does not support lazy quantifier inside choice", r.position)
+          case other =>
+        }
+
         RegexChoice(ll, rr)

[razajafri]

build

[razajafri]

Thanks @andygrove

[razajafri]

There are more test failures, I am looking into them

[sameerz]

If we are creating exceptions, we need to document these in https://github.com/NVIDIA/spark-rapids/blob/branch-24.02/docs/compatibility.md#regular-expressions

[razajafri]

build

[andygrove]

nit: indent seems off here? I would expect this second line to be indented more that the case in the prev line. Same comment for other uses of this pattern in this PR.

[andygrove]

LGTM. Lett a comment about formatting.