[SPARK-33938][SQL] Optimize Like Any/All by LikeSimplification

[beliefer]

What changes were proposed in this pull request?

We should optimize Like Any/All by LikeSimplification to improve performance.

Why are the changes needed?

Optimize Like Any/All

Does this PR introduce any user-facing change?

'No'.

How was this patch tested?

Jenkins test.

[SparkQA]

Test build #133635 has finished for PR 30975 at commit fec4fe5.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class SequenceFileRDDFunctions[K: IsWritable: ClassTag, V: IsWritable: ClassTag](
• case class ResolvedTable(
• case class StringTrim(srcStr: Expression, trimStr: Option[Expression] = None)
• case class StringTrimLeft(srcStr: Expression, trimStr: Option[Expression] = None)
• case class StringTrimRight(srcStr: Expression, trimStr: Option[Expression] = None)
• case class DescribeColumnExec(

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38208/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38208/

[SparkQA]

Test build #133619 has finished for PR 30975 at commit b6ba902.

• This patch passes all tests.
• This patch does not merge cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38226/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38226/

[cloud-fan]

we can remove it now.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38243/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38243/

[SparkQA]

Test build #133641 has finished for PR 30975 at commit 52ecedd.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #133669 has finished for PR 30975 at commit e2c7dfe.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #133654 has finished for PR 30975 at commit 454a68b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• sealed trait LikeAllBase extends MultiLikeBase
• sealed trait LikeAnyBase extends MultiLikeBase

[SparkQA]

Test build #133665 has finished for PR 30975 at commit 8f26d1b.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• sealed abstract class LikeAllBase extends MultiLikeBase
• sealed abstract class LikeAnyBase extends MultiLikeBase

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38270/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38270/

[SparkQA]

Test build #133681 has finished for PR 30975 at commit 6acf1c1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

The optimization was already there before we add the LikeAll expression in #29999 , so this PR fixes a perf regression in 3.1. Thanks, merging to master/3.1!

[cloud-fan]

It has conflicts with 3.1, @beliefer can you create a backport PR?

[beliefer]

It has conflicts with 3.1, @beliefer can you create a backport PR?

OK.

[beliefer]

@cloud-fan Thanks for your work!

[wangyum]

It may cause StackOverflowError.

scala> spark.sql("drop table SPARK_33938")
res6: org.apache.spark.sql.DataFrame = []

scala> spark.sql("create table SPARK_33938(id string) using parquet")
res7: org.apache.spark.sql.DataFrame = []

scala> val values = Range(1, 10000)
values: scala.collection.immutable.Range = Range 1 until 10000

scala> spark.sql(s"select * from SPARK_33938 where id like all (${values.map(s => s"'$s'").mkString(", ")})").show
java.lang.StackOverflowError
  at java.lang.ThreadLocal.set(ThreadLocal.java:201)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.set(TreeNode.scala:62)
  at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:72)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:317)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:322)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:407)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:243)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:405)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:358)
  at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:322)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDown$3(TreeNode.scala:322)
  at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$mapChildren$1(TreeNode.scala:407)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapProductIterator(TreeNode.scala:243)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:405)
  at org.apache.spark.sql.catalyst.trees.TreeNode.mapChildren(TreeNode.scala:358)

[beliefer]

@wangyum I will fix this issue.

[beliefer]

For example, patterns a, b, c, d, e, and f. Suppose a, b, c, and d are patterns that can be optimized with startsWith. According to the current logic, it is startsWith(a)&startsWith(b)&startsWith(c)&startsWith(d)&LikeAll(e,f). Their hierarchy is not shown here.
We can use the threshold to determine the number of patterns that can be optimized, for example, only two patterns can be optimized. Then it is startsWith(a)&startsWith(b)&LikeAll(c,d,e,f)