[SPARK-32196][SQL] Extract In convertible part if it is not convertible

[ulysses-you]

What changes were proposed in this pull request?

Modify OptimizeIn, extract In convertible part if it is not convertible.
And a new config spark.sql.optimizer.inExtractLiteralPart to control if we should extract the literal part of In.

Why are the changes needed?

Try to optimize more predicate.
First split In to 2 parts, one is convertible the other is not convertible. Then we can optimize the convertible part.

A table create table t1 (c1 int, c2 int) using parquet

select * from t1 where c1 in (1, 2, c2)
=>
select * from t1 where c1 in (1, 2) or c1 in (c2)

select * from t1 where c1 in(1, c2)
=>
select * from t1 where c1 = 1 or c1 in(c2)

select * from t1 where c1 in(1, 2, ..., c2)
=> 
select * from t1 where c1 inset(1, 2, ...) or c1 in (c2)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Add ut.

[SparkQA]

Test build #125092 has finished for PR 29013 at commit bbf63c9.

• This patch fails to generate documentation.
• This patch merges cleanly.
• This patch adds no public classes.

[ulysses-you]

Remove this test since we support convert part of list and new test include this.

[maropu]

I don't look into the impl. though (I just read the PR description), you meant this case? Or, do I miss something?

scala> sql("select * from t1").show()
+---+---+
| c1| c2|
+---+---+
|  1|  3|
|  3|  3|
+---+---+


scala> sql("select * from t1 where c1 in (1, 2, c2)").show()
+---+---+
| c1| c2|
+---+---+
|  1|  3|
|  3|  3|
+---+---+


scala> sql("select * from (select * from t1 where c1 in (1, 2)) t2(c1, c2) where t2.c1 in (1, 2, t2.c2)").show()
+---+---+
| c1| c2|
+---+---+
|  1|  3|
+---+---+

[ulysses-you]

ah morning @maropu

As your case after this pr, it changes to:

select * from t1 where c1 in (1, 2, c2)
 ||
 \/
select * from t1 where c1 in (1, 2) and c1 in (c2)

[maropu]

But, I think select * from t1 where c1 in (1, 2, c2) is equal to select * from t1 where c1=1 or c1=2 or c1=c2. Why can we transform it into where c1 in (1, 2) and c1 in (c2)?

[ulysses-you]

I make a mistake, it should be

select * from t1 where c1 in (1, 2, c2)
 ||
 \/
select * from t1 where c1 in (1, 2) or c1 in (c2)

[SparkQA]

Test build #125158 has finished for PR 29013 at commit f45d6c3.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #125148 has finished for PR 29013 at commit 0d982d2.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #125299 has finished for PR 29013 at commit 9bf23cc.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

Why is where c1 in (1, 2) or c1 in (c2) an optimized form? You mean it is fater than where c1 in (1, 2, c2)? At least, I think you need to describe more in the PR description.

[ulysses-you]

@maropu Once we have where c1 in (1, 2) or c1 in (c2), we can optimize the c1 in (1, 2) part with OptimizeIn.
e.g.

c1 in(1, c2) => c1 = 1 or c1 in(c2)

c1 in(1, 2, ..., c2) => c1 inset(1, 2, ...) or c1 in (c2)

[SparkQA]

Test build #125700 has finished for PR 29013 at commit 21c5262.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!