[SPARK-37643][SQL] when charVarcharAsString is true, char datatype predicate query incorrect #34900
Conversation
|
Can one of the admins verify this patch? |
|
anyone help to review and merge?thank you. |
|
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. |
|
|
also cc @yaooqinn @cloud-fan |
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
Outdated
Show resolved
Hide resolved
fhygh
changed the title
| sql(s"CREATE TABLE t(a STRING, c CHAR(5)) USING $format partitioned by (c)") | ||
| sql("INSERT INTO t VALUES ('abc', 'abc')") | ||
| checkAnswer(sql("SELECT c FROM t WHERE c='abc'"), Row("abc")) | ||
| checkAnswer(sql("SELECT c FROM t WHERE c in ('abc')"), Row("abc")) |
There was a problem hiding this comment.
let's test both data and partition predicates.
…edicate table query incorrect
| @@ -4082,18 +4082,21 @@ object ApplyCharTypePadding extends Rule[LogicalPlan] { | |||
|
|
|||
There was a problem hiding this comment.
shall we simply skip this rule if the conf is true? e.g. in the apply method, we return the input plan if SQLConf.get.charVarcharAsString is true.
|
The build link https://github.com/fhygh/spark/actions/runs/2158841362 |
What changes were proposed in this pull request?
after add ApplyCharTypePadding rule, when partition data type is char, if partition value length is less then defined, partition expr filter will be right-padding, then will query incorrect result
Why are the changes needed?
fix query incorrect issue when filter is partition type and partition type is char.
Does this PR introduce any user-facing change?
before this fix, if we using char partition type, then we should be careful to set charVarcharAsString to true.
How was this patch tested?
add new UT.