Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-36706][SQL][3.1] OverwriteByExpression conversion in DataSourc…
…eV2Strategy use wrong param in translateFilter ### What changes were proposed in this pull request? The wrong parameter is used in `translateFilter` in the following code ``` val filters = splitConjunctivePredicates(deleteExpr).map { filter => DataSourceStrategy.translateFilter(deleteExpr, supportNestedPredicatePushdown = true).getOrElse( throw new AnalysisException(s"Cannot translate expression to source filter: $filter")) }.toArray ``` Using this as an example ``` spark.table("source2_t").writeTo("testcat.table_name").overwrite($"id1" === 3 && $"id2" === 3) ``` The above code will generate these filters: ``` And(EqualTo(id1, 3),EqualTo(id2, 3)) And(EqualTo(id1, 3),EqualTo(id2, 3)) ``` we want to fix the code so it will generate the filters like these: ``` EqualTo(id1, 3) EqualTo(id2, 3) ``` This problem only exists in 3.1. In 3.2 and 3.3, we have ``` val filters = splitConjunctivePredicates(deleteExpr).flatMap { pred => val filter = DataSourceStrategy.translateFilter(pred, supportNestedPredicatePushdown = true) if (filter.isEmpty) { throw QueryCompilationErrors.cannotTranslateExpressionToSourceFilterError(pred) } filter }.toArray ``` ### Why are the changes needed? fix a bug in the code ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? existing tests Closes apache#33997 from huaxingao/spark-36706. Authored-by: Huaxin Gao <huaxin_gao@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
- Loading branch information