Skip to content

Commit

Permalink
[SPARK-36706][SQL][3.1] OverwriteByExpression conversion in DataSourc…
Browse files Browse the repository at this point in the history
…eV2Strategy use wrong param in translateFilter

### What changes were proposed in this pull request?
The wrong parameter is used in `translateFilter` in the following code
```
      val filters = splitConjunctivePredicates(deleteExpr).map {
        filter => DataSourceStrategy.translateFilter(deleteExpr,
          supportNestedPredicatePushdown = true).getOrElse(
            throw new AnalysisException(s"Cannot translate expression to source filter: $filter"))
      }.toArray
```

Using this as an example
```
spark.table("source2_t").writeTo("testcat.table_name").overwrite($"id1" === 3 && $"id2" === 3)
```

The above code will generate these filters:
```
And(EqualTo(id1, 3),EqualTo(id2, 3))
And(EqualTo(id1, 3),EqualTo(id2, 3))
```

 we want to fix the code so it will generate the filters like these:
```
EqualTo(id1, 3)
EqualTo(id2, 3)
```

This problem only exists in 3.1. In 3.2 and 3.3, we have

```
      val filters = splitConjunctivePredicates(deleteExpr).flatMap { pred =>
        val filter = DataSourceStrategy.translateFilter(pred, supportNestedPredicatePushdown = true)
        if (filter.isEmpty) {
          throw QueryCompilationErrors.cannotTranslateExpressionToSourceFilterError(pred)
        }
        filter
      }.toArray
```

### Why are the changes needed?
fix a bug in the code

### Does this PR introduce _any_ user-facing change?
no

### How was this patch tested?
existing tests

Closes apache#33997 from huaxingao/spark-36706.

Authored-by: Huaxin Gao <huaxin_gao@apple.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
huaxingao authored and fishcus committed Jan 12, 2022
1 parent 43b00a5 commit 6129ca4
Showing 1 changed file with 1 addition and 1 deletion.
Expand Up @@ -213,7 +213,7 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
case OverwriteByExpression(r: DataSourceV2Relation, deleteExpr, query, writeOptions, _) =>
// fail if any filter cannot be converted. correctness depends on removing all matching data.
val filters = splitConjunctivePredicates(deleteExpr).map {
filter => DataSourceStrategy.translateFilter(deleteExpr,
filter => DataSourceStrategy.translateFilter(filter,
supportNestedPredicatePushdown = true).getOrElse(
throw new AnalysisException(s"Cannot translate expression to source filter: $filter"))
}.toArray
Expand Down

0 comments on commit 6129ca4

Please sign in to comment.