Skip to content

Commit

Permalink
Enable SQL support for WHEN NOT MATCHED BY SOURCE
Browse files Browse the repository at this point in the history
# Description
The SQL syntax for merge with WHEN NOT MATCHED BY SOURCE clauses was shipped with Spark 3.4. Now that Delta picked up Spark 3.4, we can enable SQL support and mix in SQL tests for WHEN NOT MATCHED BY SOURCE.

Existing tests for WHEN NOT MATCHED BY SOURCE are now run in the Merge SQL suite.

Closes #1740

GitOrigin-RevId: 1ddd1216e13f854901da47896936527618ea4dca
  • Loading branch information
johanl-db authored and allisonport-db committed May 10, 2023
1 parent c180d15 commit c53e95c
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -459,6 +459,18 @@ class DeltaAnalysis(session: SparkSession)
s"${other.prettyName} clauses cannot be part of the WHEN NOT MATCHED clause in MERGE " +
"INTO.")
}
val notMatchedBySourceActions = merge.notMatchedBySourceActions.map {
case update: UpdateAction =>
DeltaMergeIntoNotMatchedBySourceUpdateClause(
update.condition,
DeltaMergeIntoClause.toActions(update.assignments))
case delete: DeleteAction =>
DeltaMergeIntoNotMatchedBySourceDeleteClause(delete.condition)
case other =>
throw new IllegalArgumentException(
s"${other.prettyName} clauses cannot be part of the WHEN NOT MATCHED BY SOURCE " +
"clause in MERGE INTO.")
}
// rewrites Delta from V2 to V1
val newTarget =
stripTempViewForMergeWrapper(merge.targetTable).transformUp { case DeltaRelation(lr) => lr }
Expand All @@ -468,7 +480,7 @@ class DeltaAnalysis(session: SparkSession)
newTarget,
merge.sourceTable,
merge.mergeCondition,
matchedActions ++ notMatchedActions
matchedActions ++ notMatchedActions ++ notMatchedBySourceActions
)

DeltaMergeInto.resolveReferencesAndSchema(deltaMerge, conf)(tryResolveReferences(session))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{IntegerType, StructField, StructType}

class MergeIntoSQLSuite extends MergeIntoSuiteBase with DeltaSQLCommandTest
class MergeIntoSQLSuite extends MergeIntoSuiteBase with MergeIntoNotMatchedBySourceSuite
with DeltaSQLCommandTest
with DeltaTestUtilsForTempViews {

import testImplicits._
Expand Down

0 comments on commit c53e95c

Please sign in to comment.