Skip to content

Conversation

@Hugo-WB
Copy link
Contributor

@Hugo-WB Hugo-WB commented Jan 12, 2026

When having the IcebergSparkSessionExtension on your spark session and trying to UPDATE SET ... WHERE ... a Delta table. We run into the following error:

Error here
DeltaTableV2(org.apache.spark.sql.SparkSession@5b6ddd5d,file:/var/folders/h9/_tv85syj43d9lr1lgbxqm6v508fh8r/T/bc027d17_7885_4b16_a7cf_110ae549cb62,Some(CatalogTable(
Catalog: spark_catalog
Database: default
Table: bc027d17_7885_4b16_a7cf_110ae549cb62
Created Time: Mon Jan 12 10:07:19 GMT 2026
Last Access: UNKNOWN
Created By: Spark 
Type: MANAGED
Provider: delta
Table Properties: [delta.enableChangeDataFeed=true, delta.feature.appendOnly=supported, delta.feature.changeDataFeed=supported, delta.feature.invariants=supported, delta.lastCommitTimestamp=1768212440690, delta.lastUpdateVersion=2, delta.minReaderVersion=1, delta.minWriterVersion=7]
Location: file:/var/folders/h9/_tv85syj43d9lr1lgbxqm6v508fh8r/T/bc027d17_7885_4b16_a7cf_110ae549cb62
Schema: root
 |-- a: string (nullable = true)
 |-- b: long (nullable = true)
 |-- c: string (nullable = true)
)),Some(default.bc027d17_7885_4b16_a7cf_110ae549cb62),None,Map()) (of class org.apache.spark.sql.delta.catalog.DeltaTableV2)
scala.MatchError: DeltaTableV2(org.apache.spark.sql.SparkSession@5b6ddd5d,file:/var/folders/h9/_tv85syj43d9lr1lgbxqm6v508fh8r/T/bc027d17_7885_4b16_a7cf_110ae549cb62,Some(CatalogTable(
Catalog: spark_catalog
Database: default
Table: bc027d17_7885_4b16_a7cf_110ae549cb62
Created Time: Mon Jan 12 10:07:19 GMT 2026
Last Access: UNKNOWN
Created By: Spark 
Type: MANAGED
Provider: delta
Table Properties: [delta.enableChangeDataFeed=true, delta.feature.appendOnly=supported, delta.feature.changeDataFeed=supported, delta.feature.invariants=supported, delta.lastCommitTimestamp=1768212440690, delta.lastUpdateVersion=2, delta.minReaderVersion=1, delta.minWriterVersion=7]
Location: file:/var/folders/h9/_tv85syj43d9lr1lgbxqm6v508fh8r/T/bc027d17_7885_4b16_a7cf_110ae549cb62
Schema: root
 |-- a: string (nullable = true)
 |-- b: long (nullable = true)
 |-- c: string (nullable = true)
)),Some(default.bc027d17_7885_4b16_a7cf_110ae549cb62),None,Map()) (of class org.apache.spark.sql.delta.catalog.DeltaTableV2)
	at org.apache.spark.sql.catalyst.analysis.RewriteOperationForRowLineage.shouldUpdatePlan(RewriteOperationForRowLineage.scala:42)
	at org.apache.spark.sql.catalyst.analysis.RewriteOperationForRowLineage.shouldUpdatePlan$(RewriteOperationForRowLineage.scala:39)
	at org.apache.spark.sql.catalyst.analysis.RewriteUpdateTableForRowLineage$.shouldUpdatePlan(RewriteUpdateTableForRowLineage.scala:29)
	at org.apache.spark.sql.catalyst.analysis.RewriteUpdateTableForRowLineage$$anonfun$apply$1.applyOrElse(RewriteUpdateTableForRowLineage.scala:33)
	at org.apache.spark.sql.catalyst.analysis.RewriteUpdateTableForRowLineage$$anonfun$apply$1.applyOrElse(RewriteUpdateTableForRowLineage.scala:32)
	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsDownWithPruning$2(AnalysisHelper.scala:170)

Note: This spark session has both the iceberg and delta spark session extensions configured.

This PR fixes the above by returning false if the updated table does not match SparkTable. Which DeltaTableV2 does not.

@github-actions github-actions bot added the spark label Jan 12, 2026
@Hugo-WB Hugo-WB force-pushed the fix/rewrite-operation-for-file-lineage branch 2 times, most recently from a8778be to 95f81a8 Compare January 12, 2026 16:50
@Hugo-WB Hugo-WB force-pushed the fix/rewrite-operation-for-file-lineage branch from 95f81a8 to 5f938cd Compare January 12, 2026 16:51
Comment on lines 43 to +45
case sparkTable: SparkTable =>
TableUtil.supportsRowLineage(sparkTable.table())
case _ => false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this will fail with a match error if not have default false ? can you add a test for it

case r: DataSourceV2Relation =>
r.table match {
case sparkTable: SparkTable =>
TableUtil.supportsRowLineage(sparkTable.table())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with Prashant on this and as a nit you could also be simplified by flattening the nested match

case DataSourceV2Relation(sparkTable: SparkTable, _, _, _, _) =>
  TableUtil.supportsRowLineage(sparkTable.table())
case _ => false

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants