[SPARK-53924] Reload DSv2 tables in views created using plans on each view access #52876

aokolnychyi · 2025-11-04T15:36:48Z

What changes were proposed in this pull request?

This PR makes Spark reload DSv2 tables in views created using plans on each view access.

Why are the changes needed?

The current problem is that the view definition in the session catalog captures the analyzed plan that references Table (that is supposed to pin the version). If a connector doesn’t have an internal cache and produces a new Table object on each load, the table referenced in the view will become orphan and there will be no way to refresh it unless that Table instance auto refreshes on each scan (super dangerous).

Does this PR introduce any user-facing change?

Yes, but it restores the correct behavior without requiring hacks in connectors.

How was this patch tested?

This PR comes with tests.

Was this patch authored or co-authored using generative AI tooling?

No.

cloud-fan · 2025-11-04T22:22:18Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableReference.scala

+  extends LeafNode with MultiInstanceRelation with NamedRelation {
+
+  override def name: String = {
+    s"${catalog.name()}.${identifier.quoted}"


let's follow DataSourceV2RelationBase or create a base trait/util function

override def name: String = { import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._ (catalog, identifier) match { case (Some(cat), Some(ident)) => s"${quoteIfNeeded(cat.name())}.${ident.quoted}" case _ => table.name() } }

cloud-fan · 2025-11-04T22:24:38Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableReference.scala

+
+object TableReference {
+
+  case class TableInfo(columns: Seq[Column])


shall we just use StructType?

I actually need Columns here to compare original metadata like types (e.g. char/varchar).

aokolnychyi · 2025-11-14T19:01:11Z

cc @dongjoon-hyun @cloud-fan @gengliangwang @szehon-ho @huaxingao

gengliangwang · 2025-11-14T21:10:37Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TableReference.scala

+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import org.apache.spark.util.ArrayImplicits._
+
+case class TableReference private (


let's add code comments

Also, since it is DSV2 only, shall we rename as DSV2TableReference?

gengliangwang · 2025-11-14T21:50:55Z

sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala

+  test("SPARK-53924: insert into DSv2 table invalidates cache of SQL temp views with plans") {
+    val t = "testcat.tbl"
+    withTable(t, "v") {
+      withSQLConf(SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW.key -> "true") {


Let's also test when the config is false

gengliangwang · 2025-11-14T21:50:59Z

sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala

+  test("SPARK-53924: uncache DSv2 table using SQL uncaches SQL temp views with plans") {
+    val t = "testcat.tbl"
+    withTable(t, "v") {
+      withSQLConf(SQLConf.STORE_ANALYZED_PLAN_FOR_VIEW.key -> "true") {


Let's also test when the config is false

gengliangwang · 2025-11-14T22:14:37Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala

+  private def adaptCachedRelation(cached: LogicalPlan, ref: TableReference): LogicalPlan = {
+    cached transform {
+      case r: DataSourceV2Relation if matchesReference(r, ref) =>
+        r.copy(output = ref.output, options = ref.options)


Can you explain why this is needed? When resolveReference is call, all the cached relation will be updated.

Not sure I got the question. This code exists as the cached output attributes may be different from the reference. This method replaces TableReference (which is resolved and has its output) with a relation from the cache but makes it use output and options from TableReference.

Does it make sense?

QQ: do we need to call TableReferenceUtils.validateLoadedTable if it is cached?

Actually, that's a good call. Let me add it.

gengliangwang · 2025-11-15T00:26:58Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/V2TableReference.scala

+ * For instance, temporary views with fully resolved logical plans don't allow schema changes
+ * in underlying tables.
+ */
+case class V2TableReference private(


Let's add private[sql] to make it internal

I thought everything in catalyst is considered internal, but let me update.

gengliangwang · 2025-11-15T00:27:02Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/V2TableReference.scala

+  }
+}
+
+object V2TableReference {


Let's add private[sql] to make it internal

gengliangwang · 2025-11-15T00:27:09Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/V2TableReference.scala

+  }
+}
+
+object V2TableReferenceUtils extends SQLConfHelper {


Let's add private[sql] to make it internal

dongjoon-hyun · 2025-11-15T03:59:19Z

common/utils/src/main/resources/error/error-conditions.json

  },
+  "INCOMPATIBLE_COLUMN_CHANGES_AFTER_VIEW_WITH_PLAN_CREATION" : {
+    "message" : [
+      "View <viewName> plan references table <tableName> whose <colType> columns changed since the view plan was initially captured.",


nit. Maybe, <colType> columns -> <colType> column?

The rest of the error message says "column changes" in plural, so I think using columns here should be OK.

… view access

aokolnychyi · 2025-11-16T02:33:55Z

@dongjoon-hyun @gengliangwang, could you folks help me merge this into master/4.1? All tests are passing.

dongjoon-hyun

+1, LGTM again. Thank you, @aokolnychyi .

… view access ### What changes were proposed in this pull request? This PR makes Spark reload DSv2 tables in views created using plans on each view access. ### Why are the changes needed? The current problem is that the view definition in the session catalog captures the analyzed plan that references `Table` (that is supposed to pin the version). If a connector doesn’t have an internal cache and produces a new `Table` object on each load, the table referenced in the view will become orphan and there will be no way to refresh it unless that `Table` instance auto refreshes on each scan (super dangerous). ### Does this PR introduce _any_ user-facing change? Yes, but it restores the correct behavior without requiring hacks in connectors. ### How was this patch tested? This PR comes with tests. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52876 from aokolnychyi/spark-53924. Authored-by: Anton Okolnychyi <aokolnychyi@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 407e79c) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2025-11-16T06:00:29Z

Merged to master/4.1 for Apache Spark 4.1.0.

aokolnychyi · 2025-11-18T05:41:56Z

Thanks, @dongjoon-hyun @gengliangwang!

…ng schema changes ### What changes were proposed in this pull request? Follow-up of #52876, add tests for cached temp view detecting schema changes ### Why are the changes needed? There is no test coverage after comment #52876 (comment) is addressed. This PR is to add a test case for it. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? New test case ### Was this patch authored or co-authored using generative AI tooling? No Closes #53103 from gengliangwang/SPARK-53924-test. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org>

…ng schema changes ### What changes were proposed in this pull request? Follow-up of #52876, add tests for cached temp view detecting schema changes ### Why are the changes needed? There is no test coverage after comment #52876 (comment) is addressed. This PR is to add a test case for it. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? New test case ### Was this patch authored or co-authored using generative AI tooling? No Closes #53103 from gengliangwang/SPARK-53924-test. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org> (cherry picked from commit fd683ce) Signed-off-by: Gengliang Wang <gengliang@apache.org>

manuzhang · 2025-11-20T16:21:17Z

@aokolnychyi I'm testing Spark 4.1 support for Iceberg in apache/iceberg#14155. It looks this PR has broken this Iceberg test with following error. Can you help check?

Exception in thread "test-extra-commit-message-writer-thread" java.lang.RuntimeException: org.apache.spark.SparkException: [INTERNAL_ERROR] Found the unresolved operator: 'InsertIntoStatement TableReference[id#5, data#6] default_iceberg.`/var/folders/pv/9kgp4f8j685fqb28n83cdb800000gq/T/junit-16494569145098127962`.`iceberg-table`, false, false, false SQLSTATE: XX000
== SQL (line 1, position 1) ==
INSERT INTO target VALUES (3, 'c'), (4, 'd')
^^^^^^^^^^^^^^^^^^

aokolnychyi · 2025-11-20T19:38:16Z

@manuzhang, ack, let me check.

### What changes were proposed in this pull request? Resolve `V2TableReference` for table in `InsertIntoStatement`. ### Why are the changes needed? #52876 brought in `V2TableReference` which broke relation resolution for insert into temp view on DSv2 table. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Add UT. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53196 from manuzhang/FIX-SPARK-54491. Authored-by: manuzhang <owenzhang1990@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

### What changes were proposed in this pull request? Resolve `V2TableReference` for table in `InsertIntoStatement`. ### Why are the changes needed? #52876 brought in `V2TableReference` which broke relation resolution for insert into temp view on DSv2 table. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Add UT. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #53196 from manuzhang/FIX-SPARK-54491. Authored-by: manuzhang <owenzhang1990@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 3f5a2b9) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

… view access ### What changes were proposed in this pull request? This PR makes Spark reload DSv2 tables in views created using plans on each view access. ### Why are the changes needed? The current problem is that the view definition in the session catalog captures the analyzed plan that references `Table` (that is supposed to pin the version). If a connector doesn’t have an internal cache and produces a new `Table` object on each load, the table referenced in the view will become orphan and there will be no way to refresh it unless that `Table` instance auto refreshes on each scan (super dangerous). ### Does this PR introduce _any_ user-facing change? Yes, but it restores the correct behavior without requiring hacks in connectors. ### How was this patch tested? This PR comes with tests. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52876 from aokolnychyi/spark-53924. Authored-by: Anton Okolnychyi <aokolnychyi@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…ng schema changes ### What changes were proposed in this pull request? Follow-up of apache#52876, add tests for cached temp view detecting schema changes ### Why are the changes needed? There is no test coverage after comment apache#52876 (comment) is addressed. This PR is to add a test case for it. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? New test case ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#53103 from gengliangwang/SPARK-53924-test. Authored-by: Gengliang Wang <gengliang@apache.org> Signed-off-by: Gengliang Wang <gengliang@apache.org>

### What changes were proposed in this pull request? Resolve `V2TableReference` for table in `InsertIntoStatement`. ### Why are the changes needed? apache#52876 brought in `V2TableReference` which broke relation resolution for insert into temp view on DSv2 table. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Add UT. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#53196 from manuzhang/FIX-SPARK-54491. Authored-by: manuzhang <owenzhang1990@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

github-actions bot added the SQL label Nov 4, 2025

aokolnychyi force-pushed the spark-53924 branch from f002dd1 to ccd330c Compare November 4, 2025 22:03

cloud-fan reviewed Nov 4, 2025

View reviewed changes

aokolnychyi force-pushed the spark-53924 branch from ccd330c to ac9e49e Compare November 14, 2025 18:14

aokolnychyi changed the title ~~[WIP][SPARK-53924] Reload DSv2 tables in views created using plans on each view access~~ [SPARK-53924] Reload DSv2 tables in views created using plans on each view access Nov 14, 2025

aokolnychyi force-pushed the spark-53924 branch 3 times, most recently from 706ade0 to a9251b6 Compare November 14, 2025 18:56

gengliangwang reviewed Nov 14, 2025

View reviewed changes

gengliangwang approved these changes Nov 15, 2025

View reviewed changes

gengliangwang reviewed Nov 15, 2025

View reviewed changes

dongjoon-hyun reviewed Nov 15, 2025

View reviewed changes

dongjoon-hyun approved these changes Nov 15, 2025

View reviewed changes

gengliangwang approved these changes Nov 15, 2025

View reviewed changes

aokolnychyi added 4 commits November 15, 2025 09:36

[SPARK-53924] Reload DSv2 tables in views created using plans on each…

f4badf4

… view access

Review feedback

74feb52

Review feedback

3bbdfc7

Fix write to view on top of DSv2

e8a9d1a

aokolnychyi force-pushed the spark-53924 branch from f5c8a68 to e8a9d1a Compare November 15, 2025 17:59

dongjoon-hyun approved these changes Nov 16, 2025

View reviewed changes

dongjoon-hyun closed this in 407e79c Nov 16, 2025

gengliangwang mentioned this pull request Nov 17, 2025

[SPARK-53924][FOLLOWUP][TESTS] Add tests for cached temp view detecting schema changes #53103

Closed

manuzhang mentioned this pull request Nov 24, 2025

[SPARK-54491][SQL] Fix insert into temp view on DSv2 table failure #53196

Closed


		object TableReference {

		case class TableInfo(columns: Seq[Column])

[SPARK-53924] Reload DSv2 tables in views created using plans on each view access #52876

[SPARK-53924] Reload DSv2 tables in views created using plans on each view access #52876

Uh oh!

Conversation

aokolnychyi commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

cloud-fan Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aokolnychyi commented Nov 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aokolnychyi commented Nov 16, 2025

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 16, 2025

Uh oh!

aokolnychyi commented Nov 18, 2025

Uh oh!

manuzhang commented Nov 20, 2025

Uh oh!

aokolnychyi commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aokolnychyi commented Nov 4, 2025 •

edited

Loading

cloud-fan Nov 4, 2025 •

edited

Loading