Skip to content

[SPARK-56643][SQL][TESTS] Add DSv2 temp view with stored plan tests#55540

Open
longvu-db wants to merge 4 commits intoapache:masterfrom
longvu-db:dsv2-classic-pr1-temp-views
Open

[SPARK-56643][SQL][TESTS] Add DSv2 temp view with stored plan tests#55540
longvu-db wants to merge 4 commits intoapache:masterfrom
longvu-db:dsv2-classic-pr1-temp-views

Conversation

@longvu-db
Copy link
Copy Markdown
Contributor

@longvu-db longvu-db commented Apr 24, 2026 •
edited
Loading

What changes were proposed in this pull request?

Add 11 new tests to DataSourceV2DataFrameSuite that verify temporary view behavior with stored plans when the underlying DSv2 table changes. The tests cover all 7 scenarios from the DSv2 table refresh design doc:

  1. Session and external writes (scenarios 1.1, 1.2): Temp view with filter reflects new data after session or external writes.
  2. Adding columns with data (scenarios 2.1, 2.2): Temp view preserves original schema after ADD COLUMN, both session and external.
  3. Removing columns (scenario 3.2): Temp view detects external column removal via catalog API.
  4. Drop and recreate table (scenario 4.2): Temp view resolves to externally recreated table.
  5. Drop and re-add column with same type (scenarios 5.1, 5.2): Schema validation passes, view continues working.
  6. Drop and re-add column with different type (scenarios 6.1, 6.2): Temp view detects type change.
  7. Type widening (scenario 7): Temp view detects INT to BIGINT type change.

Each test creates a DSv2 table, inserts initial data, builds a temp view with a filter (salary < 999) to demonstrate stored plan behavior, and verifies the view after table modifications.

External writes use the direct catalog API (loadTable + withData) to simulate writes from outside the session.

Why are the changes needed?

The existing SPARK-53924 tests in DataSourceV2DataFrameSuite cover basic schema change detection for temp views on DSv2 tables but are missing:

  • Filter-based stored plan tests
  • External write variants (using catalog API to simulate writes from outside the session)
  • Scenarios 5, 6, 7 (drop+re-add column same/different type, type widening)

Does this PR introduce any user-facing change?

No. This PR only adds tests.

How was this patch tested?

All 11 new tests pass:

build/sbt 'sql/testOnly *DataSourceV2DataFrameSuite -- -z "temp view with stored plan"'

All 21 SPARK-53924 tests (10 existing + 11 new) pass with no regressions:

build/sbt 'sql/testOnly *DataSourceV2DataFrameSuite -- -z "SPARK-53924"'

Was this patch authored or co-authored using generative AI tooling?

Yes.

@dongjoon-hyun dongjoon-hyun marked this pull request as draft April 27, 2026 17:23
dongjoon-hyun
dongjoon-hyun reviewed Apr 27, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please get a new valid JIRA ID and use it in the PR title before converting back from Draft status.

@longvu-db longvu-db changed the title [SPARK-XXXXX][SQL][TESTS] Add DSv2 temp view with stored plan tests [SPARK-56643][SQL][TESTS] Add DSv2 temp view with stored plan tests Apr 27, 2026
@longvu-db longvu-db mentioned this pull request Apr 27, 2026
Closed
@longvu-db longvu-db marked this pull request as ready for review April 27, 2026 21:03
@longvu-db longvu-db force-pushed the dsv2-classic-pr1-temp-views branch from 8047eb8 to e19d5b7 Compare April 27, 2026 21:08
@longvu-db longvu-db closed this Apr 27, 2026
@longvu-db longvu-db reopened this Apr 27, 2026
@longvu-db longvu-db force-pushed the dsv2-classic-pr1-temp-views branch from e19d5b7 to 648148b Compare April 27, 2026 21:16
longvu-db added a commit to longvu-db/spark that referenced this pull request Apr 27, 2026
@longvu-db
Match PR apache#55540 structure: 13 tests, section comments, two view…
66e9544 
…s in S5

Add session variants for scenarios 3 (column removal) and 4 (table
recreate). Add v_unfiltered view to scenario 5. Include InMemoryBaseTable
fix for column drop data migration. Add section comments for all 7
scenarios. Use separate alterTable calls for external variants.

Co-authored-by: Isaac
@longvu-db
Copy link
Copy Markdown
Contributor Author

longvu-db commented Apr 27, 2026

Hey @dongjoon-hyun, do you know what's the issue behind

"Workflow run detection failed
Unable to detect the workflow run for testing the changes in your PR.

If you did not enable GitHub Actions in your forked repository, please enable it by clicking the button as shown in the image below. See also Managing Github Actions Settings for a repository for more details.
It is possible your branch is based on the old master branch in Apache Spark, please sync your branch to the latest master branch. For example as below:
git fetch upstream
git rebase upstream/master
git push origin YOUR_BRANCH --force"?

I tried to fix it with the "git fetch upstream git rebase upstream/master", but it doesn't seem to work.

longvu-db added a commit to longvu-db/spark that referenced this pull request Apr 27, 2026
@longvu-db
Revert to 11 tests matching PR apache#55540 current state
f298323 
Remove InMemoryBaseTable change, extra session tests, v_unfiltered,
and section comments to match the current PR apache#55540 structure exactly.

Co-authored-by: Isaac
@dongjoon-hyun
Copy link
Copy Markdown
Member

dongjoon-hyun commented Apr 27, 2026

Today's GitHub outage might be related, @longvu-db . It's abnormal today.

This was referenced Apr 27, 2026
Open
Closed
@longvu-db longvu-db closed this Apr 27, 2026
@longvu-db longvu-db reopened this Apr 27, 2026
longvu-db added a commit to longvu-db/spark that referenced this pull request Apr 28, 2026
@longvu-db
Match PR apache#55540 structure: 13 tests, section comments, two view…
b563094 
…s in S5

Add session variants for scenarios 3 (column removal) and 4 (table
recreate). Add v_unfiltered view to scenario 5. Include InMemoryBaseTable
fix for column drop data migration. Add section comments for all 7
scenarios. Use separate alterTable calls for external variants.

Co-authored-by: Isaac
longvu-db added a commit to longvu-db/spark that referenced this pull request Apr 28, 2026
@longvu-db
Revert to 11 tests matching PR apache#55540 current state
c361251 
Remove InMemoryBaseTable change, extra session tests, v_unfiltered,
and section comments to match the current PR apache#55540 structure exactly.

Co-authored-by: Isaac
@longvu-db longvu-db force-pushed the dsv2-classic-pr1-temp-views branch from ceac7d2 to 81eefbe Compare April 28, 2026 12:48
@longvu-db
Copy link
Copy Markdown
Contributor Author

longvu-db commented Apr 29, 2026

Unrelated test failures https://github.com/longvu-db/spark/runs/73444115182

@longvu-db longvu-db force-pushed the dsv2-classic-pr1-temp-views branch from 81eefbe to 56a1d34 Compare April 29, 2026 14:13
longvu-db added a commit to longvu-db/spark that referenced this pull request Apr 29, 2026
@longvu-db
Match PR apache#55540 structure: 13 tests, section comments, two view…
a72be53 
…s in S5

Add session variants for scenarios 3 (column removal) and 4 (table
recreate). Add v_unfiltered view to scenario 5. Include InMemoryBaseTable
fix for column drop data migration. Add section comments for all 7
scenarios. Use separate alterTable calls for external variants.

Co-authored-by: Isaac
longvu-db added a commit to longvu-db/spark that referenced this pull request Apr 29, 2026
@longvu-db
Revert to 11 tests matching PR apache#55540 current state
38e185a 
Remove InMemoryBaseTable change, extra session tests, v_unfiltered,
and section comments to match the current PR apache#55540 structure exactly.

Co-authored-by: Isaac
@longvu-db longvu-db force-pushed the dsv2-classic-pr1-temp-views branch from 798eaf3 to 0370304 Compare April 30, 2026 16:53
@longvu-db longvu-db force-pushed the dsv2-classic-pr1-temp-views branch from 8419320 to a3f9fb6 Compare May 1, 2026 09:11
longvu-db added 2 commits May 1, 2026 09:18
@longvu-db
Add schema verification assertions for empty table results
3c85016 
Assert schema field names before every checkAnswer with empty
results to verify schema is preserved correctly after table
modifications.
@longvu-db
Add Scenario 3.1 session column removal test
16656d4 
Add missing test for Section 1 Scenario 3.1 where a session
ALTER TABLE DROP COLUMN should trigger an analysis exception
when querying a temp view that references the removed column.
longvu-db
longvu-db commented May 1, 2026
View reviewed changes
Comment thread sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala
}
}

// Scenario 3.2 (external column removal)
Copy link
Copy Markdown
Contributor Author

@longvu-db longvu-db May 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Scenario 3.2 (external column removal)

@longvu-db
Empty commit to retrigger CI
c37fde1 
Co-authored-by: Isaac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dongjoon-hyun dongjoon-hyun dongjoon-hyun left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@longvu-db @dongjoon-hyun