[SPARK-33653][SQL] DSv2: REFRESH TABLE should recache the table itself #30742
Conversation
|
Kubernetes integration test starting |
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
Outdated
Show resolved
Hide resolved
|
Kubernetes integration test status success |
|
Kubernetes integration test starting |
|
Kubernetes integration test status failure |
|
Test build #132680 has finished for PR 30742 at commit
|
|
Test build #132685 has finished for PR 30742 at commit
|
| val v2Relation = DataSourceV2Relation.create(r.table, Some(r.catalog), Some(r.identifier)) | ||
| val cache = session.sharedState.cacheManager.lookupCachedData(v2Relation) |
There was a problem hiding this comment.
I think cache is only necessary in if block?
There was a problem hiding this comment.
my bad - it should be checked in the if condition
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
Outdated
Show resolved
Hide resolved
|
Kubernetes integration test starting |
|
Kubernetes integration test status success |
|
Test build #132779 has finished for PR 30742 at commit
|
There was a problem hiding this comment.
Sorry, but could you rebase to the master branch once more, @sunchao ?
|
@dongjoon-hyun sure - it's done |
|
Could you make a backporting PR to |
### What changes were proposed in this pull request? This changes DSv2 refresh table semantics to also recache the target table itself. ### Why are the changes needed? Currently "REFRESH TABLE" in DSv2 only invalidate all caches referencing the table. With apache#30403 merged which adds support for caching a DSv2 table, we should also recache the target table itself to make the behavior consistent with DSv1. ### Does this PR introduce _any_ user-facing change? Yes, now refreshing table in DSv2 also recache the target table itself. ### How was this patch tested? Added coverage of this new behavior in the existing UT for v2 refresh table command Closes apache#30742 from sunchao/SPARK-33653. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
|
@dongjoon-hyun sure - opened #30769 |
|
Test build #132782 has finished for PR 30742 at commit
|
…itself This is a backport of #30742 for branch-3.1 ### What changes were proposed in this pull request? This changes DSv2 refresh table semantics to also recache the target table itself. ### Why are the changes needed? Currently "REFRESH TABLE" in DSv2 only invalidate all caches referencing the table. With #30403 merged which adds support for caching a DSv2 table, we should also recache the target table itself to make the behavior consistent with DSv1. ### Does this PR introduce _any_ user-facing change? Yes, now refreshing table in DSv2 also recache the target table itself. ### How was this patch tested? Added coverage of this new behavior in the existing UT for v2 refresh table command. Closes #30769 from sunchao/SPARK-33653-branch-3.1. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
What changes were proposed in this pull request?
This changes DSv2 refresh table semantics to also recache the target table itself.
Why are the changes needed?
Currently "REFRESH TABLE" in DSv2 only invalidate all caches referencing the table. With #30403 merged which adds support for caching a DSv2 table, we should also recache the target table itself to make the behavior consistent with DSv1.
Does this PR introduce any user-facing change?
Yes, now refreshing table in DSv2 also recache the target table itself.
How was this patch tested?
Added coverage of this new behavior in the existing UT for v2 refresh table command