-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-33652][SQL] DSv2: DeleteFrom should refresh cache #30597
Conversation
Kubernetes integration test starting |
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala
Outdated
Show resolved
Hide resolved
Kubernetes integration test status failure |
Test build #132166 has finished for PR 30597 at commit
|
Retest this please. |
Kubernetes integration test starting |
Kubernetes integration test status success |
Test build #132209 has finished for PR 30597 at commit
|
retest this please |
...lyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
Outdated
Show resolved
Hide resolved
Kubernetes integration test starting |
Kubernetes integration test status success |
Test build #132248 has finished for PR 30597 at commit
|
Test build #132257 has finished for PR 30597 at commit
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, LGTM. Thank you, @sunchao , @HyukjinKwon , and @viirya .
Merged to master/3.1.
### What changes were proposed in this pull request? This changes `DeleteFromTableExec` to also refresh caches referencing the original table, by passing the `refreshCache` callback to the class. Note that in order to construct the callback, I have to change `DataSourceV2ScanRelation` to contain a `DataSourceV2Relation` instead of a `Table`. ### Why are the changes needed? Currently DSv2 delete from table doesn't refresh caches. This could lead to correctness issue if the staled cache is queried later. ### Does this PR introduce _any_ user-facing change? Yes. Now delete from table in v2 also refreshes cache. ### How was this patch tested? Added a test case. Closes #30597 from sunchao/SPARK-33652. Authored-by: Chao Sun <sunchao@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit e857e06) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Thanks for merging @dongjoon-hyun ! |
What changes were proposed in this pull request?
This changes
DeleteFromTableExec
to also refresh caches referencing the original table, by passing therefreshCache
callback to the class. Note that in order to construct the callback, I have to changeDataSourceV2ScanRelation
to contain aDataSourceV2Relation
instead of aTable
.Why are the changes needed?
Currently DSv2 delete from table doesn't refresh caches. This could lead to correctness issue if the staled cache is queried later.
Does this PR introduce any user-facing change?
Yes. Now delete from table in v2 also refreshes cache.
How was this patch tested?
Added a test case.