[SPARK-19765][SPARK-18549][SQL] UNCACHE TABLE should un-cache all cached plans that refer to this table #17097

cloud-fan · 2017-02-28T09:10:17Z

What changes were proposed in this pull request?

When un-cache a table, we should not only remove the cache entry for this table, but also un-cache any other cached plans that refer to this table.

This PR also includes some refactors:

use java.util.LinkedList to store the cache entries, so that it's safer to remove elements while iterating
rename invalidateCache to recacheByPlan, which is more obvious about what it does.

How was this patch tested?

new regression test

cloud-fan · 2017-02-28T09:17:07Z

cc @sameeragarwal @gatorsmile @windpiger

windpiger · 2017-02-28T09:51:41Z

great job~

SparkQA · 2017-02-28T10:17:51Z

Test build #73581 has finished for PR 17097 at commit 6b57fd4.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-02-28T12:30:16Z

Test build #73586 has finished for PR 17097 at commit 97cbfe2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-02T07:32:40Z

Test build #73736 has started for PR 17097 at commit e881f29.

cloud-fan · 2017-03-02T08:31:12Z

retest this please

SparkQA · 2017-03-02T10:28:51Z

Test build #73741 has finished for PR 17097 at commit e881f29.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-03-05T21:20:22Z

retest this please

gatorsmile · 2017-03-05T21:47:51Z

sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala

+
+  /**
+   * Tries to remove the cache entry of the given plan, no operation, if it's already uncached.
+   * Note that all other caches that refer to this plan will be re-cached.


This is a behavior change.

Will uncache be a very expensive operation if we recache all the related cached plans?

SparkQA · 2017-03-05T23:23:10Z

Test build #73939 has finished for PR 17097 at commit e881f29.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-06T00:20:09Z

Test build #73940 has finished for PR 17097 at commit 6748017.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-06T00:37:59Z

Test build #73943 has finished for PR 17097 at commit 33124d5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-06T01:25:14Z

Test build #73946 has finished for PR 17097 at commit 067ddf5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

gatorsmile · 2017-03-06T05:54:00Z

.../src/main/scala/org/apache/spark/sql/execution/datasources/InsertIntoDataSourceCommand.scala

-    sparkSession.sharedState.cacheManager.invalidateCache(logicalRelation)
+    // Re-cache all cached plans(including this relation itself, if it's cached) that refer to this
+    // data source relation.
+    sparkSession.sharedState.cacheManager.recacheByPlan(sparkSession, logicalRelation)


In the current design, users need to re-cache the queries by themselves.

After this change, insertion could be super slow. Each insert could trigger the recache of many involved cached data, each of which could be very complex and expensive. That is a trade-off. Although we keep the data correctness/consistence, we could sacrifice the performance/user experience.

I didn't change the behavior, just rename invalidateCache to recacheByPlan.

I'll open a new discussion about whether we should do recache after insertion or not.

uh... My fault... invalidateCache is misleading. Renaming is good!

gatorsmile · 2017-03-06T21:20:57Z

SPARK-18549: Failed to Uncache a View that References a Dropped Table is also resolved by this PR. Could you also add a test case for that scenario?

gatorsmile · 2017-03-06T21:25:15Z

sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala

-   * No operation, if it's already uncached.
+   * Un-cache all the cache entries that refer to the given plan.
+   */
+  def uncacheQuery(query: Dataset[_], blocking: Boolean = true): Unit = writeLock {


DropTableCommand, TruncateTableCommand, AlterTableRenameCommand, UncacheTableCommand, RefreshTable and InsertIntoHiveTable will be affected by the behavior changes of this function, right?

yes, but I don't want to test all of them, just pick a typical one and prove the uncacheQuery works correctly.

there is another scenario that after alter datasource table location, we can't uncache the table,
JIRA

cloud-fan · 2017-03-07T06:43:06Z

@gatorsmile I have put the JIRA number in this PR title, without adding a test, because the new behavior this PR introduced can obviously fix that bug.

I don't want to add a lot of end tests to prove a single functionality.

gatorsmile · 2017-03-07T14:07:08Z

Different from materialized views, currently, our caches are always used for query rewriting. Thus, uncaching all the invalid data makes sense.

Will finish the review today. Thanks!

gatorsmile · 2017-03-07T16:58:53Z

LGTM

gatorsmile · 2017-03-07T17:23:11Z

Thanks! Merging to master.

gatorsmile · 2017-03-07T17:23:48Z

Should we backport it to the previous releases?

cloud-fan · 2017-03-07T18:28:13Z

yea, we should, but I probably don't have time to do this. Can you take over? thanks!

gatorsmile · 2017-03-07T18:35:54Z

Sure. Will do it.

…L] Backport Three Cache-related PRs to Spark 2.1 ### What changes were proposed in this pull request? Backport a few cache related PRs: --- [[SPARK-19093][SQL] Cached tables are not used in SubqueryExpression](#16493) Consider the plans inside subquery expressions while looking up cache manager to make use of cached data. Currently CacheManager.useCachedData does not consider the subquery expressions in the plan. --- [[SPARK-19736][SQL] refreshByPath should clear all cached plans with the specified path](#17064) Catalog.refreshByPath can refresh the cache entry and the associated metadata for all dataframes (if any), that contain the given data source path. However, CacheManager.invalidateCachedPath doesn't clear all cached plans with the specified path. It causes some strange behaviors reported in SPARK-15678. --- [[SPARK-19765][SPARK-18549][SQL] UNCACHE TABLE should un-cache all cached plans that refer to this table](#17097) When un-cache a table, we should not only remove the cache entry for this table, but also un-cache any other cached plans that refer to this table. The following commands trigger the table uncache: `DropTableCommand`, `TruncateTableCommand`, `AlterTableRenameCommand`, `UncacheTableCommand`, `RefreshTable` and `InsertIntoHiveTable` This PR also includes some refactors: - use java.util.LinkedList to store the cache entries, so that it's safer to remove elements while iterating - rename invalidateCache to recacheByPlan, which is more obvious about what it does. ### How was this patch tested? N/A Author: Xiao Li <gatorsmile@gmail.com> Closes #17319 from gatorsmile/backport-17097.

meteorchenwu · 2017-07-31T14:41:16Z

What a bad change it is!
Now it can not support for this scene any more.
https://issues.apache.org/jira/browse/SPARK-21579

gatorsmile · 2017-07-31T15:51:47Z

@meteorchenwu Correctness is more important for us. The cached plan will be reused when we build any other plans. Thus, users might see the out-of-dated results.

To achieve what you want, it requires introducing new concept, like materialized views, which will not be used by plan matching.

viirya mentioned this pull request Feb 28, 2017

[SPARK-19736][SQL] refreshByPath should clear all cached plans with the specified path #17064

Closed

cloud-fan force-pushed the cache branch from 6b57fd4 to 97cbfe2 Compare February 28, 2017 10:59

UNCACHE TABLE should re-cache all cached plans that refer to this table

e881f29

cloud-fan force-pushed the cache branch from 97cbfe2 to e881f29 Compare March 2, 2017 07:28

gatorsmile reviewed Mar 5, 2017

View reviewed changes

cloud-fan force-pushed the cache branch from 6748017 to 33124d5 Compare March 5, 2017 22:37

address comments

067ddf5

cloud-fan force-pushed the cache branch from 33124d5 to 067ddf5 Compare March 5, 2017 23:22

cloud-fan changed the title ~~[SPARK-19765][SQL] UNCACHE TABLE should re-cache all cached plans that refer to this table~~ [SPARK-19765][SQL] UNCACHE TABLE should un-cache all cached plans that refer to this table Mar 5, 2017

gatorsmile reviewed Mar 6, 2017

View reviewed changes

cloud-fan changed the title ~~[SPARK-19765][SQL] UNCACHE TABLE should un-cache all cached plans that refer to this table~~ [SPARK-19765][SPARK-18549][SQL] UNCACHE TABLE should un-cache all cached plans that refer to this table Mar 7, 2017

asfgit closed this in c05baab Mar 7, 2017

This was referenced Mar 16, 2017

[SPARK-19329][SQL]Reading from or writing to a datasource table with a non pre-existing location should succeed #16672

Closed

[SPARK-19765][SPARK-18549][SPARK-19093][SPARK-19736][BACKPORT-2.1][SQL] Backport Three Cache-related PRs to Spark 2.1 #17319

Closed

gatorsmile mentioned this pull request Jul 29, 2017

[SPARK-21478][SQL] Avoid unpersisting related Datasets #18771

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-19765][SPARK-18549][SQL] UNCACHE TABLE should un-cache all cached plans that refer to this table #17097

[SPARK-19765][SPARK-18549][SQL] UNCACHE TABLE should un-cache all cached plans that refer to this table #17097

cloud-fan commented Feb 28, 2017 •

edited

Loading

cloud-fan commented Feb 28, 2017

windpiger commented Feb 28, 2017

SparkQA commented Feb 28, 2017

SparkQA commented Feb 28, 2017

SparkQA commented Mar 2, 2017

cloud-fan commented Mar 2, 2017

SparkQA commented Mar 2, 2017

gatorsmile commented Mar 5, 2017

gatorsmile Mar 5, 2017

SparkQA commented Mar 5, 2017

SparkQA commented Mar 6, 2017

SparkQA commented Mar 6, 2017

SparkQA commented Mar 6, 2017

gatorsmile Mar 6, 2017

cloud-fan Mar 6, 2017

gatorsmile Mar 6, 2017

gatorsmile commented Mar 6, 2017

gatorsmile Mar 6, 2017 •

edited

Loading

cloud-fan Mar 6, 2017

windpiger Mar 7, 2017

cloud-fan commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

cloud-fan commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

meteorchenwu commented Jul 31, 2017

gatorsmile commented Jul 31, 2017

[SPARK-19765][SPARK-18549][SQL] UNCACHE TABLE should un-cache all cached plans that refer to this table #17097

[SPARK-19765][SPARK-18549][SQL] UNCACHE TABLE should un-cache all cached plans that refer to this table #17097

Conversation

cloud-fan commented Feb 28, 2017 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

cloud-fan commented Feb 28, 2017

windpiger commented Feb 28, 2017

SparkQA commented Feb 28, 2017

SparkQA commented Feb 28, 2017

SparkQA commented Mar 2, 2017

cloud-fan commented Mar 2, 2017

SparkQA commented Mar 2, 2017

gatorsmile commented Mar 5, 2017

gatorsmile Mar 5, 2017

Choose a reason for hiding this comment

SparkQA commented Mar 5, 2017

SparkQA commented Mar 6, 2017

SparkQA commented Mar 6, 2017

SparkQA commented Mar 6, 2017

gatorsmile Mar 6, 2017

Choose a reason for hiding this comment

cloud-fan Mar 6, 2017

Choose a reason for hiding this comment

gatorsmile Mar 6, 2017

Choose a reason for hiding this comment

gatorsmile commented Mar 6, 2017

gatorsmile Mar 6, 2017 • edited Loading

Choose a reason for hiding this comment

cloud-fan Mar 6, 2017

Choose a reason for hiding this comment

windpiger Mar 7, 2017

Choose a reason for hiding this comment

cloud-fan commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

cloud-fan commented Mar 7, 2017

gatorsmile commented Mar 7, 2017

meteorchenwu commented Jul 31, 2017

gatorsmile commented Jul 31, 2017

cloud-fan commented Feb 28, 2017 •

edited

Loading

gatorsmile Mar 6, 2017 •

edited

Loading