[SPARK-32219][SQL] Add SHOW CACHED TABLES Command #29034

ulysses-you · 2020-07-08T06:22:56Z

What changes were proposed in this pull request?

Modify SHOW TABLES to support SHOW CACHED TABLES.
After this pr, we can call

SHOW CACHED TABLES;

Why are the changes needed?

Once we cache table in sql mode, we have no way to get it back, so that we don't know which table has cached.
If we want to reduce some cache, we have to do clear cache that remove all cached tables.
SHOW CACHED TABLES can give a way to find which table is cached, then we can uncache the one we want.

Does this PR introduce any user-facing change?

Yes, a new command.

How was this patch tested?

New test.

SparkQA · 2020-07-08T07:05:02Z

Test build #125308 has finished for PR 29034 at commit 5d362f6.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/test/resources/sql-tests/inputs/show-tables.sql

maropu · 2020-07-08T08:14:06Z

Looks useful cc: @HyukjinKwon @viirya

maropu · 2020-07-08T08:14:14Z

retest this please

SparkQA · 2020-07-08T08:20:15Z

Test build #125335 has started for PR 29034 at commit 5d362f6.

SparkQA · 2020-07-08T14:09:33Z

Test build #125345 has finished for PR 29034 at commit 564725a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-08T14:14:10Z

retest this please

SparkQA · 2020-07-08T20:52:54Z

Test build #125372 has finished for PR 29034 at commit 564725a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

viirya

Catalog provides isCached(tableName: String) API. Can't we use it to know which table is cached?

dilipbiswal · 2020-07-09T07:53:07Z

@maropu @viirya Also, i was thinking, if we want to make this information available from one of the SHOW commands, can we not extend show tables to show another attribute isCached ? Do we need a special syntax for it ? I guess, the one advantage of this command is that it acts as a filter and only shows the cached tables. But having this in one command gives a complete view of all the tables, regular, temporary and cached tables.

And we could apply filter to only see individual table types.

spark.sql("show tables").where("isCached= true").show
spark.sql("show tables").where("isTemporary= true").show

What do you guys think ?

ulysses-you · 2020-07-09T09:49:26Z

@viirya we can, but in sql mode e.g. ThriftServer, we have no way to get it.

maropu · 2020-07-09T10:42:55Z

Yea, I think this fix is for SQL interfaces.

But having this in one command gives a complete view of all the tables, regular, temporary and cached tables.

hm, adding a new column isCache looks fine, too, for that purpose. WDYT? @ulysses-you
I feel that might be better than adding a new parser token CACHED.

ulysses-you · 2020-07-09T13:01:25Z

It's ok to add an attribute isCached but it is not enough. 2 reasons:

A database can contains hundreds tables, spark.sql("show tables").where("isCached= true").show can not used in sql mode. Then it's hard to find all cached tables in all tables.
SHOW TABLES in SparkSQLDriver only show the table name without other attribute which is for compatible with Hive. Then the new attribute is useless in this scene.

dilipbiswal · 2020-07-09T16:39:19Z

@ulysses-you

A database can contains hundreds tables, spark.sql("show tables").where("isCached= true").show can not used in sql mode. Then it's hard to find all cached tables in all tables.

Ok. Sounds good !!

SHOW TABLES in SparkSQLDriver only show the table name without other attribute which is for compatible with Hive. Then the new attribute is useless in this scene.

I was not aware of this !! Thanks !!

maropu · 2020-07-10T01:39:20Z

retest this please

maropu · 2020-07-10T01:39:32Z

Looks okay as it is.

SparkQA · 2020-07-10T04:48:27Z

Test build #125534 has finished for PR 29034 at commit 564725a.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-07-10T06:46:13Z

Test build #125556 has finished for PR 29034 at commit 5138cdd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-10T08:33:26Z

retest this please

SparkQA · 2020-07-10T15:46:05Z

Test build #125584 has finished for PR 29034 at commit 5138cdd.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-07-10T22:28:13Z

retest this please

maropu · 2020-07-10T22:30:26Z

@ulysses-you Ur, could you resolve the conflict?

…2219

SparkQA · 2020-07-11T02:50:45Z

Test build #125646 has finished for PR 29034 at commit 5138cdd.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2020-07-11T03:02:27Z

Test build #125655 has finished for PR 29034 at commit 14ead4e.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class IsExecutorAlive(executorId: String) extends CoarseGrainedClusterMessage
class DisableHints(conf: SQLConf) extends RemoveAllHints(conf: SQLConf)

SparkQA · 2020-07-11T07:05:01Z

Test build #125664 has finished for PR 29034 at commit 856026b.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

ulysses-you · 2020-07-18T09:28:10Z

cc @maropu @cloud-fan thanks for review.

HyukjinKwon · 2020-07-20T08:33:41Z

retest this please

SparkQA · 2020-07-20T13:50:19Z

Test build #126170 has finished for PR 29034 at commit 856026b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-08-03T14:03:08Z

retest this please

SparkQA · 2020-08-03T19:24:52Z

Test build #126987 has finished for PR 29034 at commit 856026b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/test/resources/sql-tests/results/show-tables.sql.out

maropu · 2020-08-06T04:15:50Z

I'm okay to support it and no more comment. Anyone could check this? also cc: @cloud-fan

…2219

SparkQA · 2020-08-07T12:50:50Z

Test build #127196 has finished for PR 29034 at commit 13e1c35.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-07T12:59:41Z

Test build #127195 has finished for PR 29034 at commit 3a0b049.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2020-08-07T17:17:21Z

Test build #127206 has finished for PR 29034 at commit 8262fe2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-08-09T16:49:10Z

Test build #127238 has finished for PR 29034 at commit 929c00d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…2219

SparkQA · 2020-11-11T11:10:05Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35530/

SparkQA · 2020-11-11T11:33:14Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35530/

SparkQA · 2020-11-11T13:44:50Z

Test build #130925 has finished for PR 29034 at commit a9b88ca.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala

…2219

SparkQA · 2020-12-21T13:41:04Z

Test build #133162 has finished for PR 29034 at commit 63163ed.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-12-23T16:21:13Z

Test build #133312 has finished for PR 29034 at commit 635d3ab.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

github-actions · 2021-04-03T00:17:27Z

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

ulysses-you added 2 commits July 8, 2020 14:09

init

9cc5d45

update sql-ref-ansi-compliance.md

5d362f6

probot-autolabeler bot added DOCS SQL labels Jul 8, 2020

maropu reviewed Jul 8, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/inputs/show-tables.sql Show resolved Hide resolved

add cache as query test

564725a

viirya reviewed Jul 9, 2020

View reviewed changes

fix

5138cdd

Merge branch 'master' of https://github.com/apache/spark into SPARK-3…

14ead4e

…2219

fix

856026b

maropu reviewed Aug 6, 2020

View reviewed changes

sql/core/src/test/resources/sql-tests/results/show-tables.sql.out Show resolved Hide resolved

ulysses-you added 2 commits August 7, 2020 17:40

add test

3a0b049

Merge branch 'master' of https://github.com/apache/spark into SPARK-3…

13e1c35

…2219

fix

8262fe2

fix

929c00d

Merge branch 'master' of https://github.com/apache/spark into SPARK-3…

a9b88ca

…2219

sunchao reviewed Nov 11, 2020

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala Show resolved Hide resolved

Merge branch 'master' of https://github.com/apache/spark into SPARK-3…

63163ed

…2219

fix

635d3ab

github-actions bot added the Stale label Apr 3, 2021

github-actions bot closed this Apr 4, 2021

[SPARK-32219][SQL] Add SHOW CACHED TABLES Command #29034

[SPARK-32219][SQL] Add SHOW CACHED TABLES Command #29034

Conversation

ulysses-you commented Jul 8, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Jul 8, 2020

maropu commented Jul 8, 2020

maropu commented Jul 8, 2020

SparkQA commented Jul 8, 2020

SparkQA commented Jul 8, 2020

maropu commented Jul 8, 2020

SparkQA commented Jul 8, 2020

viirya left a comment • edited

Choose a reason for hiding this comment

dilipbiswal commented Jul 9, 2020 • edited

ulysses-you commented Jul 9, 2020

maropu commented Jul 9, 2020

ulysses-you commented Jul 9, 2020

dilipbiswal commented Jul 9, 2020

maropu commented Jul 10, 2020

maropu commented Jul 10, 2020

SparkQA commented Jul 10, 2020

SparkQA commented Jul 10, 2020

maropu commented Jul 10, 2020

SparkQA commented Jul 10, 2020

maropu commented Jul 10, 2020

maropu commented Jul 10, 2020

SparkQA commented Jul 11, 2020

SparkQA commented Jul 11, 2020

SparkQA commented Jul 11, 2020

ulysses-you commented Jul 18, 2020

HyukjinKwon commented Jul 20, 2020

SparkQA commented Jul 20, 2020

maropu commented Aug 3, 2020

SparkQA commented Aug 3, 2020

maropu commented Aug 6, 2020

SparkQA commented Aug 7, 2020

SparkQA commented Aug 7, 2020

SparkQA commented Aug 7, 2020

SparkQA commented Aug 9, 2020

SparkQA commented Nov 11, 2020

SparkQA commented Nov 11, 2020

SparkQA commented Nov 11, 2020

SparkQA commented Dec 21, 2020

SparkQA commented Dec 23, 2020

github-actions bot commented Apr 3, 2021

viirya left a comment •

edited

dilipbiswal commented Jul 9, 2020 •

edited