[SPARK-14014] [SQL] Replace existing catalog with SessionCatalog #11836

andrewor14 · 2016-03-18T23:03:46Z

What changes were proposed in this pull request?

SessionCatalog, introduced in #11750, is a catalog that keeps track of temporary functions and tables, and delegates metastore operations to ExternalCatalog. This functionality overlaps a lot with the existing analysis.Catalog.

As of this commit, SessionCatalog and ExternalCatalog will no longer be dead code. There are still things that need to be done after this patch, namely:

SPARK-14013: Properly implement temporary functions in SessionCatalog
SPARK-13879: Decide which DDL/DML commands to support natively in Spark
SPARK-?????: Implement the ones we do want to support through SessionCatalog.
SPARK-?????: Merge SQL/HiveContext

How was this patch tested?

This is largely a refactoring task so there are no new tests introduced. The particularly relevant tests are SessionCatalogSuite and ExternalCatalogSuite.

@yhuai

commit ad43a5f Author: Andrew Or <andrew@databricks.com> Date: Wed Mar 16 14:35:02 2016 -0700 Expand test scope + clean up test code commit 08969cd Author: Andrew Or <andrew@databricks.com> Date: Wed Mar 16 13:21:50 2016 -0700 Fix tests commit 6d9fa2f Author: Andrew Or <andrew@databricks.com> Date: Wed Mar 16 12:31:52 2016 -0700 Keep track of current database in SessionCatalog This allows us to not pass it into every single method like we used to before this commit. commit ff1c2c4 Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 19:42:22 2016 -0700 Add TODO commit 8c84dd8 Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 19:41:30 2016 -0700 Implement tests for functions commit 3da16fb Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 19:04:03 2016 -0700 Implement tests for table partitions commit 7947445 Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 18:52:30 2016 -0700 Implement tests for databases and tables commit 2f5121b Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 16:59:38 2016 -0700 Fix infinite loop (woops) commit d3f252d Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 16:12:55 2016 -0700 Refactor CatalogTestCases to make methods accessible commit caa4013 Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 15:44:23 2016 -0700 Clean up duplicate code in Table/FunctionIdentifier commit 90ccdbb Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 15:33:30 2016 -0700 Fix style commit 5587a49 Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 15:32:38 2016 -0700 Implement SessionCatalog using ExternalCatalog commit 196f7ce Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 14:39:22 2016 -0700 Document and clean up function methods commit 6d530a9 Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 14:38:50 2016 -0700 Fix tests commit 2118212 Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 14:33:20 2016 -0700 Refactor CatalogFunction to use FunctionIdentifier commit dd1fbae Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 14:22:37 2016 -0700 Refactor CatalogTable to use TableIdentifier This is a standalone commit such that in the future we can split it out into a separate patch if preferrable. commit 39a153c Author: Andrew Or <andrew@databricks.com> Date: Tue Mar 15 13:53:42 2016 -0700 Take into account current database in table methods commit 5bf695c Author: Andrew Or <andrew@databricks.com> Date: Mon Mar 14 17:14:59 2016 -0700 Do the same for functions and partitions commit 1d12578 Author: Andrew Or <andrew@databricks.com> Date: Mon Mar 14 16:27:11 2016 -0700 Clean up table method signatures + add comments commit 98c8a3b Author: Andrew Or <andrew@databricks.com> Date: Thu Mar 10 16:35:35 2016 -0800 Merge in @yhuai's changes

We need to be able to pass in ExternalCatalog in the constructor of SQLContext and subclasses because these should be persistent across sessions. Unfortunately without significant refactoring in the HiveContext and TestHive code we cannot make this simple change happen.

This failed because SessionCatalog does not implement refreshTable. This is a bigger problem because SessionCatalog has no notion of caching tables in the first place and so it doesn't really make sense to implement refreshTable. More refactoring involving HiveMetastoreCatalog is required to make this work.

…alog

This commit deletes the trait analysis.Catalog and all of its subclasses, with one notable exception: HiveMetastoreCatalog is kept because a lot of existing functionality (like caching data source tables) are still needed. All other occurrences are now replaced with SessionCatalog. Unfortunately, because HiveMetastoreCatalog is a massive sprawl of unmaintainable code, there is no clean way to integrate it nicely with the new HiveCatalog. The path of least resistance, then, route previous usages of HiveMetastoreCatalog through HiveCatalog. This requires some whacky initialization order hacks because HMC takes in HiveContext but HiveContext takes in HiveCatalog.

The biggest change here is moving HiveMetastoreCatalog from HiveCatalog (the external one) to HiveSessionCatalog (the session specific one). This is needed because HMC depends on a lot of session specific things for, e.g. creating data source tables. This was failing tests that do things with multiple sessions, i.e. HiveQuerySuite.

There were some issues with case sensitivity analysis and error messages not being exactly as expected. The latter is now relaxed where possible.

andrewor14 · 2016-03-18T23:03:56Z

@yhuai @rxin

SparkQA · 2016-03-18T23:17:23Z

Test build #53579 has finished for PR 11836 at commit 5e16480.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class SimpleAnalyzer(conf: CatalystConf)
- class SessionCatalog(externalCatalog: ExternalCatalog, conf: CatalystConf)

…alog

SparkQA · 2016-03-19T01:46:32Z

Test build #53582 has finished for PR 11836 at commit c439280.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-19T02:04:37Z

Test build #53587 has finished for PR 11836 at commit f089e2b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-19T03:57:31Z

Test build #53585 has finished for PR 11836 at commit 34948a5.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-03-19T04:04:06Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/InMemoryCatalog.scala

-  }
-
-  private def existsPartition(db: String, table: String, spec: TablePartitionSpec): Boolean = {
+  private def partitionExists(db: String, table: String, spec: TablePartitionSpec): Boolean = {


in the future let's separate renames likes this from large prs.

rxin · 2016-03-19T04:26:50Z

The changes look pretty good. We'd need to make Jenkins pass though.

…alog

andrewor14 · 2016-03-23T06:39:13Z

all good, let's try this again test this please

SparkQA · 2016-03-23T07:18:12Z

Test build #2664 has finished for PR 11836 at commit 5ea8469.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-03-23T07:29:29Z

looks really close ...

SparkQA · 2016-03-23T07:29:53Z

Test build #2662 has finished for PR 11836 at commit 5ea8469.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-23T07:31:24Z

Test build #2665 has finished for PR 11836 at commit 5ea8469.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-23T07:34:33Z

Test build #53900 has finished for PR 11836 at commit 5ea8469.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-23T08:40:24Z

Test build #53906 has finished for PR 11836 at commit 9519cd8.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

yhuai · 2016-03-23T14:52:05Z

Ah, we also need to update the python test after we changed the behavior of list tables in a non-existent db.

yhuai · 2016-03-23T16:53:55Z

Looks like the conflict is introduced by https://github.com/apache/spark/pull/11889/files. Should be pretty minor. But, let me see.

yhuai · 2016-03-23T17:02:28Z

looks like the conflicts are very easy to resolve.

yhuai · 2016-03-23T17:15:54Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala

-    dir
-  }
-
-  private lazy val temporaryConfig = newTemporaryConfiguration(useInMemoryDerby = false)


I see why we have to remove

TestHiveContext.hiveClientConfigurations(hiveconf, warehousePath, scratchDirPath) .foreach { case (k, v) => metadataHive.runSqlHive(s"SET $k=$v") }

in 16a54ba. temporaryConfig was originally a lazy val. But, if we use hiveClientConfigurations, we are actually generate new confs every time we call reset.

…alog

SparkQA · 2016-03-23T19:29:58Z

Test build #53952 has finished for PR 11836 at commit c53f483.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-03-23T20:21:25Z

Test build #53955 has finished for PR 11836 at commit ea1645b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-03-23T20:33:53Z

Merging in master - thanks!

SparkQA · 2016-03-23T22:02:39Z

Test build #53956 has finished for PR 11836 at commit 350bd2e.

This patch fails from timeout after a configured wait of 250m.
This patch merges cleanly.
This patch adds no public classes.

## What changes were proposed in this pull request? `SessionCatalog`, introduced in apache#11750, is a catalog that keeps track of temporary functions and tables, and delegates metastore operations to `ExternalCatalog`. This functionality overlaps a lot with the existing `analysis.Catalog`. As of this commit, `SessionCatalog` and `ExternalCatalog` will no longer be dead code. There are still things that need to be done after this patch, namely: - SPARK-14013: Properly implement temporary functions in `SessionCatalog` - SPARK-13879: Decide which DDL/DML commands to support natively in Spark - SPARK-?????: Implement the ones we do want to support through `SessionCatalog`. - SPARK-?????: Merge SQL/HiveContext ## How was this patch tested? This is largely a refactoring task so there are no new tests introduced. The particularly relevant tests are `SessionCatalogSuite` and `ExternalCatalogSuite`. Author: Andrew Or <andrew@databricks.com> Author: Yin Huai <yhuai@databricks.com> Closes apache#11836 from andrewor14/use-session-catalog.

## What changes were proposed in this pull request? This reopens #11836, which was merged but promptly reverted because it introduced flaky Hive tests. ## How was this patch tested? See `CatalogTestCases`, `SessionCatalogSuite` and `HiveContextSuite`. Author: Andrew Or <andrew@databricks.com> Closes #11938 from andrewor14/session-catalog-again.

Andrew Or added 8 commits March 16, 2016 15:50

Merge branch 'master' of github.com:apache/spark into use-session-cat…

5daa696

…alog

Fix style

71a01e0

Fix tests round 2

5e16480

There were some issues with case sensitivity analysis and error messages not being exactly as expected. The latter is now relaxed where possible.

Andrew Or added 2 commits March 18, 2016 16:29

Fix MiMa

57c8c29

Merge branch 'master' of github.com:apache/spark into use-session-cat…

c439280

…alog

andrewor14 force-pushed the use-session-catalog branch from c0f2733 to c439280 Compare March 18, 2016 23:30

Andrew Or added 3 commits March 18, 2016 16:39

Minor fixes

a3c6bf7

sessionState.sessionCatalog -> sessionState.catalog

193d93c

Fix tests round 3 (small round)

f089e2b

andrewor14 force-pushed the use-session-catalog branch from 34948a5 to f089e2b Compare March 18, 2016 23:52

rxin reviewed Mar 19, 2016
View reviewed changes

Andrew Or added 2 commits March 19, 2016 11:06

Merge branch 'master' of github.com:apache/spark into use-session-cat…

9cd89f8

…alog

Don't bother sessionizing HiveCatalog

f41346b

Merge branch 'master' of github.com:apache/spark into use-session-cat…

9519cd8

…alog

yhuai mentioned this pull request Mar 23, 2016

[SPARK-14014] [SQL] Replace existing catalog with SessionCatalog #11918

Closed

yhuai reviewed Mar 23, 2016
View reviewed changes

yhuai and others added 3 commits March 23, 2016 10:31

Use default as the db.

e5f82b2

Merge branch 'master' of github.com:apache/spark into use-session-cat…

c53f483

…alog

Fix R test?

350bd2e

andrewor14 force-pushed the use-session-catalog branch from ea1645b to 350bd2e Compare March 23, 2016 17:48

asfgit closed this in 5dfc019 Mar 23, 2016

andrewor14 deleted the use-session-catalog branch March 23, 2016 20:53

andrewor14 mentioned this pull request Mar 23, 2016

[SPARK-13923][SPARK-14014][SQL] Session catalog follow-ups #11923

Closed

liancheng mentioned this pull request Mar 24, 2016

[SPARK-14070] [SQL] Use ORC data source for SQL queries on ORC tables #11891

Closed

andrewor14 mentioned this pull request Mar 24, 2016

[SPARK-14014][SQL] Integrate session catalog (attempt #2) #11938

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-14014] [SQL] Replace existing catalog with SessionCatalog #11836

[SPARK-14014] [SQL] Replace existing catalog with SessionCatalog #11836

andrewor14 commented Mar 18, 2016

andrewor14 commented Mar 18, 2016

SparkQA commented Mar 18, 2016

SparkQA commented Mar 19, 2016

SparkQA commented Mar 19, 2016

SparkQA commented Mar 19, 2016

rxin Mar 19, 2016

rxin commented Mar 19, 2016

andrewor14 commented Mar 23, 2016

SparkQA commented Mar 23, 2016

rxin commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

yhuai commented Mar 23, 2016

yhuai commented Mar 23, 2016

yhuai commented Mar 23, 2016

yhuai Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

rxin commented Mar 23, 2016

SparkQA commented Mar 23, 2016

[SPARK-14014] [SQL] Replace existing catalog with SessionCatalog #11836

[SPARK-14014] [SQL] Replace existing catalog with SessionCatalog #11836

Conversation

andrewor14 commented Mar 18, 2016

What changes were proposed in this pull request?

How was this patch tested?

andrewor14 commented Mar 18, 2016

SparkQA commented Mar 18, 2016

SparkQA commented Mar 19, 2016

SparkQA commented Mar 19, 2016

SparkQA commented Mar 19, 2016

rxin Mar 19, 2016

Choose a reason for hiding this comment

rxin commented Mar 19, 2016

andrewor14 commented Mar 23, 2016

SparkQA commented Mar 23, 2016

rxin commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

yhuai commented Mar 23, 2016

yhuai commented Mar 23, 2016

yhuai commented Mar 23, 2016

yhuai Mar 23, 2016

Choose a reason for hiding this comment

SparkQA commented Mar 23, 2016

SparkQA commented Mar 23, 2016

rxin commented Mar 23, 2016

SparkQA commented Mar 23, 2016