Skip to content

Commit

Permalink
[SPARK-33965][SQL][TESTS] Recognize spark_catalog by CACHE TABLE
Browse files Browse the repository at this point in the history
…in Hive table names

### What changes were proposed in this pull request?
Remove special handling of `CacheTable` in `TestHiveQueryExecution. analyzed` because it does not allow to support of `spark_catalog` in Hive table names. `spark_catalog` could be handled by a few lines below:
```scala
      case UnresolvedRelation(ident, _, _) =>
        if (ident.length > 1 && ident.head.equalsIgnoreCase(CatalogManager.SESSION_CATALOG_NAME)) {
```
added by #30883.

### Why are the changes needed?
1. To have feature parity with v1 In-Memory catalog.
2. To be able to write unified tests for In-Memory and Hive external catalogs.

### Does this PR introduce _any_ user-facing change?
Should not.

### How was this patch tested?
By running the test suite with new UT:
```
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *CachedTableSuite"
```

Closes #30997 from MaxGekk/cache-table-spark_catalog.

Authored-by: Max Gekk <max.gekk@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
  • Loading branch information
MaxGekk authored and cloud-fan committed Jan 4, 2021
1 parent 0b647fe commit 8b3fb43
Show file tree
Hide file tree
Showing 2 changed files with 15 additions and 7 deletions.
Expand Up @@ -439,4 +439,17 @@ class CachedTableSuite extends QueryTest with SQLTestUtils with TestHiveSingleto
assert(spark.catalog.isCached(t))
}
}

test("SPARK-33965: cache table in spark_catalog") {
withNamespace("spark_catalog.ns") {
sql("CREATE NAMESPACE spark_catalog.ns")
val t = "spark_catalog.ns.tbl"
withTable(t) {
sql(s"CREATE TABLE $t (col int)")
assert(!spark.catalog.isCached(t))
sql(s"CACHE TABLE $t")
assert(spark.catalog.isCached(t))
}
}
}
}
Expand Up @@ -39,7 +39,7 @@ import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
import org.apache.spark.sql.catalyst.catalog.ExternalCatalogWithListener
import org.apache.spark.sql.catalyst.expressions.CodegenObjectFactoryMode
import org.apache.spark.sql.catalyst.optimizer.ConvertToLocalRelation
import org.apache.spark.sql.catalyst.plans.logical.{CacheTable, LogicalPlan, OneRowRelation}
import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, OneRowRelation}
import org.apache.spark.sql.connector.catalog.CatalogManager
import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
import org.apache.spark.sql.execution.{QueryExecution, SQLExecution}
Expand Down Expand Up @@ -596,13 +596,8 @@ private[hive] class TestHiveQueryExecution(
}

override lazy val analyzed: LogicalPlan = sparkSession.withActive {
val describedTables = logical match {
case CacheTable(_, tbl, _, _) => tbl.asTableIdentifier :: Nil
case _ => Nil
}

// Make sure any test tables referenced are loaded.
val referencedTables = describedTables ++ logical.collect {
val referencedTables = logical.collect {
case UnresolvedRelation(ident, _, _) =>
if (ident.length > 1 && ident.head.equalsIgnoreCase(CatalogManager.SESSION_CATALOG_NAME)) {
ident.tail.asTableIdentifier
Expand Down

0 comments on commit 8b3fb43

Please sign in to comment.