[SPARK-10327][SQL] Cache Table is not working while subquery has alias in its project list #8494

chenghao-intel · 2015-08-28T02:53:19Z

    import org.apache.spark.sql.hive.execution.HiveTableScan
    sql("select key, value, key + 1 from src").registerTempTable("abc")
    cacheTable("abc")

    val sparkPlan = sql(
      """select a.key, b.key, c.key from
        |abc a join abc b on a.key=b.key
        |join abc c on a.key=c.key""".stripMargin).queryExecution.sparkPlan

    assert(sparkPlan.collect { case e: InMemoryColumnarTableScan => e }.size === 3) // failed
    assert(sparkPlan.collect { case e: HiveTableScan => e }.size === 0) // failed

The actual plan is:

== Parsed Logical Plan ==
'Project [unresolvedalias('a.key),unresolvedalias('b.key),unresolvedalias('c.key)]
 'Join Inner, Some(('a.key = 'c.key))
  'Join Inner, Some(('a.key = 'b.key))
   'UnresolvedRelation [abc], Some(a)
   'UnresolvedRelation [abc], Some(b)
  'UnresolvedRelation [abc], Some(c)

== Analyzed Logical Plan ==
key: int, key: int, key: int
Project [key#14,key#61,key#66]
 Join Inner, Some((key#14 = key#66))
  Join Inner, Some((key#14 = key#61))
   Subquery a
    Subquery abc
     Project [key#14,value#15,(key#14 + 1) AS _c2#16]
      MetastoreRelation default, src, None
   Subquery b
    Subquery abc
     Project [key#61,value#62,(key#61 + 1) AS _c2#58]
      MetastoreRelation default, src, None
  Subquery c
   Subquery abc
    Project [key#66,value#67,(key#66 + 1) AS _c2#63]
     MetastoreRelation default, src, None

== Optimized Logical Plan ==
Project [key#14,key#61,key#66]
 Join Inner, Some((key#14 = key#66))
  Project [key#14,key#61]
   Join Inner, Some((key#14 = key#61))
    Project [key#14]
     InMemoryRelation [key#14,value#15,_c2#16], true, 10000, StorageLevel(true, true, false, true, 1), (Project [key#14,value#15,(key#14 + 1) AS _c2#16]), Some(abc)
    Project [key#61]
     MetastoreRelation default, src, None
  Project [key#66]
   MetastoreRelation default, src, None

== Physical Plan ==
TungstenProject [key#14,key#61,key#66]
 BroadcastHashJoin [key#14], [key#66], BuildRight
  TungstenProject [key#14,key#61]
   BroadcastHashJoin [key#14], [key#61], BuildRight
    ConvertToUnsafe
     InMemoryColumnarTableScan [key#14], (InMemoryRelation [key#14,value#15,_c2#16], true, 10000, StorageLevel(true, true, false, true, 1), (Project [key#14,value#15,(key#14 + 1) AS _c2#16]), Some(abc))
    ConvertToUnsafe
     HiveTableScan [key#61], (MetastoreRelation default, src, None)
  ConvertToUnsafe
   HiveTableScan [key#66], (MetastoreRelation default, src, None)

chenghao-intel · 2015-08-28T02:54:23Z

cc @marmbrus

SparkQA · 2015-08-28T05:13:13Z

Test build #41720 has finished for PR 8494 at commit bfd40d9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2015-08-29T20:10:10Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/namedExpressions.scala

@@ -155,6 +155,8 @@ case class Alias(child: Expression, name: String)(
        explicitMetadata == a.explicitMetadata
    case _ => false
  }
+
+  def cleanExprId(): Alias = Alias(child, name)(ExprId(-1), qualifiers)


Why does this have ()? It also needs scala doc. However, since its only ever used once, I'd consider just inlining it.

SparkQA · 2015-08-31T03:49:30Z

Test build #41812 has finished for PR 8494 at commit fc63b89.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2015-09-01T05:09:24Z

Test build #41852 has finished for PR 8494 at commit b6c0cb9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

marmbrus · 2015-09-08T23:16:05Z

Thanks, merging to master.

Author: Michael Armbrust <michael@databricks.com> Closes #8659 from marmbrus/testBuildBreak.

@SInCE

* apache/master: (65 commits) [SPARK-10065] [SQL] avoid the extra copy when generate unsafe array [SPARK-10497] [BUILD] [TRIVIAL] Handle both locations for JIRAError with python-jira [MINOR] [MLLIB] [ML] [DOC] fixed typo: label for negative result should be 0.0 (original: 1.0) [SPARK-9772] [PYSPARK] [ML] Add Python API for ml.feature.VectorSlicer [SPARK-9730] [SQL] Add Full Outer Join support for SortMergeJoin [SPARK-10461] [SQL] make sure `input.primitive` is always variable name not code at `GenerateUnsafeProjection` [SPARK-10481] [YARN] SPARK_PREPEND_CLASSES make spark-yarn related jar could n… [SPARK-10117] [MLLIB] Implement SQL data source API for reading LIBSVM data [SPARK-10227] fatal warnings with sbt on Scala 2.11 [SPARK-10249] [ML] [DOC] Add Python Code Example to StopWordsRemover User Guide [SPARK-9654] [ML] [PYSPARK] Add IndexToString to PySpark [SPARK-10094] Pyspark ML Feature transformers marked as experimental [SPARK-10373] [PYSPARK] move @SInCE into pyspark from sql [SPARK-10464] [MLLIB] Add WeibullGenerator for RandomDataGenerator [SPARK-9834] [MLLIB] implement weighted least squares via normal equation [SPARK-10071] [STREAMING] Output a warning when writing QueueInputDStream and throw a better exception when reading QueueInputDStream [RELEASE] Add more contributors & only show names in release notes. [HOTFIX] Fix build break caused by apache#8494 [SPARK-10327] [SQL] Cache Table is not working while subquery has alias in its project list [SPARK-10492] [STREAMING] [DOCUMENTATION] Update Streaming documentation about rate limiting and backpressure ...

…as in its project list ```scala import org.apache.spark.sql.hive.execution.HiveTableScan sql("select key, value, key + 1 from src").registerTempTable("abc") cacheTable("abc") val sparkPlan = sql( """select a.key, b.key, c.key from |abc a join abc b on a.key=b.key |join abc c on a.key=c.key""".stripMargin).queryExecution.sparkPlan assert(sparkPlan.collect { case e: InMemoryColumnarTableScan => e }.size === 3) // failed assert(sparkPlan.collect { case e: HiveTableScan => e }.size === 0) // failed ``` The actual plan is: ``` == Parsed Logical Plan == 'Project [unresolvedalias('a.key),unresolvedalias('b.key),unresolvedalias('c.key)] 'Join Inner, Some(('a.key = 'c.key)) 'Join Inner, Some(('a.key = 'b.key)) 'UnresolvedRelation [abc], Some(a) 'UnresolvedRelation [abc], Some(b) 'UnresolvedRelation [abc], Some(c) == Analyzed Logical Plan == key: int, key: int, key: int Project [key#14,key#61,key#66] Join Inner, Some((key#14 = key#66)) Join Inner, Some((key#14 = key#61)) Subquery a Subquery abc Project [key#14,value#15,(key#14 + 1) AS _c2#16] MetastoreRelation default, src, None Subquery b Subquery abc Project [key#61,value#62,(key#61 + 1) AS _c2#58] MetastoreRelation default, src, None Subquery c Subquery abc Project [key#66,value#67,(key#66 + 1) AS _c2#63] MetastoreRelation default, src, None == Optimized Logical Plan == Project [key#14,key#61,key#66] Join Inner, Some((key#14 = key#66)) Project [key#14,key#61] Join Inner, Some((key#14 = key#61)) Project [key#14] InMemoryRelation [key#14,value#15,_c2#16], true, 10000, StorageLevel(true, true, false, true, 1), (Project [key#14,value#15,(key#14 + 1) AS _c2#16]), Some(abc) Project [key#61] MetastoreRelation default, src, None Project [key#66] MetastoreRelation default, src, None == Physical Plan == TungstenProject [key#14,key#61,key#66] BroadcastHashJoin [key#14], [key#66], BuildRight TungstenProject [key#14,key#61] BroadcastHashJoin [key#14], [key#61], BuildRight ConvertToUnsafe InMemoryColumnarTableScan [key#14], (InMemoryRelation [key#14,value#15,_c2#16], true, 10000, StorageLevel(true, true, false, true, 1), (Project [key#14,value#15,(key#14 + 1) AS _c2#16]), Some(abc)) ConvertToUnsafe HiveTableScan [key#61], (MetastoreRelation default, src, None) ConvertToUnsafe HiveTableScan [key#66], (MetastoreRelation default, src, None) ``` Author: Cheng Hao <hao.cheng@intel.com> Closes apache#8494 from chenghao-intel/weird_cache.

mbautin · 2015-10-30T02:46:07Z

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala

+        BindReferences.bindReference(cleanedExprId, input, allowFailures = true)
+      case other => BindReferences.bindReference(other, input, allowFailures = true)
+    }
+
    productIterator.map {


Would it make sense to use transformAllExpressions here instead of enumerating all possible ways of how expressions may occur in the logical plan, or would that produce a different result?

Hmmm, yeah that would probably work.

…as in its project list ```scala import org.apache.spark.sql.hive.execution.HiveTableScan sql("select key, value, key + 1 from src").registerTempTable("abc") cacheTable("abc") val sparkPlan = sql( """select a.key, b.key, c.key from |abc a join abc b on a.key=b.key |join abc c on a.key=c.key""".stripMargin).queryExecution.sparkPlan assert(sparkPlan.collect { case e: InMemoryColumnarTableScan => e }.size === 3) // failed assert(sparkPlan.collect { case e: HiveTableScan => e }.size === 0) // failed ``` The actual plan is: ``` == Parsed Logical Plan == 'Project [unresolvedalias('a.key),unresolvedalias('b.key),unresolvedalias('c.key)] 'Join Inner, Some(('a.key = 'c.key)) 'Join Inner, Some(('a.key = 'b.key)) 'UnresolvedRelation [abc], Some(a) 'UnresolvedRelation [abc], Some(b) 'UnresolvedRelation [abc], Some(c) == Analyzed Logical Plan == key: int, key: int, key: int Project [key#14,key#61,key#66] Join Inner, Some((key#14 = key#66)) Join Inner, Some((key#14 = key#61)) Subquery a Subquery abc Project [key#14,value#15,(key#14 + 1) AS _c2#16] MetastoreRelation default, src, None Subquery b Subquery abc Project [key#61,value#62,(key#61 + 1) AS _c2#58] MetastoreRelation default, src, None Subquery c Subquery abc Project [key#66,value#67,(key#66 + 1) AS _c2#63] MetastoreRelation default, src, None == Optimized Logical Plan == Project [key#14,key#61,key#66] Join Inner, Some((key#14 = key#66)) Project [key#14,key#61] Join Inner, Some((key#14 = key#61)) Project [key#14] InMemoryRelation [key#14,value#15,_c2#16], true, 10000, StorageLevel(true, true, false, true, 1), (Project [key#14,value#15,(key#14 + 1) AS _c2#16]), Some(abc) Project [key#61] MetastoreRelation default, src, None Project [key#66] MetastoreRelation default, src, None == Physical Plan == TungstenProject [key#14,key#61,key#66] BroadcastHashJoin [key#14], [key#66], BuildRight TungstenProject [key#14,key#61] BroadcastHashJoin [key#14], [key#61], BuildRight ConvertToUnsafe InMemoryColumnarTableScan [key#14], (InMemoryRelation [key#14,value#15,_c2#16], true, 10000, StorageLevel(true, true, false, true, 1), (Project [key#14,value#15,(key#14 + 1) AS _c2#16]), Some(abc)) ConvertToUnsafe HiveTableScan [key#61], (MetastoreRelation default, src, None) ConvertToUnsafe HiveTableScan [key#66], (MetastoreRelation default, src, None) ``` Author: Cheng Hao <hao.cheng@intel.com> Closes apache#8494 from chenghao-intel/weird_cache.

marmbrus reviewed Aug 29, 2015
View reviewed changes

Cache Table is not working while subquery has alias in its project list

fc63b89

chenghao-intel force-pushed the weird_cache branch from bfd40d9 to fc63b89 Compare August 31, 2015 01:27

move the unit test from Hive packge to sql/core

b6c0cb9

asfgit closed this in d637a66 Sep 8, 2015

marmbrus added a commit to marmbrus/spark that referenced this pull request Sep 8, 2015

[HOTFIX] Fix build break caused by apache#8494

47780b6

asfgit pushed a commit that referenced this pull request Sep 8, 2015

[HOTFIX] Fix build break caused by #8494

2143d59

Author: Michael Armbrust <michael@databricks.com> Closes #8659 from marmbrus/testBuildBreak.

chenghao-intel deleted the weird_cache branch September 9, 2015 05:43

mbautin mentioned this pull request Oct 27, 2015

[csd-1.5] [SPARK-10327] [SQL] Cache Table is not working while subquery has ali… alteryx/spark#108

Merged

mbautin reviewed Oct 30, 2015
View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-10327][SQL] Cache Table is not working while subquery has alias in its project list #8494

[SPARK-10327][SQL] Cache Table is not working while subquery has alias in its project list #8494

chenghao-intel commented Aug 28, 2015

chenghao-intel commented Aug 28, 2015

SparkQA commented Aug 28, 2015

marmbrus Aug 29, 2015

SparkQA commented Aug 31, 2015

SparkQA commented Sep 1, 2015

marmbrus commented Sep 8, 2015

mbautin Oct 30, 2015

marmbrus Oct 30, 2015

[SPARK-10327][SQL] Cache Table is not working while subquery has alias in its project list #8494

[SPARK-10327][SQL] Cache Table is not working while subquery has alias in its project list #8494

Conversation

chenghao-intel commented Aug 28, 2015

chenghao-intel commented Aug 28, 2015

SparkQA commented Aug 28, 2015

marmbrus Aug 29, 2015

Choose a reason for hiding this comment

SparkQA commented Aug 31, 2015

SparkQA commented Sep 1, 2015

marmbrus commented Sep 8, 2015

mbautin Oct 30, 2015

Choose a reason for hiding this comment

marmbrus Oct 30, 2015

Choose a reason for hiding this comment