[SPARK-2917] [SQL] Avoid table creation in logical plan analyzing for CTAS #1846

chenghao-intel · 2014-08-08T02:17:35Z

No description provided.

chenghao-intel · 2014-08-08T05:14:09Z

Hello, Jenkins, test this please.

SparkQA · 2014-08-08T05:19:24Z

QA tests have started for PR 1846. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18186/consoleFull

SparkQA · 2014-08-08T06:29:36Z

QA results for PR 1846:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class CreateTableAsSelect(

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18186/consoleFull

chenghao-intel · 2014-08-08T06:32:12Z

Failed in Running PySpark tests

yhuai · 2014-08-11T01:11:43Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala

@@ -110,18 +110,11 @@ private[hive] class HiveMetastoreCatalog(hive: HiveContext) extends Catalog with
   */
  object CreateTables extends Rule[LogicalPlan] {
    def apply(plan: LogicalPlan): LogicalPlan = plan transform {
-      case InsertIntoCreatedTable(db, tableName, child) =>
+      case InsertIntoCreatedTable(db, tableName, child) if (db == None) =>


I do not think we want to check if db is None or not. We have

val databaseName = dbName.getOrElse(hive.sessionState.getCurrentDatabase)

right in line 115.

Oh, yes, thank you for noticing this.

yhuai · 2014-08-11T01:33:56Z

Can you explain your intention? Why do we want to add a new physical operator to take care CTAS queries and put all actual operations into sideEffectResult?

chenghao-intel · 2014-08-11T02:04:17Z

Thank you @yhuai for reviewing this.
The motivation I create this PR is currently the table will be created in logical plan analyzing (see object CreateTables) for CTAS, which may causes some side effect / bugs.

For example in #1847, I was trying to "explain" the CTAS like explain create table abc as select * from src. And probably the same in some case when developers debug their logical plan before real execution.

I think we'd better move the table creation from logical plan analyzing to executing. So I created a new Physical Operator CreateTableAsSelect

SparkQA · 2014-08-11T04:34:49Z

QA tests have started for PR 1846. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18298/consoleFull

SparkQA · 2014-08-11T04:39:41Z

QA tests have started for PR 1846. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18299/consoleFull

SparkQA · 2014-08-11T05:41:56Z

QA results for PR 1846:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class CreateTableAsSelect(

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18298/consoleFull

SparkQA · 2014-08-11T05:47:19Z

QA results for PR 1846:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class CreateTableAsSelect(

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18299/consoleFull

marmbrus · 2014-08-12T03:14:28Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateTableAsSelect.scala

+  database: String, 
+  tableName: String, 
+  child: SparkPlan,
+  insert: MetastoreRelation => InsertIntoHiveTable)(@transient sc: HiveContext) 


The tests are failing because you have parameters that are in a second parameter list but not in otherCopyArgs. However, you shouldn't need to pass the HiveContext in manually at all. It is available automatically through the sqlContext method in SparkPlan.

SparkQA · 2014-08-13T04:39:51Z

QA tests have started for PR 1846. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18409/consoleFull

SparkQA · 2014-08-13T05:47:38Z

QA results for PR 1846:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class CreateTableAsSelect(

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18409/consoleFull

chenghao-intel · 2014-08-13T06:30:30Z

Is this my fault? Jenkins, retest this please.

SparkQA · 2014-08-13T06:34:00Z

QA tests have started for PR 1846. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18424/consoleFull

SparkQA · 2014-08-13T07:38:43Z

QA results for PR 1846:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class CreateTableAsSelect(

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18424/consoleFull

chenghao-intel · 2014-08-13T08:12:02Z

Seems failed due to some other reasons.

chenghao-intel · 2014-08-14T00:34:55Z

Failed in difference places, Jenkins, retest this please.

chenghao-intel · 2014-08-14T01:15:31Z

@marmbrus seems Jenkins doesn't answer me, can you also check if the unit test failure for me? Seems failed in something else.

marmbrus · 2014-08-14T01:16:37Z

test this please

SparkQA · 2014-08-14T01:20:02Z

QA tests have started for PR 1846. This patch merges cleanly.
View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18508/consoleFull

SparkQA · 2014-08-14T02:29:00Z

QA results for PR 1846:
- This patch FAILED unit tests.
- This patch merges cleanly
- This patch adds the following public classes (experimental):
case class CreateTableAsSelect(

For more information see test ouptut:
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18508/consoleFull

chenghao-intel · 2014-08-14T02:57:42Z

Still failed in pyspark. But it passed in my local. Any suggestion?

=========================================================================
Running PySpark tests
=========================================================================
Running PySpark tests. Output is in python/unit-tests.log.
Testing with Python version:
Python 2.6.6
Running test: pyspark/rdd.py
Spark assembly has been built with Hive, including Datanucleus jars on classpath
Traceback (most recent call last):
  File "pyspark/rdd.py", line 1849, in <module>
    _test()
  File "pyspark/rdd.py", line 1840, in _test
    globs['sc'] = SparkContext('local[4]', 'PythonTest', batchSize=2)
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/context.py", line 107, in __init__
    conf)
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/context.py", line 155, in _do_init
    self._jsc = self._initialize_context(self._conf._jconf)
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/context.py", line 201, in _initialize_context
    return self._jvm.JavaSparkContext(jconf)
  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 699, in __call__

  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 369, in send_command

  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 362, in send_command

  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 318, in _get_connection

  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 325, in _create_connection

  File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 432, in start

py4j.protocol.Py4JNetworkError: An error occurred while trying to connect to the Java server

chenghao-intel · 2014-08-15T05:03:22Z

test this please

SparkQA · 2014-08-15T05:10:14Z

QA tests have started for PR 1846 at commit 5be9554.

This patch merges cleanly.

chenghao-intel · 2014-08-21T03:53:19Z

test this please

SparkQA · 2014-08-21T03:55:33Z

QA tests have started for PR 1846 at commit 5be9554.

This patch merges cleanly.

marmbrus · 2014-08-26T01:39:57Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/CreateTableAsSelect.scala

+  tableName: String, 
+  subquery: SparkPlan,
+  insert: MetastoreRelation => InsertIntoHiveTable)
+    extends LeafNode with Command {


Why is this a leaf node? I think that is going to break things like the addition of exchange operators.

Yes, that's a leaf node, and actually the child would be InsertIntoHiveTable, however, the InsertIntoHiveTable requires the table was existed in logical plan analysis, hence I pass in the MetastoreRelation => InsertIntoHiveTable as the parameter of CreateTableAsSelect.

You're right, the name subquery is quite confusing, I will change it, but the logical plan it represented is well transformed in HiveStrategies, see case logical.InsertIntoCreatedTable(database, tableName, child) in HiveStrategies.

SparkQA · 2014-08-27T04:51:47Z

QA tests have started for PR 1846 at commit ae1e072.

This patch merges cleanly.

SparkQA · 2014-08-27T06:08:03Z

QA tests have finished for PR 1846 at commit ae1e072.

This patch fails unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- "$FWDIR"/bin/spark-submit --class $CLASS "$
- class ExternalSorter(object):
- "$FWDIR"/bin/spark-submit --class $CLASS "$
- protected class AttributeEquals(val a: Attribute)
- case class CreateTableAsSelect(

chenghao-intel · 2014-08-27T14:35:19Z

retest this please

SparkQA · 2014-08-27T14:41:08Z

QA tests have started for PR 1846 at commit ae1e072.

This patch merges cleanly.

SparkQA · 2014-08-27T16:01:35Z

QA tests have finished for PR 1846 at commit ae1e072.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CreateTableAsSelect(

chenghao-intel · 2014-08-28T08:54:18Z

@marmbrus @yhuai Can you review this for me?

marmbrus · 2014-08-29T01:35:19Z

This is a pretty big change, will investigate further after the 1.1. release.

chenghao-intel · 2014-08-29T01:36:50Z

OK, sounds good to me.

marmbrus · 2014-09-10T02:23:48Z

@chenghao-intel, I'd like to this merged before I begin on my next big refactoring. One small thing, do you think you could add a test to make sure we don't regress and start creating tables for EXPLAIN queries?

chenghao-intel · 2014-09-10T02:44:38Z

Thank you @marmbrus I will add an unit test for this soon.

chenghao-intel · 2014-09-10T05:14:48Z

test this please.

SparkQA · 2014-09-10T05:51:37Z

QA tests have started for PR 1846 at commit 6521ce8.

This patch merges cleanly.

SparkQA · 2014-09-10T07:34:14Z

QA tests have finished for PR 1846 at commit 6521ce8.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CreateTableAsSelect(

marmbrus · 2014-09-10T17:55:23Z

sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLQuerySuite.scala

@@ -47,4 +47,16 @@ class SQLQuerySuite extends QueryTest {
          GROUP BY key, value
          ORDER BY value) a""").collect().toSeq)
  }
+
+  test("test CTAS") {
+    sql("CREATE TABLE test_ctas_123 AS SELECT key, value FROM src").schema


This actually isn't the semantic we want. All DDL commands should act as actions and be executed immediately. It is only when there is an EXPLAIN that we should not run the command eagerly.

I think InsertIntoCreatedTable should inherit from Command so that this happens automatically. I'd also consider renaming it CreateTableAsSelect as you did with the physical operator.

Oh, I understood. But I'd like to make that as 2 independent PRs:

Revert the SchemaRDDLike.scala and SQLQuerySuite.scala, and only keep the logic of table creation move from HiveMetastoreCatalog.scala to the physical operator CreateTableAsSelect.scala

Update the [SPARK-2918] [SQL] [WIP] Support the extended & native command for EXPLAIN #1847 for supporting the EXPLAIN for command (e.g. ctas, InsertIntoTable, WriteToFile etc.)

What do you think about this?

Yeah, as long as you don't remove InsertIntoCreatedTable from SchemaRDDLike.scala I'm okay with pushing the other changes to another PR.

SparkQA · 2014-09-11T08:53:22Z

QA tests have started for PR 1846 at commit 56a0578.

This patch merges cleanly.

SparkQA · 2014-09-11T10:38:58Z

QA tests have finished for PR 1846 at commit 56a0578.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- case class CreateTableAsSelect(
- case class CreateTableAsSelect(

marmbrus · 2014-09-11T18:57:33Z

Thanks! Merged to master.

chenghao-intel mentioned this pull request Aug 8, 2014

[SPARK-2918] [SQL] [WIP] Support the extended & native command for EXPLAIN #1847

Closed

yhuai reviewed Aug 11, 2014
View reviewed changes

marmbrus reviewed Aug 12, 2014
View reviewed changes

marmbrus reviewed Aug 26, 2014
View reviewed changes

marmbrus mentioned this pull request Sep 3, 2014

[SPARK-2781][SQL] Check resolution of LogicalPlans in Analyzer. #1706

Closed

chenghao-intel force-pushed the ctas branch from ae1e072 to 6521ce8 Compare September 10, 2014 05:12

marmbrus reviewed Sep 10, 2014
View reviewed changes

Avoid table creation in logical plan analyzing

9a57abc

chenghao-intel force-pushed the ctas branch from 6521ce8 to 9a57abc Compare September 11, 2014 08:34

remove the unused imports

56a0578

asfgit closed this in ca83f1e Sep 11, 2014

[SPARK-2917] [SQL] Avoid table creation in logical plan analyzing for CTAS #1846

[SPARK-2917] [SQL] Avoid table creation in logical plan analyzing for CTAS #1846

Conversation

chenghao-intel commented Aug 8, 2014

chenghao-intel commented Aug 8, 2014

SparkQA commented Aug 8, 2014

SparkQA commented Aug 8, 2014

chenghao-intel commented Aug 8, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yhuai commented Aug 11, 2014

chenghao-intel commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

SparkQA commented Aug 11, 2014

Choose a reason for hiding this comment

SparkQA commented Aug 13, 2014

SparkQA commented Aug 13, 2014

chenghao-intel commented Aug 13, 2014

SparkQA commented Aug 13, 2014

SparkQA commented Aug 13, 2014

chenghao-intel commented Aug 13, 2014

chenghao-intel commented Aug 14, 2014

chenghao-intel commented Aug 14, 2014

marmbrus commented Aug 14, 2014

SparkQA commented Aug 14, 2014

SparkQA commented Aug 14, 2014

chenghao-intel commented Aug 14, 2014

chenghao-intel commented Aug 15, 2014

SparkQA commented Aug 15, 2014

chenghao-intel commented Aug 21, 2014

SparkQA commented Aug 21, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Aug 27, 2014

SparkQA commented Aug 27, 2014

chenghao-intel commented Aug 27, 2014

SparkQA commented Aug 27, 2014

SparkQA commented Aug 27, 2014

chenghao-intel commented Aug 28, 2014

marmbrus commented Aug 29, 2014

chenghao-intel commented Aug 29, 2014

marmbrus commented Sep 10, 2014

chenghao-intel commented Sep 10, 2014

chenghao-intel commented Sep 10, 2014

SparkQA commented Sep 10, 2014

SparkQA commented Sep 10, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

marmbrus commented Sep 11, 2014