[SPARK-13235] [SQL] Removed an Extra Distinct from the Plan when Using Union in SQL

[gatorsmile]

Currently, the parser added two Distinct operators in the plan if we are using Union or Union Distinct in the SQL. This PR is to remove the extra Distinct from the plan.

For example, before the fix, the following query has a plan with two Distinct

sql("select * from t0 union select * from t0").explain(true)

== Parsed Logical Plan ==
'Project [unresolvedalias(*,None)]
+- 'Subquery u_2
   +- 'Distinct
      +- 'Project [unresolvedalias(*,None)]
         +- 'Subquery u_1
            +- 'Distinct
               +- 'Union
                  :- 'Project [unresolvedalias(*,None)]
                  :  +- 'UnresolvedRelation `t0`, None
                  +- 'Project [unresolvedalias(*,None)]
                     +- 'UnresolvedRelation `t0`, None

== Analyzed Logical Plan ==
id: bigint
Project [id#16L]
+- Subquery u_2
   +- Distinct
      +- Project [id#16L]
         +- Subquery u_1
            +- Distinct
               +- Union
                  :- Project [id#16L]
                  :  +- Subquery t0
                  :     +- Relation[id#16L] ParquetRelation
                  +- Project [id#16L]
                     +- Subquery t0
                        +- Relation[id#16L] ParquetRelation

== Optimized Logical Plan ==
Aggregate [id#16L], [id#16L]
+- Aggregate [id#16L], [id#16L]
   +- Union
      :- Project [id#16L]
      :  +- Relation[id#16L] ParquetRelation
      +- Project [id#16L]
         +- Relation[id#16L] ParquetRelation

After the fix, the plan is changed without the extra Distinct as follows:

== Parsed Logical Plan ==
'Project [unresolvedalias(*,None)]
+- 'Subquery u_1
   +- 'Distinct
      +- 'Union
         :- 'Project [unresolvedalias(*,None)]
         :  +- 'UnresolvedRelation `t0`, None
         +- 'Project [unresolvedalias(*,None)]
           +- 'UnresolvedRelation `t0`, None

== Analyzed Logical Plan ==
id: bigint
Project [id#17L]
+- Subquery u_1
   +- Distinct
      +- Union
        :- Project [id#16L]
        :  +- Subquery t0
        :     +- Relation[id#16L] ParquetRelation
        +- Project [id#16L]
          +- Subquery t0
          +- Relation[id#16L] ParquetRelation

== Optimized Logical Plan ==
Aggregate [id#17L], [id#17L]
+- Union
  :- Project [id#16L]
  :  +- Relation[id#16L] ParquetRelation
  +- Project [id#16L]
    +- Relation[id#16L] ParquetRelation

[SparkQA]

Test build #50938 has finished for PR 11120 at commit 2e78d2c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • s\"Unable to generate an encoder for inner class$`
  • case class AssertNotNull(child: Expression, walkedTypePath: Seq[String])
  • case class ReturnAnswer(child: LogicalPlan) extends UnaryNode
  • case class CollectLimit(limit: Int, child: SparkPlan) extends UnaryNode
  • trait BaseLimit extends UnaryNode
  • case class LocalLimit(limit: Int, child: SparkPlan) extends BaseLimit
  • case class GlobalLimit(limit: Int, child: SparkPlan) extends BaseLimit
  • case class TakeOrderedAndProject(

[gatorsmile]

This change is on the parser. @hvanhovell Could you please take a look? Thanks!

[hvanhovell]

@gatorsmile This looks pretty solid. Could you add a test for this to CatalystQlSuite? I'll have a better look at the grammar change tonight.

[viirya]

Yea, this is redundant because originally TOK_UNIONALL is used here. So an additional distinct is necessary. As we use TOK_UNIONDISTINCT now, we can skip it.

[gatorsmile]

@hvanhovell @viirya Thank you for your reviews!

Just added two test cases for this, as suggested by @hvanhovell

Yeah. I did check the original Hive JIRA: https://issues.apache.org/jira/browse/HIVE-9039 . The reason their parser added this is that they do not add another Distinct above Union All.

[SparkQA]

Test build #50975 has finished for PR 11120 at commit a5e81f4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

LGTM

[viirya]

LGTM

[hvanhovell]

Merging to master. thanks!