Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SQL] [Minor] Print detail query execution info when spark answer is not right #5774

Closed
wants to merge 1 commit into from

Conversation

scwf
Copy link
Contributor

@scwf scwf commented Apr 29, 2015

Print detail query execution info including parsed/analyzed/optimized/Physical plan for query when spak answer is not rignt.

Results do not match for query:
== Parsed Logical Plan ==
'Aggregate ['x.str], ['x.str,SUM('x.strCount) AS c1#46]
 'Join Inner, Some(('x.str = 'y.str))
  'UnresolvedRelation [df], Some(x)
  'UnresolvedRelation [df], Some(y)

== Analyzed Logical Plan ==
Aggregate [str#44], [str#44,SUM(strCount#45L) AS c1#46L]
 Join Inner, Some((str#44 = str#51))
  Subquery x
   Subquery df
    Aggregate [str#44], [str#44,COUNT(str#44) AS strCount#45L]
     Project [_1#41 AS int#43,_2#42 AS str#44]
      LocalRelation [_1#41,_2#42], [[1,1],[2,2],[3,3]]
  Subquery y
   Subquery df
    Aggregate [str#51], [str#51,COUNT(str#51) AS strCount#47L]
     Project [_1#41 AS int#50,_2#42 AS str#51]
      LocalRelation [_1#41,_2#42], [[1,1],[2,2],[3,3]]

== Optimized Logical Plan ==
Aggregate [str#44], [str#44,SUM(strCount#45L) AS c1#46L]
 Project [str#44,strCount#45L]
  Join Inner, Some((str#44 = str#51))
   Aggregate [str#44], [str#44,COUNT(str#44) AS strCount#45L]
    LocalRelation [str#44], [[1],[2],[3]]
   Aggregate [str#51], [str#51]
    LocalRelation [str#51], [[1],[2],[3]]

== Physical Plan ==
Aggregate false, [str#44], [str#44,CombineSum(PartialSum#53L) AS c1#46L]
 Aggregate true, [str#44], [str#44,SUM(strCount#45L) AS PartialSum#53L]
  Project [str#44,strCount#45L]
   BroadcastHashJoin [str#44], [str#51], BuildRight
    Aggregate false, [str#44], [str#44,Coalesce(SUM(PartialCount#55L),0) AS strCount#45L]
     Exchange (HashPartitioning [str#44], 5), []
      Aggregate true, [str#44], [str#44,COUNT(str#44) AS PartialCount#55L]
       LocalTableScan [str#44], [[1],[2],[3]]
    Aggregate false, [str#51], [str#51]
     Exchange (HashPartitioning [str#51], 5), []
      Aggregate true, [str#51], [str#51]
       LocalTableScan [str#51], [[1],[2],[3]]

Code Generation: false
== RDD ==
== Results ==
!== Correct Answer - 3 ==   == Spark Answer - 3 ==
 [1,1]                      [1,1]
![2,3]                      [2,1]
 [3,1]                      [3,1]

@SparkQA
Copy link

SparkQA commented Apr 29, 2015

Test build #31268 has finished for PR 5774 at commit 5be6f78.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public final class UnsafeFixedWidthAggregationMap
    • public static class MapEntry
    • public final class UnsafeRow implements MutableRow
    • class UnsafeRowConverter(fieldTypes: Array[DataType])
    • public final class PlatformDependent
    • public class ByteArrayMethods
    • public final class LongArray
    • public final class BitSet
    • public final class BitSetMethods
    • public final class Murmur3_x86_32
    • public final class BytesToBytesMap
    • public final class Location
    • class Doubling implements HashMapGrowthStrategy
    • public class ExecutorMemoryManager
    • public class HeapMemoryAllocator implements MemoryAllocator
    • public class MemoryBlock extends MemoryLocation
    • public class MemoryLocation
    • public final class TaskMemoryManager
    • public class UnsafeMemoryAllocator implements MemoryAllocator
  • This patch adds the following new dependencies:
    • spark-unsafe_2.10-1.4.0-SNAPSHOT.jar

@scwf
Copy link
Contributor Author

scwf commented Apr 29, 2015

/cc @rxin

@rxin
Copy link
Contributor

rxin commented Apr 29, 2015

Can you paste an example output from the new change? Thanks.

@scwf
Copy link
Contributor Author

scwf commented Apr 29, 2015

@rxin ,Done

@marmbrus
Copy link
Contributor

LGTM

@rxin
Copy link
Contributor

rxin commented Apr 30, 2015

Thanks. Merging in master.

@asfgit asfgit closed this in 1fdfdb4 Apr 30, 2015
@scwf scwf deleted the checkanswer branch April 30, 2015 00:02
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
…not right

Print detail query execution info including parsed/analyzed/optimized/Physical plan for query when spak answer is not rignt.

```
Results do not match for query:
== Parsed Logical Plan ==
'Aggregate ['x.str], ['x.str,SUM('x.strCount) AS c1#46]
 'Join Inner, Some(('x.str = 'y.str))
  'UnresolvedRelation [df], Some(x)
  'UnresolvedRelation [df], Some(y)

== Analyzed Logical Plan ==
Aggregate [str#44], [str#44,SUM(strCount#45L) AS c1#46L]
 Join Inner, Some((str#44 = str#51))
  Subquery x
   Subquery df
    Aggregate [str#44], [str#44,COUNT(str#44) AS strCount#45L]
     Project [_1#41 AS int#43,_2#42 AS str#44]
      LocalRelation [_1#41,_2#42], [[1,1],[2,2],[3,3]]
  Subquery y
   Subquery df
    Aggregate [str#51], [str#51,COUNT(str#51) AS strCount#47L]
     Project [_1#41 AS int#50,_2#42 AS str#51]
      LocalRelation [_1#41,_2#42], [[1,1],[2,2],[3,3]]

== Optimized Logical Plan ==
Aggregate [str#44], [str#44,SUM(strCount#45L) AS c1#46L]
 Project [str#44,strCount#45L]
  Join Inner, Some((str#44 = str#51))
   Aggregate [str#44], [str#44,COUNT(str#44) AS strCount#45L]
    LocalRelation [str#44], [[1],[2],[3]]
   Aggregate [str#51], [str#51]
    LocalRelation [str#51], [[1],[2],[3]]

== Physical Plan ==
Aggregate false, [str#44], [str#44,CombineSum(PartialSum#53L) AS c1#46L]
 Aggregate true, [str#44], [str#44,SUM(strCount#45L) AS PartialSum#53L]
  Project [str#44,strCount#45L]
   BroadcastHashJoin [str#44], [str#51], BuildRight
    Aggregate false, [str#44], [str#44,Coalesce(SUM(PartialCount#55L),0) AS strCount#45L]
     Exchange (HashPartitioning [str#44], 5), []
      Aggregate true, [str#44], [str#44,COUNT(str#44) AS PartialCount#55L]
       LocalTableScan [str#44], [[1],[2],[3]]
    Aggregate false, [str#51], [str#51]
     Exchange (HashPartitioning [str#51], 5), []
      Aggregate true, [str#51], [str#51]
       LocalTableScan [str#51], [[1],[2],[3]]

Code Generation: false
== RDD ==
== Results ==
!== Correct Answer - 3 ==   == Spark Answer - 3 ==
 [1,1]                      [1,1]
![2,3]                      [2,1]
 [3,1]                      [3,1]
```

Author: wangfei <wangfei1@huawei.com>

Closes apache#5774 from scwf/checkanswer and squashes the following commits:

5be6f78 [wangfei] print detail query execution info when Spark Answer is not right
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
…not right

Print detail query execution info including parsed/analyzed/optimized/Physical plan for query when spak answer is not rignt.

```
Results do not match for query:
== Parsed Logical Plan ==
'Aggregate ['x.str], ['x.str,SUM('x.strCount) AS c1#46]
 'Join Inner, Some(('x.str = 'y.str))
  'UnresolvedRelation [df], Some(x)
  'UnresolvedRelation [df], Some(y)

== Analyzed Logical Plan ==
Aggregate [str#44], [str#44,SUM(strCount#45L) AS c1#46L]
 Join Inner, Some((str#44 = str#51))
  Subquery x
   Subquery df
    Aggregate [str#44], [str#44,COUNT(str#44) AS strCount#45L]
     Project [_1#41 AS int#43,_2#42 AS str#44]
      LocalRelation [_1#41,_2#42], [[1,1],[2,2],[3,3]]
  Subquery y
   Subquery df
    Aggregate [str#51], [str#51,COUNT(str#51) AS strCount#47L]
     Project [_1#41 AS int#50,_2#42 AS str#51]
      LocalRelation [_1#41,_2#42], [[1,1],[2,2],[3,3]]

== Optimized Logical Plan ==
Aggregate [str#44], [str#44,SUM(strCount#45L) AS c1#46L]
 Project [str#44,strCount#45L]
  Join Inner, Some((str#44 = str#51))
   Aggregate [str#44], [str#44,COUNT(str#44) AS strCount#45L]
    LocalRelation [str#44], [[1],[2],[3]]
   Aggregate [str#51], [str#51]
    LocalRelation [str#51], [[1],[2],[3]]

== Physical Plan ==
Aggregate false, [str#44], [str#44,CombineSum(PartialSum#53L) AS c1#46L]
 Aggregate true, [str#44], [str#44,SUM(strCount#45L) AS PartialSum#53L]
  Project [str#44,strCount#45L]
   BroadcastHashJoin [str#44], [str#51], BuildRight
    Aggregate false, [str#44], [str#44,Coalesce(SUM(PartialCount#55L),0) AS strCount#45L]
     Exchange (HashPartitioning [str#44], 5), []
      Aggregate true, [str#44], [str#44,COUNT(str#44) AS PartialCount#55L]
       LocalTableScan [str#44], [[1],[2],[3]]
    Aggregate false, [str#51], [str#51]
     Exchange (HashPartitioning [str#51], 5), []
      Aggregate true, [str#51], [str#51]
       LocalTableScan [str#51], [[1],[2],[3]]

Code Generation: false
== RDD ==
== Results ==
!== Correct Answer - 3 ==   == Spark Answer - 3 ==
 [1,1]                      [1,1]
![2,3]                      [2,1]
 [3,1]                      [3,1]
```

Author: wangfei <wangfei1@huawei.com>

Closes apache#5774 from scwf/checkanswer and squashes the following commits:

5be6f78 [wangfei] print detail query execution info when Spark Answer is not right
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
…not right

Print detail query execution info including parsed/analyzed/optimized/Physical plan for query when spak answer is not rignt.

```
Results do not match for query:
== Parsed Logical Plan ==
'Aggregate ['x.str], ['x.str,SUM('x.strCount) AS c1#46]
 'Join Inner, Some(('x.str = 'y.str))
  'UnresolvedRelation [df], Some(x)
  'UnresolvedRelation [df], Some(y)

== Analyzed Logical Plan ==
Aggregate [str#44], [str#44,SUM(strCount#45L) AS c1#46L]
 Join Inner, Some((str#44 = str#51))
  Subquery x
   Subquery df
    Aggregate [str#44], [str#44,COUNT(str#44) AS strCount#45L]
     Project [_1#41 AS int#43,_2#42 AS str#44]
      LocalRelation [_1#41,_2#42], [[1,1],[2,2],[3,3]]
  Subquery y
   Subquery df
    Aggregate [str#51], [str#51,COUNT(str#51) AS strCount#47L]
     Project [_1#41 AS int#50,_2#42 AS str#51]
      LocalRelation [_1#41,_2#42], [[1,1],[2,2],[3,3]]

== Optimized Logical Plan ==
Aggregate [str#44], [str#44,SUM(strCount#45L) AS c1#46L]
 Project [str#44,strCount#45L]
  Join Inner, Some((str#44 = str#51))
   Aggregate [str#44], [str#44,COUNT(str#44) AS strCount#45L]
    LocalRelation [str#44], [[1],[2],[3]]
   Aggregate [str#51], [str#51]
    LocalRelation [str#51], [[1],[2],[3]]

== Physical Plan ==
Aggregate false, [str#44], [str#44,CombineSum(PartialSum#53L) AS c1#46L]
 Aggregate true, [str#44], [str#44,SUM(strCount#45L) AS PartialSum#53L]
  Project [str#44,strCount#45L]
   BroadcastHashJoin [str#44], [str#51], BuildRight
    Aggregate false, [str#44], [str#44,Coalesce(SUM(PartialCount#55L),0) AS strCount#45L]
     Exchange (HashPartitioning [str#44], 5), []
      Aggregate true, [str#44], [str#44,COUNT(str#44) AS PartialCount#55L]
       LocalTableScan [str#44], [[1],[2],[3]]
    Aggregate false, [str#51], [str#51]
     Exchange (HashPartitioning [str#51], 5), []
      Aggregate true, [str#51], [str#51]
       LocalTableScan [str#51], [[1],[2],[3]]

Code Generation: false
== RDD ==
== Results ==
!== Correct Answer - 3 ==   == Spark Answer - 3 ==
 [1,1]                      [1,1]
![2,3]                      [2,1]
 [3,1]                      [3,1]
```

Author: wangfei <wangfei1@huawei.com>

Closes apache#5774 from scwf/checkanswer and squashes the following commits:

5be6f78 [wangfei] print detail query execution info when Spark Answer is not right
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants