HIVE-24969: Predicates may be removed when decorrelating subqueries with lateral #2145
Conversation
kgyrtkirk
added
tests pending
tests unstable
and removed
tests pending
tests unstable
labels
kgyrtkirk
added
tests pending
tests unstable
and removed
tests unstable
tests pending
labels
kgyrtkirk
added
tests pending
tests passed
and removed
tests unstable
tests pending
labels
dengzhhu653
changed the title
kgyrtkirk
added
tests pending
tests unstable
tests passed
and removed
tests passed
tests pending
tests unstable
labels
| @@ -4769,7 +4769,7 @@ POSTHOOK: Input: default@tempty | |||
| 85768 almond antique chartreuse lavender yellow Manufacturer#1 Brand#12 LARGE BRUSHED STEEL 34 SM BAG 1753.76 refull | |||
| 86428 almond aquamarine burnished black steel Manufacturer#1 Brand#12 STANDARD ANODIZED STEEL 28 WRAP BAG 1414.42 arefully | |||
| 90681 almond antique chartreuse khaki white Manufacturer#3 Brand#31 MEDIUM BURNISHED TIN 17 SM CASE 1671.68 are slyly after the sl | |||
| Warning: Shuffle Join MERGEJOIN[39][tables = [$hdt$_0, $hdt$_1, $hdt$_2]] in Stage 'Reducer 3' is a cross product | |||
| Warning: Shuffle Join MERGEJOIN[39][tables = [$hdt$_0, $hdt$_2]] in Stage 'Reducer 3' is a cross product | |||
There was a problem hiding this comment.
Need to figure out why $hdt$_1 is omited...
There was a problem hiding this comment.
Same as above, the $hdt$_1 does not participate in Reducer 3, as it produces no output after the semi join(p_name IN (select p_name from part_null))
| @@ -246,7 +246,7 @@ POSTHOOK: Input: default@part_null | |||
| 85768 almond antique chartreuse lavender yellow Manufacturer#1 Brand#12 LARGE BRUSHED STEEL 34 SM BAG 1753.76 refull | |||
| 86428 almond aquamarine burnished black steel Manufacturer#1 Brand#12 STANDARD ANODIZED STEEL 28 WRAP BAG 1414.42 arefully | |||
| 90681 almond antique chartreuse khaki white Manufacturer#3 Brand#31 MEDIUM BURNISHED TIN 17 SM CASE 1671.68 are slyly after the sl | |||
| Warning: Shuffle Join MERGEJOIN[57][tables = [$hdt$_0, $hdt$_1, $hdt$_2, $hdt$_3]] in Stage 'Reducer 4' is a cross product | |||
| Warning: Shuffle Join MERGEJOIN[57][tables = [$hdt$_0, $hdt$_2, $hdt$_3]] in Stage 'Reducer 4' is a cross product | |||
There was a problem hiding this comment.
The $hdt$_1 is omitted because $hdt$_0 and $hdt$_1 perform a left semi join, that is p_name IN (select p_name from part_null) in Reducer 2, the $hdt$_1 does not produces output after this join, only $hdt$_0, $hdt$_2(distinct p_brand) and $hdt$_2(count p_name) take place in $hdt$_4.
kgyrtkirk
added
tests pending
tests passed
and removed
tests passed
tests pending
labels
| @@ -0,0 +1,305 @@ | |||
| Warning: Shuffle Join MERGEJOIN[54][tables = [sq_1_notin_nullcheck]] in Stage 'Reducer 3' is a cross product | |||
There was a problem hiding this comment.
One of the parent of Reducer 3 is a union work(alias T), when CrossProductHandler analyzes the union work, as work.getAllRootOperators() returns an empty set, so the inputs of t and lv do not added to the reduce sink info of the join, cause the waring message missing some table aliases.
There was a problem hiding this comment.
Using the MR, the warning message is Warning: Shuffle Join JOIN[34][tables = [lv, t, sq_1_notin_nullcheck]] in Stage 'Stage-1:MAPRED' is a cross product.
|
@kgyrtkirk @jcamachor @vineetgarg02, any thoughts here? |
kgyrtkirk
added
tests pending
tests failed
and removed
tests passed
tests pending
labels
dengzhhu653
force-pushed
the
HIVE-24969
branch
from
7de5d08
to
8890e6b
Compare
kgyrtkirk
added
tests pending
tests unstable
and removed
tests failed
tests pending
labels
kgyrtkirk
added
tests pending
tests passed
and removed
tests unstable
tests pending
labels
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?