[SPARK-17517][SQL]Improve generated Code for BroadcastHashJoinExec #15071

yaooqinn · 2016-09-13T05:45:59Z

What changes were proposed in this pull request?

For current BroadcastHashJoinExec, we generate join code for key is not unique like this:

while (matches.hasnext) {
    matched = matches.next
    check and read stream side row fields
    check and read build side row fieldes
    reset result row
    write stream side row fields to result row
    write build side row fields to result row
    append(result row)
}

For some cases, we don't need to check/read/write the steam side repeatedly in such while circle, e.g. Inner Join with BuildRight, or BuildLeft && all left side fields are fixed length and so on. we may generate the code as below:

check and read stream side row fields
reset result row
write stream side row fields to result row
while (matches.hasnext)
{
    matched = matches.next
    check and read build side row fieldes
    write build side row fields to result row
    append(result row)
}

How was this patch tested?

1. add ut

2. benchmark

   Java HotSpot(TM) 64-Bit Server VM 1.8.0_65-b17 on Mac OS X 10.11.6
    Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz

    BroadcastHashJoin w duplicated keys:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
    ------------------------------------------------------------------------------------------------
    codegen = F                                 13933 / 13974          7.5         132.9       1.0X
    codegen = T avoid = F                          253 /  273        414.0           2.4      55.0X
    codegen = T avoid = T                          202 /  211        518.7           1.9      68.9X

3. manually check

before

/* 105 */     while (scan_batch != null) {
/* 106 */       int numRows = scan_batch.numRows();
/* 107 */       while (scan_batchIdx < numRows) {
/* 108 */         int scan_rowIdx = scan_batchIdx++;
/* 109 */         boolean scan_isNull = scan_colInstance0.isNullAt(scan_rowIdx);
/* 110 */         int scan_value = scan_isNull ? -1 : (scan_colInstance0.getInt(scan_rowIdx));
/* 111 */
/* 112 */         if (!(!(scan_isNull))) continue;
/* 113 */
/* 114 */         boolean filter_isNull2 = false;
/* 115 */
/* 116 */         boolean filter_value2 = false;
/* 117 */         filter_value2 = scan_value == 1;
/* 118 */         if (!filter_value2) continue;
/* 119 */
/* 120 */         filter_numOutputRows.add(1);
/* 121 */
/* 122 */         // generate join key for stream side
/* 123 */
/* 124 */         boolean bhj_isNull = false;
/* 125 */         long bhj_value = -1L;
/* 126 */         if (!false) {
/* 127 */           bhj_value = (long) scan_value;
/* 128 */         }
/* 129 */         // find matches from HashRelation
/* 130 */         scala.collection.Iterator bhj_matches =
/* 131 */         bhj_isNull ? null : (scala.collection.Iterator)bhj_relation.get(bhj_value);
/* 132 */         if (bhj_matches == null) continue;
/* 133 */         while (bhj_matches.hasNext()) {
/* 134 */           UnsafeRow bhj_matched = (UnsafeRow) bhj_matches.next();
/* 135 */
/* 136 */           bhj_numOutputRows.add(1);
/* 137 */
/* 138 */           boolean scan_isNull1 = scan_colInstance1.isNullAt(scan_rowIdx);
/* 139 */           UTF8String scan_value1 = scan_isNull1 ? null : (scan_colInstance1.getUTF8String(scan_rowIdx));
/* 140 */           boolean scan_isNull2 = scan_colInstance2.isNullAt(scan_rowIdx);
/* 141 */           UTF8String scan_value2 = scan_isNull2 ? null : (scan_colInstance2.getUTF8String(scan_rowIdx));
/* 142 */           boolean scan_isNull3 = scan_colInstance3.isNullAt(scan_rowIdx);
/* 143 */           int scan_value3 = scan_isNull3 ? -1 : (scan_colInstance3.getInt(scan_rowIdx));
/* 144 */           boolean scan_isNull4 = scan_colInstance4.isNullAt(scan_rowIdx);
/* 145 */           UTF8String scan_value4 = scan_isNull4 ? null : (scan_colInstance4.getUTF8String(scan_rowIdx));
/* 146 */           boolean scan_isNull5 = scan_colInstance5.isNullAt(scan_rowIdx);
/* 147 */           UTF8String scan_value5 = scan_isNull5 ? null : (scan_colInstance5.getUTF8String(scan_rowIdx));
/* 148 */           boolean scan_isNull6 = scan_colInstance6.isNullAt(scan_rowIdx);
/* 149 */           UTF8String scan_value6 = scan_isNull6 ? null : (scan_colInstance6.getUTF8String(scan_rowIdx));
/* 150 */           boolean scan_isNull7 = scan_colInstance7.isNullAt(scan_rowIdx);
/* 151 */           UTF8String scan_value7 = scan_isNull7 ? null : (scan_colInstance7.getUTF8String(scan_rowIdx));
/* 152 */           boolean scan_isNull8 = scan_colInstance8.isNullAt(scan_rowIdx);
/* 153 */           int scan_value8 = scan_isNull8 ? -1 : (scan_colInstance8.getInt(scan_rowIdx));
/* 154 */           boolean bhj_isNull2 = bhj_matched.isNullAt(0);
/* 155 */           int bhj_value2 = bhj_isNull2 ? -1 : (bhj_matched.getInt(0));
/* 156 */           boolean bhj_isNull3 = bhj_matched.isNullAt(1);
/* 157 */           UTF8String bhj_value3 = bhj_isNull3 ? null : (bhj_matched.getUTF8String(1));
/* 158 */           boolean bhj_isNull4 = bhj_matched.isNullAt(2);
/* 159 */           UTF8String bhj_value4 = bhj_isNull4 ? null : (bhj_matched.getUTF8String(2));
/* 160 */           boolean bhj_isNull5 = bhj_matched.isNullAt(3);
/* 161 */           int bhj_value5 = bhj_isNull5 ? -1 : (bhj_matched.getInt(3));
/* 162 */           boolean bhj_isNull6 = bhj_matched.isNullAt(4);
/* 163 */           UTF8String bhj_value6 = bhj_isNull6 ? null : (bhj_matched.getUTF8String(4));
/* 164 */           boolean bhj_isNull7 = bhj_matched.isNullAt(5);
/* 165 */           UTF8String bhj_value7 = bhj_isNull7 ? null : (bhj_matched.getUTF8String(5));
/* 166 */           boolean bhj_isNull8 = bhj_matched.isNullAt(6);
/* 167 */           UTF8String bhj_value8 = bhj_isNull8 ? null : (bhj_matched.getUTF8String(6));
/* 168 */           boolean bhj_isNull9 = bhj_matched.isNullAt(7);
/* 169 */           UTF8String bhj_value9 = bhj_isNull9 ? null : (bhj_matched.getUTF8String(7));
/* 170 */           boolean bhj_isNull10 = bhj_matched.isNullAt(8);
/* 171 */           int bhj_value10 = bhj_isNull10 ? -1 : (bhj_matched.getInt(8));
/* 172 */           bhj_holder.reset();
/* 173 */
/* 174 */           bhj_rowWriter.zeroOutNullBytes();
/* 175 */
/* 176 */           bhj_rowWriter.write(0, scan_value);
/* 177 */
/* 178 */           if (scan_isNull1) {
/* 179 */             bhj_rowWriter.setNullAt(1);
/* 180 */           } else {
/* 181 */             bhj_rowWriter.write(1, scan_value1);
/* 182 */           }
/* 183 */
/* 184 */           if (scan_isNull2) {
/* 185 */             bhj_rowWriter.setNullAt(2);
/* 186 */           } else {
/* 187 */             bhj_rowWriter.write(2, scan_value2);
/* 188 */           }
/* 189 */
/* 190 */           if (scan_isNull3) {
/* 191 */             bhj_rowWriter.setNullAt(3);
/* 192 */           } else {
/* 193 */             bhj_rowWriter.write(3, scan_value3);
/* 194 */           }
/* 195 */
/* 196 */           if (scan_isNull4) {
/* 197 */             bhj_rowWriter.setNullAt(4);
/* 198 */           } else {
/* 199 */             bhj_rowWriter.write(4, scan_value4);
/* 200 */           }
/* 201 */
/* 202 */           if (scan_isNull5) {
/* 203 */             bhj_rowWriter.setNullAt(5);
/* 204 */           } else {
/* 205 */             bhj_rowWriter.write(5, scan_value5);
/* 206 */           }
/* 207 */
/* 208 */           if (scan_isNull6) {
/* 209 */             bhj_rowWriter.setNullAt(6);
/* 210 */           } else {
/* 211 */             bhj_rowWriter.write(6, scan_value6);
/* 212 */           }
/* 213 */
/* 214 */           if (scan_isNull7) {
/* 215 */             bhj_rowWriter.setNullAt(7);
/* 216 */           } else {
/* 217 */             bhj_rowWriter.write(7, scan_value7);
/* 218 */           }
/* 219 */
/* 220 */           if (scan_isNull8) {
/* 221 */             bhj_rowWriter.setNullAt(8);
/* 222 */           } else {
/* 223 */             bhj_rowWriter.write(8, scan_value8);
/* 224 */           }
/* 225 */
/* 226 */           if (bhj_isNull2) {
/* 227 */             bhj_rowWriter.setNullAt(9);
/* 228 */           } else {
/* 229 */             bhj_rowWriter.write(9, bhj_value2);
/* 230 */           }
/* 231 */
/* 232 */           if (bhj_isNull3) {
/* 233 */             bhj_rowWriter.setNullAt(10);
/* 234 */           } else {
/* 235 */             bhj_rowWriter.write(10, bhj_value3);
/* 236 */           }
/* 237 */
/* 238 */           if (bhj_isNull4) {
/* 239 */             bhj_rowWriter.setNullAt(11);
/* 240 */           } else {
/* 241 */             bhj_rowWriter.write(11, bhj_value4);
/* 242 */           }
/* 243 */
/* 244 */           if (bhj_isNull5) {
/* 245 */             bhj_rowWriter.setNullAt(12);
/* 246 */           } else {
/* 247 */             bhj_rowWriter.write(12, bhj_value5);
/* 248 */           }
/* 249 */
/* 250 */           if (bhj_isNull6) {
/* 251 */             bhj_rowWriter.setNullAt(13);
/* 252 */           } else {
/* 253 */             bhj_rowWriter.write(13, bhj_value6);
/* 254 */           }
/* 255 */
/* 256 */           if (bhj_isNull7) {
/* 257 */             bhj_rowWriter.setNullAt(14);
/* 258 */           } else {
/* 259 */             bhj_rowWriter.write(14, bhj_value7);
/* 260 */           }
/* 261 */
/* 262 */           if (bhj_isNull8) {
/* 263 */             bhj_rowWriter.setNullAt(15);
/* 264 */           } else {
/* 265 */             bhj_rowWriter.write(15, bhj_value8);
/* 266 */           }
/* 267 */
/* 268 */           if (bhj_isNull9) {
/* 269 */             bhj_rowWriter.setNullAt(16);
/* 270 */           } else {
/* 271 */             bhj_rowWriter.write(16, bhj_value9);
/* 272 */           }
/* 273 */
/* 274 */           if (bhj_isNull10) {
/* 275 */             bhj_rowWriter.setNullAt(17);
/* 276 */           } else {
/* 277 */             bhj_rowWriter.write(17, bhj_value10);
/* 278 */           }
/* 279 */           bhj_result.setTotalSize(bhj_holder.totalSize());
/* 280 */
/* 281 */           append(bhj_result.copy());
/* 282 */
/* 283 */         }
/* 284 */         if (shouldStop()) return;
/* 285 */       }
/* 286 */       scan_batch = null;
/* 287 */       scan_nextBatch();
/* 288 */     }

after

/* 105 */     while (scan_batch != null) {
/* 106 */       int numRows = scan_batch.numRows();
/* 107 */       while (scan_batchIdx < numRows) {
/* 108 */         int scan_rowIdx = scan_batchIdx++;
/* 109 */         boolean scan_isNull = scan_colInstance0.isNullAt(scan_rowIdx);
/* 110 */         int scan_value = scan_isNull ? -1 : (scan_colInstance0.getInt(scan_rowIdx));
/* 111 */
/* 112 */         if (!(!(scan_isNull))) continue;
/* 113 */
/* 114 */         boolean filter_isNull2 = false;
/* 115 */
/* 116 */         boolean filter_value2 = false;
/* 117 */         filter_value2 = scan_value == 1;
/* 118 */         if (!filter_value2) continue;
/* 119 */
/* 120 */         filter_numOutputRows.add(1);
/* 121 */
/* 122 */         // generate join key for stream side
/* 123 */
/* 124 */         boolean bhj_isNull = false;
/* 125 */         long bhj_value = -1L;
/* 126 */         if (!false) {
/* 127 */           bhj_value = (long) scan_value;
/* 128 */         }
/* 129 */         // find matches from HashRelation
/* 130 */         scala.collection.Iterator bhj_matches =
/* 131 */         bhj_isNull ? null : (scala.collection.Iterator)bhj_relation.get(bhj_value);
/* 132 */         if (bhj_matches == null) continue;
/* 133 */
/* 134 */         boolean scan_isNull1 = scan_colInstance1.isNullAt(scan_rowIdx);
/* 135 */         UTF8String scan_value1 = scan_isNull1 ? null : (scan_colInstance1.getUTF8String(scan_rowIdx));
/* 136 */         boolean scan_isNull2 = scan_colInstance2.isNullAt(scan_rowIdx);
/* 137 */         UTF8String scan_value2 = scan_isNull2 ? null : (scan_colInstance2.getUTF8String(scan_rowIdx));
/* 138 */         boolean scan_isNull3 = scan_colInstance3.isNullAt(scan_rowIdx);
/* 139 */         int scan_value3 = scan_isNull3 ? -1 : (scan_colInstance3.getInt(scan_rowIdx));
/* 140 */         boolean scan_isNull4 = scan_colInstance4.isNullAt(scan_rowIdx);
/* 141 */         UTF8String scan_value4 = scan_isNull4 ? null : (scan_colInstance4.getUTF8String(scan_rowIdx));
/* 142 */         boolean scan_isNull5 = scan_colInstance5.isNullAt(scan_rowIdx);
/* 143 */         UTF8String scan_value5 = scan_isNull5 ? null : (scan_colInstance5.getUTF8String(scan_rowIdx));
/* 144 */         boolean scan_isNull6 = scan_colInstance6.isNullAt(scan_rowIdx);
/* 145 */         UTF8String scan_value6 = scan_isNull6 ? null : (scan_colInstance6.getUTF8String(scan_rowIdx));
/* 146 */         boolean scan_isNull7 = scan_colInstance7.isNullAt(scan_rowIdx);
/* 147 */         UTF8String scan_value7 = scan_isNull7 ? null : (scan_colInstance7.getUTF8String(scan_rowIdx));
/* 148 */         boolean scan_isNull8 = scan_colInstance8.isNullAt(scan_rowIdx);
/* 149 */         int scan_value8 = scan_isNull8 ? -1 : (scan_colInstance8.getInt(scan_rowIdx));
/* 150 */         bhj_holder.reset();
/* 151 */         bhj_rowWriter.zeroOutNullBytes();
/* 152 */
/* 153 */         bhj_rowWriter.write(0, scan_value);
/* 154 */
/* 155 */         if (scan_isNull1) {
/* 156 */           bhj_rowWriter.setNullAt(1);
/* 157 */         } else {
/* 158 */           bhj_rowWriter.write(1, scan_value1);
/* 159 */         }
/* 160 */
/* 161 */         if (scan_isNull2) {
/* 162 */           bhj_rowWriter.setNullAt(2);
/* 163 */         } else {
/* 164 */           bhj_rowWriter.write(2, scan_value2);
/* 165 */         }
/* 166 */
/* 167 */         if (scan_isNull3) {
/* 168 */           bhj_rowWriter.setNullAt(3);
/* 169 */         } else {
/* 170 */           bhj_rowWriter.write(3, scan_value3);
/* 171 */         }
/* 172 */
/* 173 */         if (scan_isNull4) {
/* 174 */           bhj_rowWriter.setNullAt(4);
/* 175 */         } else {
/* 176 */           bhj_rowWriter.write(4, scan_value4);
/* 177 */         }
/* 178 */
/* 179 */         if (scan_isNull5) {
/* 180 */           bhj_rowWriter.setNullAt(5);
/* 181 */         } else {
/* 182 */           bhj_rowWriter.write(5, scan_value5);
/* 183 */         }
/* 184 */
/* 185 */         if (scan_isNull6) {
/* 186 */           bhj_rowWriter.setNullAt(6);
/* 187 */         } else {
/* 188 */           bhj_rowWriter.write(6, scan_value6);
/* 189 */         }
/* 190 */
/* 191 */         if (scan_isNull7) {
/* 192 */           bhj_rowWriter.setNullAt(7);
/* 193 */         } else {
/* 194 */           bhj_rowWriter.write(7, scan_value7);
/* 195 */         }
/* 196 */
/* 197 */         if (scan_isNull8) {
/* 198 */           bhj_rowWriter.setNullAt(8);
/* 199 */         } else {
/* 200 */           bhj_rowWriter.write(8, scan_value8);
/* 201 */         }
/* 202 */
/* 203 */         while (bhj_matches.hasNext()) {
/* 204 */           UnsafeRow bhj_matched = (UnsafeRow) bhj_matches.next();
/* 205 */
/* 206 */           bhj_numOutputRows.add(1);
/* 207 */
/* 208 */           boolean bhj_isNull2 = bhj_matched.isNullAt(0);
/* 209 */           int bhj_value2 = bhj_isNull2 ? -1 : (bhj_matched.getInt(0));
/* 210 */           boolean bhj_isNull3 = bhj_matched.isNullAt(1);
/* 211 */           UTF8String bhj_value3 = bhj_isNull3 ? null : (bhj_matched.getUTF8String(1));
/* 212 */           boolean bhj_isNull4 = bhj_matched.isNullAt(2);
/* 213 */           UTF8String bhj_value4 = bhj_isNull4 ? null : (bhj_matched.getUTF8String(2));
/* 214 */           boolean bhj_isNull5 = bhj_matched.isNullAt(3);
/* 215 */           int bhj_value5 = bhj_isNull5 ? -1 : (bhj_matched.getInt(3));
/* 216 */           boolean bhj_isNull6 = bhj_matched.isNullAt(4);
/* 217 */           UTF8String bhj_value6 = bhj_isNull6 ? null : (bhj_matched.getUTF8String(4));
/* 218 */           boolean bhj_isNull7 = bhj_matched.isNullAt(5);
/* 219 */           UTF8String bhj_value7 = bhj_isNull7 ? null : (bhj_matched.getUTF8String(5));
/* 220 */           boolean bhj_isNull8 = bhj_matched.isNullAt(6);
/* 221 */           UTF8String bhj_value8 = bhj_isNull8 ? null : (bhj_matched.getUTF8String(6));
/* 222 */           boolean bhj_isNull9 = bhj_matched.isNullAt(7);
/* 223 */           UTF8String bhj_value9 = bhj_isNull9 ? null : (bhj_matched.getUTF8String(7));
/* 224 */           boolean bhj_isNull10 = bhj_matched.isNullAt(8);
/* 225 */           int bhj_value10 = bhj_isNull10 ? -1 : (bhj_matched.getInt(8));
/* 226 */           if (bhj_isNull2) {
/* 227 */             bhj_rowWriter.setNullAt(9);
/* 228 */           } else {
/* 229 */             bhj_rowWriter.write(9, bhj_value2);
/* 230 */           }
/* 231 */
/* 232 */           if (bhj_isNull3) {
/* 233 */             bhj_rowWriter.setNullAt(10);
/* 234 */           } else {
/* 235 */             bhj_rowWriter.write(10, bhj_value3);
/* 236 */           }
/* 237 */
/* 238 */           if (bhj_isNull4) {
/* 239 */             bhj_rowWriter.setNullAt(11);
/* 240 */           } else {
/* 241 */             bhj_rowWriter.write(11, bhj_value4);
/* 242 */           }
/* 243 */
/* 244 */           if (bhj_isNull5) {
/* 245 */             bhj_rowWriter.setNullAt(12);
/* 246 */           } else {
/* 247 */             bhj_rowWriter.write(12, bhj_value5);
/* 248 */           }
/* 249 */
/* 250 */           if (bhj_isNull6) {
/* 251 */             bhj_rowWriter.setNullAt(13);
/* 252 */           } else {
/* 253 */             bhj_rowWriter.write(13, bhj_value6);
/* 254 */           }
/* 255 */
/* 256 */           if (bhj_isNull7) {
/* 257 */             bhj_rowWriter.setNullAt(14);
/* 258 */           } else {
/* 259 */             bhj_rowWriter.write(14, bhj_value7);
/* 260 */           }
/* 261 */
/* 262 */           if (bhj_isNull8) {
/* 263 */             bhj_rowWriter.setNullAt(15);
/* 264 */           } else {
/* 265 */             bhj_rowWriter.write(15, bhj_value8);
/* 266 */           }
/* 267 */
/* 268 */           if (bhj_isNull9) {
/* 269 */             bhj_rowWriter.setNullAt(16);
/* 270 */           } else {
/* 271 */             bhj_rowWriter.write(16, bhj_value9);
/* 272 */           }
/* 273 */
/* 274 */           if (bhj_isNull10) {
/* 275 */             bhj_rowWriter.setNullAt(17);
/* 276 */           } else {
/* 277 */             bhj_rowWriter.write(17, bhj_value10);
/* 278 */           }
/* 279 */           bhj_result.setTotalSize(bhj_holder.totalSize());
/* 280 */
/* 281 */           append(bhj_result.copy());
/* 282 */
/* 283 */         }
/* 284 */         if (shouldStop()) return;
/* 285 */       }
/* 286 */       scan_batch = null;
/* 287 */       scan_nextBatch();
/* 288 */     }

hvanhovell · 2016-09-13T08:33:30Z

What is the improvement?

yaooqinn · 2016-09-13T11:09:03Z

@hvanhovell

For current BroadcastHashJoinExec, we generate join code for key is not unique like this:

while (matches.hasnext) {
    matched = matches.next
    check and read stream side row fields
    check and read build side row fieldes
    reset result row
    write stream side row fields to result row
    write build side row fields to result row
    append(result row)
}

For some cases, we don't need to check/read/write the steam side repeatedly in such while circle, e.g. Inner Join with BuildRight, or BuildLeft && all left side fields are fixed length and so on. we may generate the code as below:

check and read stream side row fields
reset result row
write stream side row fields to result row
while (matches.hasnext)
{
    matched = matches.next
    check and read build side row fieldes
    write build side row fields to result row
    append(result row)
}

hvanhovell · 2016-09-13T12:00:11Z

Ok, how about you add your comment to the description?

The JVM is pretty clever during C2, so this might not yield as much as one might think. Could you provide some performance benchmarks?

yaooqinn · 2016-09-13T12:09:05Z

@hvanhovell thanks very much for your suggestions. I have added my comments to the description, and I will run benchmark later to see if it works

yaooqinn · 2016-09-19T14:37:58Z

@hvanhovell I have added a benchmark test for this, could you please help me to review? thanks.

hvanhovell · 2016-09-19T21:05:26Z

...ore/src/test/scala/org/apache/spark/sql/execution/benchmark/BroadcastHashJoinBenchmark.scala

+    BroadcastHashJoin w duplicated keys:     Best/Avg Time(ms)    Rate(M/s)   Per Row(ns)   Relative
+    ------------------------------------------------------------------------------------------------
+    codegen = F                                    226 /  344          0.3        3447.1       1.0X
+    codegen = T avoid = F                          168 /  173          0.4        2565.2       1.3X


Could you make these benchmarks bigger? I think you are mainly measuring all sorts of initialization slow downs here.

Check this blog post, for some context on these figures: https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html

OK，maybe now it is big enough

yaooqinn · 2016-09-21T07:05:52Z

ping @hvanhovell

hvanhovell · 2016-09-21T23:04:45Z

@yaooqinn why do the fields have to be fixed length for this to work?

yaooqinn · 2016-09-22T01:43:07Z

@hvanhovell
I think unfixed length fields may lead to memory overlapping when BuildLeft, since we are reusing the BufferHolder to avoid writing the stream side repeatedly. In this case, the holder can not grow properly to avoid the left side overlap the right side.

When BuildRight, there is no such a problem.

yaooqinn · 2016-09-24T02:46:24Z

cc @davies

SparkQA · 2016-12-10T16:55:16Z

Test build #3489 has finished for PR 15071 at commit 3e9c790.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

maropu · 2018-07-23T15:11:09Z

@yaooqinn Can you close this because it's not long time for a long time.

yaooqinn added 2 commits September 13, 2016 13:42

init commit

bb731f2

modify for better understanding

88584a7

yaooqinn added 6 commits September 13, 2016 23:13

fix bug - evaluted code evaluated again

6fdcbc7

bug fix for satisfide condition

8b8e939

happy moon cake day

2bff285

add ut and fix some bugs

6b772a2

support outer

5bef2fb

add benchmark test

5c7d074

yaooqinn changed the title ~~[WIP][SPARK-17517][SQL]Improve generated Code for BroadcastHashJoinExec~~ [SPARK-17517][SQL]Improve generated Code for BroadcastHashJoinExec Sep 19, 2016

update benchmark result

bd8e9e5

hvanhovell reviewed Sep 19, 2016

View reviewed changes

yaooqinn added 4 commits September 20, 2016 15:40

enlarger benchmark

dc669ed

add benchmark result

c330eb0

rerun benchmark

46d27bd

ignore benchmark

3e9c790

yaooqinn closed this Jul 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-17517][SQL]Improve generated Code for BroadcastHashJoinExec #15071

[SPARK-17517][SQL]Improve generated Code for BroadcastHashJoinExec #15071

yaooqinn commented Sep 13, 2016 •

edited

hvanhovell commented Sep 13, 2016

yaooqinn commented Sep 13, 2016 •

edited

hvanhovell commented Sep 13, 2016

yaooqinn commented Sep 13, 2016

yaooqinn commented Sep 19, 2016

hvanhovell Sep 19, 2016

hvanhovell Sep 19, 2016

yaooqinn Sep 20, 2016

yaooqinn commented Sep 21, 2016

hvanhovell commented Sep 21, 2016

yaooqinn commented Sep 22, 2016

yaooqinn commented Sep 24, 2016

SparkQA commented Dec 10, 2016

maropu commented Jul 23, 2018

[SPARK-17517][SQL]Improve generated Code for BroadcastHashJoinExec #15071

[SPARK-17517][SQL]Improve generated Code for BroadcastHashJoinExec #15071

Conversation

yaooqinn commented Sep 13, 2016 • edited

What changes were proposed in this pull request?

How was this patch tested?

1. add ut

2. benchmark

3. manually check

before

after

hvanhovell commented Sep 13, 2016

yaooqinn commented Sep 13, 2016 • edited

hvanhovell commented Sep 13, 2016

yaooqinn commented Sep 13, 2016

yaooqinn commented Sep 19, 2016

hvanhovell Sep 19, 2016

Choose a reason for hiding this comment

hvanhovell Sep 19, 2016

Choose a reason for hiding this comment

yaooqinn Sep 20, 2016

Choose a reason for hiding this comment

yaooqinn commented Sep 21, 2016

hvanhovell commented Sep 21, 2016

yaooqinn commented Sep 22, 2016

yaooqinn commented Sep 24, 2016

SparkQA commented Dec 10, 2016

maropu commented Jul 23, 2018

yaooqinn commented Sep 13, 2016 •

edited

yaooqinn commented Sep 13, 2016 •

edited