New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test `named edges` breaks codegen with Spark 2.1 #165

Closed
thunterdb opened this Issue Jan 9, 2017 · 4 comments

Comments

Projects
None yet
3 participants
@thunterdb
Contributor

thunterdb commented Jan 9, 2017

Here is the stack trace:

16/12/30 07:06:33 WARN WholeStageCodegenExec: Whole-stage codegen disabled for this plan:
 *HashAggregate(keys=[e#18378, v#18381], functions=[], output=[e#18378, v#18381])
+- *BroadcastHashJoin [coalesce(e#18378, [0,0,]), coalesce(v#18381, [0,,])], [coalesce(e#18417, [0,0,]), coalesce(v#18419, [0,,])], LeftAnti, BuildRight, ((e#18378 <=> e#18417) && (v#18381 <=> v#18419))
   :- *BroadcastHashJoin [e#18378.dst], [v#18381.id], LeftOuter, BuildRight
   :  :- LocalTableScan [e#18378]
   :  +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, struct<id:bigint,attr:string,gender:string>, false].id))
   :     +- LocalTableScan [v#18381]
   +- BroadcastExchange HashedRelationBroadcastMode(List(coalesce(input[0, struct<src:bigint,dst:bigint,relationship:string>, false], [0,0,]), coalesce(input[1, struct<id:bigint,attr:string,gender:string>, true], [0,,])))
      +- *Project [e#18417, v#18419]
         +- *BroadcastHashJoin [v#18419.id], [__tmp-1042178570564257691#18396.src], Inner, BuildRight
            :- *BroadcastHashJoin [e#18417.dst], [v#18419.id], LeftOuter, BuildRight
            :  :- LocalTableScan [e#18417]
            :  +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, struct<id:bigint,attr:string,gender:string>, false].id))
            :     +- LocalTableScan [v#18419]
            +- BroadcastExchange HashedRelationBroadcastMode(List(input[0, struct<src:bigint,dst:bigint,relationship:string>, false].src))
               +- LocalTableScan [__tmp-1042178570564257691#18396]

and the complete message:

16/12/30 07:06:33 ERROR CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 472, Column 36: Expression "inputadapter_value" is not an rvalue
/* 001 */ public Object generate(Object[] references) {
/* 002 */   return new GeneratedIterator(references);
/* 003 */ }
/* 004 */
/* 005 */ final class GeneratedIterator extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 006 */   private Object[] references;
/* 007 */   private scala.collection.Iterator[] inputs;
/* 008 */   private boolean agg_initAgg;
/* 009 */   private org.apache.spark.sql.execution.aggregate.HashAggregateExec agg_plan;
/* 010 */   private org.apache.spark.sql.execution.UnsafeFixedWidthAggregationMap agg_hashMap;
/* 011 */   private org.apache.spark.sql.execution.UnsafeKVExternalSorter agg_sorter;
/* 012 */   private org.apache.spark.unsafe.KVIterator agg_mapIter;
/* 013 */   private org.apache.spark.sql.execution.metric.SQLMetric agg_peakMemory;
/* 014 */   private org.apache.spark.sql.execution.metric.SQLMetric agg_spillSize;
/* 015 */   private scala.collection.Iterator inputadapter_input;
/* 016 */   private org.apache.spark.broadcast.TorrentBroadcast bhj_broadcast;
/* 017 */   private org.apache.spark.sql.execution.joins.LongHashedRelation bhj_relation;
/* 018 */   private org.apache.spark.sql.execution.metric.SQLMetric bhj_numOutputRows;
/* 019 */   private UnsafeRow bhj_result;
/* 020 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder bhj_holder;
/* 021 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter bhj_rowWriter;
/* 022 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter bhj_rowWriter1;
/* 023 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter bhj_rowWriter2;
/* 024 */   private org.apache.spark.broadcast.TorrentBroadcast bhj_broadcast1;
/* 025 */   private org.apache.spark.sql.execution.joins.UnsafeHashedRelation bhj_relation1;
/* 026 */   private UnsafeRow bhj_result1;
/* 027 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder bhj_holder1;
/* 028 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter bhj_rowWriter3;
/* 029 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter bhj_rowWriter4;
/* 030 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter bhj_rowWriter5;
/* 031 */   private org.apache.spark.sql.execution.metric.SQLMetric bhj_numOutputRows1;
/* 032 */   private UnsafeRow bhj_result2;
/* 033 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder bhj_holder2;
/* 034 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter bhj_rowWriter6;
/* 035 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter bhj_rowWriter7;
/* 036 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter bhj_rowWriter8;
/* 037 */   private UnsafeRow agg_result;
/* 038 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder agg_holder;
/* 039 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter agg_rowWriter;
/* 040 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter agg_rowWriter1;
/* 041 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter agg_rowWriter2;
/* 042 */   private int agg_value4;
/* 043 */   private UnsafeRow agg_result1;
/* 044 */   private org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder agg_holder1;
/* 045 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter agg_rowWriter3;
/* 046 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter agg_rowWriter4;
/* 047 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter agg_rowWriter5;
/* 048 */   private org.apache.spark.sql.execution.metric.SQLMetric wholestagecodegen_numOutputRows;
/* 049 */   private org.apache.spark.sql.execution.metric.SQLMetric wholestagecodegen_aggTime;
/* 050 */
/* 051 */   public GeneratedIterator(Object[] references) {
/* 052 */     this.references = references;
/* 053 */   }
/* 054 */
/* 055 */   public void init(int index, scala.collection.Iterator[] inputs) {
/* 056 */     partitionIndex = index;
/* 057 */     this.inputs = inputs;
/* 058 */     wholestagecodegen_init_0();
/* 059 */     wholestagecodegen_init_1();
/* 060 */     wholestagecodegen_init_2();
/* 061 */     wholestagecodegen_init_3();
/* 062 */
/* 063 */   }
/* 064 */
/* 065 */   private void wholestagecodegen_init_0() {
/* 066 */     agg_initAgg = false;
/* 067 */     this.agg_plan = (org.apache.spark.sql.execution.aggregate.HashAggregateExec) references[0];
/* 068 */
/* 069 */     this.agg_peakMemory = (org.apache.spark.sql.execution.metric.SQLMetric) references[1];
/* 070 */     this.agg_spillSize = (org.apache.spark.sql.execution.metric.SQLMetric) references[2];
/* 071 */     inputadapter_input = inputs[0];
/* 072 */     this.bhj_broadcast = (org.apache.spark.broadcast.TorrentBroadcast) references[3];
/* 073 */
/* 074 */     bhj_relation = ((org.apache.spark.sql.execution.joins.LongHashedRelation) bhj_broadcast.value()).asReadOnlyCopy();
/* 075 */     incPeakExecutionMemory(bhj_relation.estimatedSize());
/* 076 */
/* 077 */     this.bhj_numOutputRows = (org.apache.spark.sql.execution.metric.SQLMetric) references[4];
/* 078 */     bhj_result = new UnsafeRow(2);
/* 079 */     this.bhj_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(bhj_result, 64);
/* 080 */     this.bhj_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(bhj_holder, 2);
/* 081 */     this.bhj_rowWriter1 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(bhj_holder, 3);
/* 082 */     this.bhj_rowWriter2 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(bhj_holder, 3);
/* 083 */
/* 084 */   }
/* 085 */
/* 086 */   private void wholestagecodegen_init_3() {
/* 087 */     this.agg_rowWriter5 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_holder1, 3);
/* 088 */     this.wholestagecodegen_numOutputRows = (org.apache.spark.sql.execution.metric.SQLMetric) references[9];
/* 089 */     this.wholestagecodegen_aggTime = (org.apache.spark.sql.execution.metric.SQLMetric) references[10];
/* 090 */
/* 091 */   }
/* 092 */
/* 093 */   private void agg_doAggregateWithKeys() throws java.io.IOException {
/* 094 */     agg_hashMap = agg_plan.createHashMap();
/* 095 */
/* 096 */     while (inputadapter_input.hasNext()) {
/* 097 */       InternalRow inputadapter_row = (InternalRow) inputadapter_input.next();
/* 098 */       InternalRow inputadapter_value = inputadapter_row.getStruct(0, 3);
/* 099 */
/* 100 */       // generate join key for stream side
/* 101 */
/* 102 */       boolean bhj_isNull = false;
/* 103 */
/* 104 */       long bhj_value = -1L;
/* 105 */
/* 106 */       bhj_value = inputadapter_value.getLong(1);
/* 107 */       // find matches from HashedRelation
/* 108 */       UnsafeRow bhj_matched = false ? null: (UnsafeRow)bhj_relation.getValue(bhj_value);
/* 109 */       final boolean bhj_conditionPassed = true;
/* 110 */       if (!bhj_conditionPassed) {
/* 111 */         bhj_matched = null;
/* 112 */         // reset the variables those are already evaluated.
/* 113 */
/* 114 */       }
/* 115 */       bhj_numOutputRows.add(1);
/* 116 */
/* 117 */       boolean bhj_isNull3 = true;
/* 118 */       InternalRow bhj_value3 = null;
/* 119 */       if (bhj_matched != null) {
/* 120 */         InternalRow bhj_value2 = bhj_matched.getStruct(0, 3);
/* 121 */         bhj_isNull3 = false;
/* 122 */         bhj_value3 = bhj_value2;
/* 123 */       }
/* 124 */
/* 125 */       // generate join key for stream side
/* 126 */
/* 127 */       bhj_holder1.reset();
/* 128 */
/* 129 */       bhj_rowWriter3.zeroOutNullBytes();
/* 130 */
/* 131 */       boolean bhj_isNull6 = false;
/* 132 */       InternalRow bhj_value6 = inputadapter_value;
/* 133 */       if (bhj_isNull6) {
/* 134 */         Object bhj_obj = ((Expression) references[6]).eval(null);
/* 135 */         InternalRow bhj_value8 = (InternalRow) bhj_obj;
/* 136 */         if (!false) {
/* 137 */           bhj_isNull6 = false;
/* 138 */           bhj_value6 = bhj_value8;
/* 139 */         }
/* 140 */       }
/* 141 */       if (bhj_isNull6) {
/* 142 */         bhj_rowWriter3.setNullAt(0);
/* 143 */       } else {
/* 144 */         // Remember the current cursor so that we can calculate how many bytes are
/* 145 */         // written later.
/* 146 */         final int bhj_tmpCursor8 = bhj_holder1.cursor;
/* 147 */
/* 148 */         if (bhj_value6 instanceof UnsafeRow) {
/* 149 */           final int bhj_sizeInBytes2 = ((UnsafeRow) bhj_value6).getSizeInBytes();
/* 150 */           // grow the global buffer before writing data.
/* 151 */           bhj_holder1.grow(bhj_sizeInBytes2);
/* 152 */           ((UnsafeRow) bhj_value6).writeToMemory(bhj_holder1.buffer, bhj_holder1.cursor);
/* 153 */           bhj_holder1.cursor += bhj_sizeInBytes2;
/* 154 */
/* 155 */         } else {
/* 156 */           bhj_rowWriter4.reset();
/* 157 */
/* 158 */           final long bhj_fieldName6 = bhj_value6.getLong(0);
/* 159 */           if (bhj_value6.isNullAt(0)) {
/* 160 */             bhj_rowWriter4.setNullAt(0);
/* 161 */           } else {
/* 162 */             bhj_rowWriter4.write(0, bhj_fieldName6);
/* 163 */           }
/* 164 */
/* 165 */           final long bhj_fieldName7 = bhj_value6.getLong(1);
/* 166 */           if (bhj_value6.isNullAt(1)) {
/* 167 */             bhj_rowWriter4.setNullAt(1);
/* 168 */           } else {
/* 169 */             bhj_rowWriter4.write(1, bhj_fieldName7);
/* 170 */           }
/* 171 */
/* 172 */           final UTF8String bhj_fieldName8 = bhj_value6.getUTF8String(2);
/* 173 */           if (bhj_value6.isNullAt(2)) {
/* 174 */             bhj_rowWriter4.setNullAt(2);
/* 175 */           } else {
/* 176 */             bhj_rowWriter4.write(2, bhj_fieldName8);
/* 177 */           }
/* 178 */         }
/* 179 */
/* 180 */         bhj_rowWriter3.setOffsetAndSize(0, bhj_tmpCursor8, bhj_holder1.cursor - bhj_tmpCursor8);
/* 181 */       }
/* 182 */
/* 183 */       boolean bhj_isNull9 = bhj_isNull3;
/* 184 */       InternalRow bhj_value9 = bhj_value3;
/* 185 */       if (bhj_isNull9) {
/* 186 */         Object bhj_obj1 = ((Expression) references[7]).eval(null);
/* 187 */         InternalRow bhj_value11 = (InternalRow) bhj_obj1;
/* 188 */         if (!false) {
/* 189 */           bhj_isNull9 = false;
/* 190 */           bhj_value9 = bhj_value11;
/* 191 */         }
/* 192 */       }
/* 193 */       if (bhj_isNull9) {
/* 194 */         bhj_rowWriter3.setNullAt(1);
/* 195 */       } else {
/* 196 */         // Remember the current cursor so that we can calculate how many bytes are
/* 197 */         // written later.
/* 198 */         final int bhj_tmpCursor12 = bhj_holder1.cursor;
/* 199 */
/* 200 */         if (bhj_value9 instanceof UnsafeRow) {
/* 201 */           final int bhj_sizeInBytes3 = ((UnsafeRow) bhj_value9).getSizeInBytes();
/* 202 */           // grow the global buffer before writing data.
/* 203 */           bhj_holder1.grow(bhj_sizeInBytes3);
/* 204 */           ((UnsafeRow) bhj_value9).writeToMemory(bhj_holder1.buffer, bhj_holder1.cursor);
/* 205 */           bhj_holder1.cursor += bhj_sizeInBytes3;
/* 206 */
/* 207 */         } else {
/* 208 */           bhj_rowWriter5.reset();
/* 209 */
/* 210 */           final long bhj_fieldName9 = bhj_value9.getLong(0);
/* 211 */           if (bhj_value9.isNullAt(0)) {
/* 212 */             bhj_rowWriter5.setNullAt(0);
/* 213 */           } else {
/* 214 */             bhj_rowWriter5.write(0, bhj_fieldName9);
/* 215 */           }
/* 216 */
/* 217 */           final UTF8String bhj_fieldName10 = bhj_value9.getUTF8String(1);
/* 218 */           if (bhj_value9.isNullAt(1)) {
/* 219 */             bhj_rowWriter5.setNullAt(1);
/* 220 */           } else {
/* 221 */             bhj_rowWriter5.write(1, bhj_fieldName10);
/* 222 */           }
/* 223 */
/* 224 */           final UTF8String bhj_fieldName11 = bhj_value9.getUTF8String(2);
/* 225 */           if (bhj_value9.isNullAt(2)) {
/* 226 */             bhj_rowWriter5.setNullAt(2);
/* 227 */           } else {
/* 228 */             bhj_rowWriter5.write(2, bhj_fieldName11);
/* 229 */           }
/* 230 */         }
/* 231 */
/* 232 */         bhj_rowWriter3.setOffsetAndSize(1, bhj_tmpCursor12, bhj_holder1.cursor - bhj_tmpCursor12);
/* 233 */       }
/* 234 */       bhj_result1.setTotalSize(bhj_holder1.totalSize());
/* 235 */
/* 236 */       // Check if the key has nulls.
/* 237 */       if (!(bhj_result1.anyNull())) {
/* 238 */         // Check if the HashedRelation exists.
/* 239 */         scala.collection.Iterator bhj_matches = (scala.collection.Iterator)bhj_relation1.get(bhj_result1);
/* 240 */         if (bhj_matches != null) {
/* 241 */           // Evaluate the condition.
/* 242 */           boolean bhj_found = false;
/* 243 */           while (!bhj_found && bhj_matches.hasNext()) {
/* 244 */             UnsafeRow bhj_matched1 = (UnsafeRow) bhj_matches.next();
/* 245 */
/* 246 */             boolean bhj_isNull13 = true;
/* 247 */             InternalRow bhj_value13 = null;
/* 248 */             if (bhj_matched1 != null) {
/* 249 */               InternalRow bhj_value12 = bhj_matched1.getStruct(0, 3);
/* 250 */               bhj_isNull13 = false;
/* 251 */               bhj_value13 = bhj_value12;
/* 252 */             }
/* 253 */             boolean bhj_isNull15 = true;
/* 254 */             InternalRow bhj_value15 = null;
/* 255 */             if (bhj_matched1 != null) {
/* 256 */               boolean bhj_isNull14 = bhj_matched1.isNullAt(1);
/* 257 */               InternalRow bhj_value14 = bhj_isNull14 ? null : (bhj_matched1.getStruct(1, 3));
/* 258 */               bhj_isNull15 = bhj_isNull14;
/* 259 */               bhj_value15 = bhj_value14;
/* 260 */             }
/* 261 */
/* 262 */             boolean bhj_value17 = (false && bhj_isNull13) ||
/* 263 */             (!false && !bhj_isNull13 && this.bhj_compareStruct(inputadapter_value, bhj_value13) == 0);
/* 264 */             boolean bhj_value16 = false;
/* 265 */
/* 266 */             if (bhj_value17) {
/* 267 */               boolean bhj_value23 = (bhj_isNull3 && bhj_isNull15) ||
/* 268 */               (!bhj_isNull3 && !bhj_isNull15 && this.bhj_compareStruct1(bhj_value3, bhj_value15) == 0);
/* 269 */               bhj_value16 = bhj_value23;
/* 270 */             }
/* 271 */             if (false || !bhj_value16) continue;
/* 272 */
/* 273 */             bhj_found = true;
/* 274 */           }
/* 275 */           if (bhj_found) continue;
/* 276 */         }
/* 277 */       }
/* 278 */       bhj_numOutputRows1.add(1);
/* 279 */
/* 280 */       UnsafeRow agg_unsafeRowAggBuffer = null;
/* 281 */
/* 282 */       UnsafeRow agg_fastAggBuffer = null;
/* 283 */
/* 284 */       if (agg_fastAggBuffer == null) {
/* 285 */         // generate grouping key
/* 286 */         agg_holder.reset();
/* 287 */
/* 288 */         agg_rowWriter.zeroOutNullBytes();
/* 289 */
/* 290 */         // Remember the current cursor so that we can calculate how many bytes are
/* 291 */         // written later.
/* 292 */         final int agg_tmpCursor = agg_holder.cursor;
/* 293 */
/* 294 */         if (inputadapter_value instanceof UnsafeRow) {
/* 295 */           final int agg_sizeInBytes = ((UnsafeRow) inputadapter_value).getSizeInBytes();
/* 296 */           // grow the global buffer before writing data.
/* 297 */           agg_holder.grow(agg_sizeInBytes);
/* 298 */           ((UnsafeRow) inputadapter_value).writeToMemory(agg_holder.buffer, agg_holder.cursor);
/* 299 */           agg_holder.cursor += agg_sizeInBytes;
/* 300 */
/* 301 */         } else {
/* 302 */           agg_rowWriter1.reset();
/* 303 */
/* 304 */           final long agg_fieldName = inputadapter_value.getLong(0);
/* 305 */           if (inputadapter_value.isNullAt(0)) {
/* 306 */             agg_rowWriter1.setNullAt(0);
/* 307 */           } else {
/* 308 */             agg_rowWriter1.write(0, agg_fieldName);
/* 309 */           }
/* 310 */
/* 311 */           final long agg_fieldName1 = inputadapter_value.getLong(1);
/* 312 */           if (inputadapter_value.isNullAt(1)) {
/* 313 */             agg_rowWriter1.setNullAt(1);
/* 314 */           } else {
/* 315 */             agg_rowWriter1.write(1, agg_fieldName1);
/* 316 */           }
/* 317 */
/* 318 */           final UTF8String agg_fieldName2 = inputadapter_value.getUTF8String(2);
/* 319 */           if (inputadapter_value.isNullAt(2)) {
/* 320 */             agg_rowWriter1.setNullAt(2);
/* 321 */           } else {
/* 322 */             agg_rowWriter1.write(2, agg_fieldName2);
/* 323 */           }
/* 324 */         }
/* 325 */
/* 326 */         agg_rowWriter.setOffsetAndSize(0, agg_tmpCursor, agg_holder.cursor - agg_tmpCursor);
/* 327 */
/* 328 */         if (bhj_isNull3) {
/* 329 */           agg_rowWriter.setNullAt(1);
/* 330 */         } else {
/* 331 */           // Remember the current cursor so that we can calculate how many bytes are
/* 332 */           // written later.
/* 333 */           final int agg_tmpCursor4 = agg_holder.cursor;
/* 334 */
/* 335 */           if (bhj_value3 instanceof UnsafeRow) {
/* 336 */             final int agg_sizeInBytes1 = ((UnsafeRow) bhj_value3).getSizeInBytes();
/* 337 */             // grow the global buffer before writing data.
/* 338 */             agg_holder.grow(agg_sizeInBytes1);
/* 339 */             ((UnsafeRow) bhj_value3).writeToMemory(agg_holder.buffer, agg_holder.cursor);
/* 340 */             agg_holder.cursor += agg_sizeInBytes1;
/* 341 */
/* 342 */           } else {
/* 343 */             agg_rowWriter2.reset();
/* 344 */
/* 345 */             final long agg_fieldName3 = bhj_value3.getLong(0);
/* 346 */             if (bhj_value3.isNullAt(0)) {
/* 347 */               agg_rowWriter2.setNullAt(0);
/* 348 */             } else {
/* 349 */               agg_rowWriter2.write(0, agg_fieldName3);
/* 350 */             }
/* 351 */
/* 352 */             final UTF8String agg_fieldName4 = bhj_value3.getUTF8String(1);
/* 353 */             if (bhj_value3.isNullAt(1)) {
/* 354 */               agg_rowWriter2.setNullAt(1);
/* 355 */             } else {
/* 356 */               agg_rowWriter2.write(1, agg_fieldName4);
/* 357 */             }
/* 358 */
/* 359 */             final UTF8String agg_fieldName5 = bhj_value3.getUTF8String(2);
/* 360 */             if (bhj_value3.isNullAt(2)) {
/* 361 */               agg_rowWriter2.setNullAt(2);
/* 362 */             } else {
/* 363 */               agg_rowWriter2.write(2, agg_fieldName5);
/* 364 */             }
/* 365 */           }
/* 366 */
/* 367 */           agg_rowWriter.setOffsetAndSize(1, agg_tmpCursor4, agg_holder.cursor - agg_tmpCursor4);
/* 368 */         }
/* 369 */         agg_result.setTotalSize(agg_holder.totalSize());
/* 370 */         agg_value4 = 42;
/* 371 */
/* 372 */         final long agg_element = inputadapter_value.getLong(0);
/* 373 */         agg_value4 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashLong(agg_element, agg_value4);
/* 374 */
/* 375 */         final long agg_element1 = inputadapter_value.getLong(1);
/* 376 */         agg_value4 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashLong(agg_element1, agg_value4);
/* 377 */
/* 378 */         if (!inputadapter_value.isNullAt(2)) {
/* 379 */           final UTF8String agg_element2 = inputadapter_value.getUTF8String(2);
/* 380 */           agg_value4 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(agg_element2.getBaseObject(), agg_element2.getBaseOffset(), agg_element2.numBytes(), agg_value4);
/* 381 */
/* 382 */         }
/* 383 */
/* 384 */         if (!bhj_isNull3) {
/* 385 */           final long agg_element3 = bhj_value3.getLong(0);
/* 386 */           agg_value4 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashLong(agg_element3, agg_value4);
/* 387 */
/* 388 */           if (!bhj_value3.isNullAt(1)) {
/* 389 */             final UTF8String agg_element4 = bhj_value3.getUTF8String(1);
/* 390 */             agg_value4 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(agg_element4.getBaseObject(), agg_element4.getBaseOffset(), agg_element4.numBytes(), agg_value4);
/* 391 */
/* 392 */           }
/* 393 */
/* 394 */           if (!bhj_value3.isNullAt(2)) {
/* 395 */             final UTF8String agg_element5 = bhj_value3.getUTF8String(2);
/* 396 */             agg_value4 = org.apache.spark.unsafe.hash.Murmur3_x86_32.hashUnsafeBytes(agg_element5.getBaseObject(), agg_element5.getBaseOffset(), agg_element5.numBytes(), agg_value4);
/* 397 */
/* 398 */           }
/* 399 */
/* 400 */         }
/* 401 */         if (true) {
/* 402 */           // try to get the buffer from hash map
/* 403 */           agg_unsafeRowAggBuffer =
/* 404 */           agg_hashMap.getAggregationBufferFromUnsafeRow(agg_result, agg_value4);
/* 405 */         }
/* 406 */         if (agg_unsafeRowAggBuffer == null) {
/* 407 */           if (agg_sorter == null) {
/* 408 */             agg_sorter = agg_hashMap.destructAndCreateExternalSorter();
/* 409 */           } else {
/* 410 */             agg_sorter.merge(agg_hashMap.destructAndCreateExternalSorter());
/* 411 */           }
/* 412 */
/* 413 */           // the hash map had be spilled, it should have enough memory now,
/* 414 */           // try  to allocate buffer again.
/* 415 */           agg_unsafeRowAggBuffer =
/* 416 */           agg_hashMap.getAggregationBufferFromUnsafeRow(agg_result, agg_value4);
/* 417 */           if (agg_unsafeRowAggBuffer == null) {
/* 418 */             // failed to allocate the first page
/* 419 */             throw new OutOfMemoryError("No enough memory for aggregation");
/* 420 */           }
/* 421 */         }
/* 422 */       }
/* 423 */
/* 424 */       if (agg_fastAggBuffer != null) {
/* 425 */         // update fast row
/* 426 */
/* 427 */       } else {
/* 428 */         // update unsafe row
/* 429 */
/* 430 */         // common sub-expressions
/* 431 */
/* 432 */         // evaluate aggregate function
/* 433 */
/* 434 */         // update unsafe row buffer
/* 435 */
/* 436 */       }
/* 437 */       if (shouldStop()) return;
/* 438 */     }
/* 439 */
/* 440 */     agg_mapIter = agg_plan.finishAggregate(agg_hashMap, agg_sorter, agg_peakMemory, agg_spillSize);
/* 441 */   }
/* 442 */
/* 443 */   private void wholestagecodegen_init_2() {
/* 444 */     this.bhj_rowWriter7 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(bhj_holder2, 3);
/* 445 */     this.bhj_rowWriter8 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(bhj_holder2, 3);
/* 446 */     agg_result = new UnsafeRow(2);
/* 447 */     this.agg_holder = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(agg_result, 64);
/* 448 */     this.agg_rowWriter = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_holder, 2);
/* 449 */     this.agg_rowWriter1 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_holder, 3);
/* 450 */     this.agg_rowWriter2 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_holder, 3);
/* 451 */
/* 452 */     agg_result1 = new UnsafeRow(2);
/* 453 */     this.agg_holder1 = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(agg_result1, 64);
/* 454 */     this.agg_rowWriter3 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_holder1, 2);
/* 455 */     this.agg_rowWriter4 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(agg_holder1, 3);
/* 456 */
/* 457 */   }
/* 458 */
/* 459 */   public int bhj_compareStruct(InternalRow a, InternalRow b) {
/* 460 */     // when comparing unsafe rows, try equals first as it compares the binary directly
/* 461 */     // which is very fast.
/* 462 */     if (a instanceof UnsafeRow && b instanceof UnsafeRow && a.equals(b)) {
/* 463 */       return 0;
/* 464 */     }
/* 465 */     InternalRow i = null;
/* 466 */
/* 467 */     i = a;
/* 468 */     boolean bhj_isNullA;
/* 469 */     long bhj_primitiveA;
/* 470 */     {
/* 471 */       bhj_isNullA = false;
/* 472 */       bhj_primitiveA = inputadapter_value;
/* 473 */     }
/* 474 */     i = b;
/* 475 */     boolean bhj_isNullB;
/* 476 */     long bhj_primitiveB;
/* 477 */     {
/* 478 */       bhj_isNullB = false;
/* 479 */       bhj_primitiveB = inputadapter_value;
/* 480 */     }
/* 481 */     if (bhj_isNullA && bhj_isNullB) {
/* 482 */       // Nothing
/* 483 */     } else if (bhj_isNullA) {
/* 484 */       return -1;
/* 485 */     } else if (bhj_isNullB) {
/* 486 */       return 1;
/* 487 */     } else {
/* 488 */       int comp = (bhj_primitiveA > bhj_primitiveB ? 1 : bhj_primitiveA < bhj_primitiveB ? -1 : 0);
/* 489 */       if (comp != 0) {
/* 490 */         return comp;
/* 491 */       }
/* 492 */     }
/* 493 */
/* 494 */     i = a;
/* 495 */     boolean bhj_isNullA1;
/* 496 */     long bhj_primitiveA1;
/* 497 */     {
/* 498 */       bhj_isNullA1 = bhj_isNull3;
/* 499 */       bhj_primitiveA1 = bhj_value3;
/* 500 */     }
/* 501 */     i = b;
/* 502 */     boolean bhj_isNullB1;
/* 503 */     long bhj_primitiveB1;
/* 504 */     {
/* 505 */       bhj_isNullB1 = bhj_isNull3;
/* 506 */       bhj_primitiveB1 = bhj_value3;
/* 507 */     }
/* 508 */     if (bhj_isNullA1 && bhj_isNullB1) {
/* 509 */       // Nothing
/* 510 */     } else if (bhj_isNullA1) {
/* 511 */       return -1;
/* 512 */     } else if (bhj_isNullB1) {
/* 513 */       return 1;
/* 514 */     } else {
/* 515 */       int comp = (bhj_primitiveA1 > bhj_primitiveB1 ? 1 : bhj_primitiveA1 < bhj_primitiveB1 ? -1 : 0);
/* 516 */       if (comp != 0) {
/* 517 */         return comp;
/* 518 */       }
/* 519 */     }
/* 520 */
/* 521 */     i = a;
/* 522 */     boolean bhj_isNullA2;
/* 523 */     UTF8String bhj_primitiveA2;
/* 524 */     {
/* 525 */       bhj_isNullA2 = bhj_isNull13;
/* 526 */       bhj_primitiveA2 = bhj_value13;
/* 527 */     }
/* 528 */     i = b;
/* 529 */     boolean bhj_isNullB2;
/* 530 */     UTF8String bhj_primitiveB2;
/* 531 */     {
/* 532 */       bhj_isNullB2 = bhj_isNull13;
/* 533 */       bhj_primitiveB2 = bhj_value13;
/* 534 */     }
/* 535 */     if (bhj_isNullA2 && bhj_isNullB2) {
/* 536 */       // Nothing
/* 537 */     } else if (bhj_isNullA2) {
/* 538 */       return -1;
/* 539 */     } else if (bhj_isNullB2) {
/* 540 */       return 1;
/* 541 */     } else {
/* 542 */       int comp = bhj_primitiveA2.compare(bhj_primitiveB2);
/* 543 */       if (comp != 0) {
/* 544 */         return comp;
/* 545 */       }
/* 546 */     }
/* 547 */
/* 548 */     return 0;
/* 549 */   }
/* 550 */
/* 551 */   private void wholestagecodegen_init_1() {
/* 552 */     this.bhj_broadcast1 = (org.apache.spark.broadcast.TorrentBroadcast) references[5];
/* 553 */
/* 554 */     bhj_relation1 = ((org.apache.spark.sql.execution.joins.UnsafeHashedRelation) bhj_broadcast1.value()).asReadOnlyCopy();
/* 555 */     incPeakExecutionMemory(bhj_relation1.estimatedSize());
/* 556 */
/* 557 */     bhj_result1 = new UnsafeRow(2);
/* 558 */     this.bhj_holder1 = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(bhj_result1, 64);
/* 559 */     this.bhj_rowWriter3 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(bhj_holder1, 2);
/* 560 */     this.bhj_rowWriter4 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(bhj_holder1, 3);
/* 561 */     this.bhj_rowWriter5 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(bhj_holder1, 3);
/* 562 */     this.bhj_numOutputRows1 = (org.apache.spark.sql.execution.metric.SQLMetric) references[8];
/* 563 */     bhj_result2 = new UnsafeRow(2);
/* 564 */     this.bhj_holder2 = new org.apache.spark.sql.catalyst.expressions.codegen.BufferHolder(bhj_result2, 64);
/* 565 */     this.bhj_rowWriter6 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter(bhj_holder2, 2);
/* 566 */
/* 567 */   }
/* 568 */
/* 569 */   public int bhj_compareStruct1(InternalRow a, InternalRow b) {
/* 570 */     // when comparing unsafe rows, try equals first as it compares the binary directly
/* 571 */     // which is very fast.
/* 572 */     if (a instanceof UnsafeRow && b instanceof UnsafeRow && a.equals(b)) {
/* 573 */       return 0;
/* 574 */     }
/* 575 */     InternalRow i = null;
/* 576 */
/* 577 */     i = a;
/* 578 */     boolean bhj_isNullA3;
/* 579 */     long bhj_primitiveA3;
/* 580 */     {
/* 581 */       bhj_isNullA3 = false;
/* 582 */       bhj_primitiveA3 = inputadapter_value;
/* 583 */     }
/* 584 */     i = b;
/* 585 */     boolean bhj_isNullB3;
/* 586 */     long bhj_primitiveB3;
/* 587 */     {
/* 588 */       bhj_isNullB3 = false;
/* 589 */       bhj_primitiveB3 = inputadapter_value;
/* 590 */     }
/* 591 */     if (bhj_isNullA3 && bhj_isNullB3) {
/* 592 */       // Nothing
/* 593 */     } else if (bhj_isNullA3) {
/* 594 */       return -1;
/* 595 */     } else if (bhj_isNullB3) {
/* 596 */       return 1;
/* 597 */     } else {
/* 598 */       int comp = (bhj_primitiveA3 > bhj_primitiveB3 ? 1 : bhj_primitiveA3 < bhj_primitiveB3 ? -1 : 0);
/* 599 */       if (comp != 0) {
/* 600 */         return comp;
/* 601 */       }
/* 602 */     }
/* 603 */
/* 604 */     i = a;
/* 605 */     boolean bhj_isNullA4;
/* 606 */     UTF8String bhj_primitiveA4;
/* 607 */     {
/* 608 */       bhj_isNullA4 = bhj_isNull3;
/* 609 */       bhj_primitiveA4 = bhj_value3;
/* 610 */     }
/* 611 */     i = b;
/* 612 */     boolean bhj_isNullB4;
/* 613 */     UTF8String bhj_primitiveB4;
/* 614 */     {
/* 615 */       bhj_isNullB4 = bhj_isNull3;
/* 616 */       bhj_primitiveB4 = bhj_value3;
/* 617 */     }
/* 618 */     if (bhj_isNullA4 && bhj_isNullB4) {
/* 619 */       // Nothing
/* 620 */     } else if (bhj_isNullA4) {
/* 621 */       return -1;
/* 622 */     } else if (bhj_isNullB4) {
/* 623 */       return 1;
/* 624 */     } else {
/* 625 */       int comp = bhj_primitiveA4.compare(bhj_primitiveB4);
/* 626 */       if (comp != 0) {
/* 627 */         return comp;
/* 628 */       }
/* 629 */     }
/* 630 */
/* 631 */     i = a;
/* 632 */     boolean bhj_isNullA5;
/* 633 */     UTF8String bhj_primitiveA5;
/* 634 */     {
/* 635 */       bhj_isNullA5 = bhj_isNull13;
/* 636 */       bhj_primitiveA5 = bhj_value13;
/* 637 */     }
/* 638 */     i = b;
/* 639 */     boolean bhj_isNullB5;
/* 640 */     UTF8String bhj_primitiveB5;
/* 641 */     {
/* 642 */       bhj_isNullB5 = bhj_isNull13;
/* 643 */       bhj_primitiveB5 = bhj_value13;
/* 644 */     }
/* 645 */     if (bhj_isNullA5 && bhj_isNullB5) {
/* 646 */       // Nothing
/* 647 */     } else if (bhj_isNullA5) {
/* 648 */       return -1;
/* 649 */     } else if (bhj_isNullB5) {
/* 650 */       return 1;
/* 651 */     } else {
/* 652 */       int comp = bhj_primitiveA5.compare(bhj_primitiveB5);
/* 653 */       if (comp != 0) {
/* 654 */         return comp;
/* 655 */       }
/* 656 */     }
/* 657 */
/* 658 */     return 0;
/* 659 */   }
/* 660 */
/* 661 */   protected void processNext() throws java.io.IOException {
/* 662 */     if (!agg_initAgg) {
/* 663 */       agg_initAgg = true;
/* 664 */       long wholestagecodegen_beforeAgg = System.nanoTime();
/* 665 */       agg_doAggregateWithKeys();
/* 666 */       wholestagecodegen_aggTime.add((System.nanoTime() - wholestagecodegen_beforeAgg) / 1000000);
/* 667 */     }
/* 668 */
/* 669 */     // output the result
/* 670 */
/* 671 */     while (agg_mapIter.next()) {
/* 672 */       wholestagecodegen_numOutputRows.add(1);
/* 673 */       UnsafeRow agg_aggKey = (UnsafeRow) agg_mapIter.getKey();
/* 674 */       UnsafeRow agg_aggBuffer = (UnsafeRow) agg_mapIter.getValue();
/* 675 */
/* 676 */       InternalRow agg_value7 = agg_aggKey.getStruct(0, 3);
/* 677 */       boolean agg_isNull8 = agg_aggKey.isNullAt(1);
/* 678 */       InternalRow agg_value8 = agg_isNull8 ? null : (agg_aggKey.getStruct(1, 3));
/* 679 */       agg_holder1.reset();
/* 680 */
/* 681 */       agg_rowWriter3.zeroOutNullBytes();
/* 682 */
/* 683 */       // Remember the current cursor so that we can calculate how many bytes are
/* 684 */       // written later.
/* 685 */       final int agg_tmpCursor8 = agg_holder1.cursor;
/* 686 */
/* 687 */       if (agg_value7 instanceof UnsafeRow) {
/* 688 */         final int agg_sizeInBytes2 = ((UnsafeRow) agg_value7).getSizeInBytes();
/* 689 */         // grow the global buffer before writing data.
/* 690 */         agg_holder1.grow(agg_sizeInBytes2);
/* 691 */         ((UnsafeRow) agg_value7).writeToMemory(agg_holder1.buffer, agg_holder1.cursor);
/* 692 */         agg_holder1.cursor += agg_sizeInBytes2;
/* 693 */
/* 694 */       } else {
/* 695 */         agg_rowWriter4.reset();
/* 696 */
/* 697 */         final long agg_fieldName6 = agg_value7.getLong(0);
/* 698 */         if (agg_value7.isNullAt(0)) {
/* 699 */           agg_rowWriter4.setNullAt(0);
/* 700 */         } else {
/* 701 */           agg_rowWriter4.write(0, agg_fieldName6);
/* 702 */         }
/* 703 */
/* 704 */         final long agg_fieldName7 = agg_value7.getLong(1);
/* 705 */         if (agg_value7.isNullAt(1)) {
/* 706 */           agg_rowWriter4.setNullAt(1);
/* 707 */         } else {
/* 708 */           agg_rowWriter4.write(1, agg_fieldName7);
/* 709 */         }
/* 710 */
/* 711 */         final UTF8String agg_fieldName8 = agg_value7.getUTF8String(2);
/* 712 */         if (agg_value7.isNullAt(2)) {
/* 713 */           agg_rowWriter4.setNullAt(2);
/* 714 */         } else {
/* 715 */           agg_rowWriter4.write(2, agg_fieldName8);
/* 716 */         }
/* 717 */       }
/* 718 */
/* 719 */       agg_rowWriter3.setOffsetAndSize(0, agg_tmpCursor8, agg_holder1.cursor - agg_tmpCursor8);
/* 720 */
/* 721 */       if (agg_isNull8) {
/* 722 */         agg_rowWriter3.setNullAt(1);
/* 723 */       } else {
/* 724 */         // Remember the current cursor so that we can calculate how many bytes are
/* 725 */         // written later.
/* 726 */         final int agg_tmpCursor12 = agg_holder1.cursor;
/* 727 */
/* 728 */         if (agg_value8 instanceof UnsafeRow) {
/* 729 */           final int agg_sizeInBytes3 = ((UnsafeRow) agg_value8).getSizeInBytes();
/* 730 */           // grow the global buffer before writing data.
/* 731 */           agg_holder1.grow(agg_sizeInBytes3);
/* 732 */           ((UnsafeRow) agg_value8).writeToMemory(agg_holder1.buffer, agg_holder1.cursor);
/* 733 */           agg_holder1.cursor += agg_sizeInBytes3;
/* 734 */
/* 735 */         } else {
/* 736 */           agg_rowWriter5.reset();
/* 737 */
/* 738 */           final long agg_fieldName9 = agg_value8.getLong(0);
/* 739 */           if (agg_value8.isNullAt(0)) {
/* 740 */             agg_rowWriter5.setNullAt(0);
/* 741 */           } else {
/* 742 */             agg_rowWriter5.write(0, agg_fieldName9);
/* 743 */           }
/* 744 */
/* 745 */           final UTF8String agg_fieldName10 = agg_value8.getUTF8String(1);
/* 746 */           if (agg_value8.isNullAt(1)) {
/* 747 */             agg_rowWriter5.setNullAt(1);
/* 748 */           } else {
/* 749 */             agg_rowWriter5.write(1, agg_fieldName10);
/* 750 */           }
/* 751 */
/* 752 */           final UTF8String agg_fieldName11 = agg_value8.getUTF8String(2);
/* 753 */           if (agg_value8.isNullAt(2)) {
/* 754 */             agg_rowWriter5.setNullAt(2);
/* 755 */           } else {
/* 756 */             agg_rowWriter5.write(2, agg_fieldName11);
/* 757 */           }
/* 758 */         }
/* 759 */
/* 760 */         agg_rowWriter3.setOffsetAndSize(1, agg_tmpCursor12, agg_holder1.cursor - agg_tmpCursor12);
/* 761 */       }
/* 762 */       agg_result1.setTotalSize(agg_holder1.totalSize());
/* 763 */       append(agg_result1);
/* 764 */
/* 765 */       if (shouldStop()) return;
/* 766 */     }
/* 767 */
/* 768 */     agg_mapIter.close();
/* 769 */     if (agg_sorter == null) {
/* 770 */       agg_hashMap.free();
/* 771 */     }
/* 772 */   }
/* 773 */ }
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 472, Column 36: Expression "inputadapter_value" is not an rvalue
	at org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:11004)
	at org.codehaus.janino.UnitCompiler.toRvalueOrCompileException(UnitCompiler.java:6639)
	at org.codehaus.janino.UnitCompiler.getConstantValue2(UnitCompiler.java:5001)
	at org.codehaus.janino.UnitCompiler.access$10500(UnitCompiler.java:206)
	at org.codehaus.janino.UnitCompiler$13.visitAmbiguousName(UnitCompiler.java:4984)
	at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:3633)
	at org.codehaus.janino.Java$Lvalue.accept(Java.java:3563)
	at org.codehaus.janino.UnitCompiler.getConstantValue(UnitCompiler.java:4956)
	at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:4925)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:3189)
	at org.codehaus.janino.UnitCompiler.access$5100(UnitCompiler.java:206)
	at org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3143)
	at org.codehaus.janino.UnitCompiler$9.visitAssignment(UnitCompiler.java:3139)
	at org.codehaus.janino.Java$Assignment.accept(Java.java:3847)
	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3139)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2112)
	at org.codehaus.janino.UnitCompiler.access$1700(UnitCompiler.java:206)
	at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1377)
	at org.codehaus.janino.UnitCompiler$6.visitExpressionStatement(UnitCompiler.java:1370)
	at org.codehaus.janino.Java$ExpressionStatement.accept(Java.java:2558)
	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
	at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:1436)
	at org.codehaus.janino.UnitCompiler.access$1600(UnitCompiler.java:206)
	at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1376)
	at org.codehaus.janino.UnitCompiler$6.visitBlock(UnitCompiler.java:1370)
	at org.codehaus.janino.Java$Block.accept(Java.java:2471)
	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1370)
	at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1450)
	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:2811)
	at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1262)
	at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1234)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:538)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:890)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:894)
	at org.codehaus.janino.UnitCompiler.access$600(UnitCompiler.java:206)
	at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:377)
	at org.codehaus.janino.UnitCompiler$2.visitMemberClassDeclaration(UnitCompiler.java:369)
	at org.codehaus.janino.Java$MemberClassDeclaration.accept(Java.java:1128)
	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369)
	at org.codehaus.janino.UnitCompiler.compileDeclaredMemberTypes(UnitCompiler.java:1209)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:564)
	at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:420)
	at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:206)
	at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:374)
	at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:369)
	at org.codehaus.janino.Java$AbstractPackageMemberClassDeclaration.accept(Java.java:1309)
	at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:369)
	at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:345)
	at org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:396)
	at org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:311)
	at org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:229)
	at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:196)
	at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:91)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.org$apache$spark$sql$catalyst$expressions$codegen$CodeGenerator$$doCompile(CodeGenerator.scala:935)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:998)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$$anon$1.load(CodeGenerator.scala:995)
	at org.spark_project.guava.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
	at org.spark_project.guava.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
	at org.spark_project.guava.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
	at org.spark_project.guava.cache.LocalCache$Segment.get(LocalCache.java:2257)
	at org.spark_project.guava.cache.LocalCache.get(LocalCache.java:4000)
	at org.spark_project.guava.cache.LocalCache.getOrLoad(LocalCache.java:4004)
	at org.spark_project.guava.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
	at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator$.compile(CodeGenerator.scala:890)
	at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:357)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
	at org.apache.spark.sql.execution.exchange.ShuffleExchange.prepareShuffleDependency(ShuffleExchange.scala:85)
	at org.apache.spark.sql.execution.exchange.ShuffleExchange$$anonfun$doExecute$1.apply(ShuffleExchange.scala:121)
	at org.apache.spark.sql.execution.exchange.ShuffleExchange$$anonfun$doExecute$1.apply(ShuffleExchange.scala:112)
	at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
	at org.apache.spark.sql.execution.exchange.ShuffleExchange.doExecute(ShuffleExchange.scala:112)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
	at org.apache.spark.sql.execution.InputAdapter.inputRDDs(WholeStageCodegenExec.scala:235)
	at org.apache.spark.sql.execution.aggregate.HashAggregateExec.inputRDDs(HashAggregateExec.scala:141)
	at org.apache.spark.sql.execution.WholeStageCodegenExec.doExecute(WholeStageCodegenExec.scala:368)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:114)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:135)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:132)
	at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:113)
	at org.apache.spark.sql.execution.SparkPlan.getByteArrayRdd(SparkPlan.scala:225)
	at org.apache.spark.sql.execution.SparkPlan.executeCollect(SparkPlan.scala:272)
	at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2371)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
	at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2765)
	at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2370)
	at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collect$1.apply(Dataset.scala:2375)
	at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$collect$1.apply(Dataset.scala:2375)
	at org.apache.spark.sql.Dataset.withCallback(Dataset.scala:2778)
	at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2375)
	at org.apache.spark.sql.Dataset.collect(Dataset.scala:2351)
	at org.graphframes.PatternMatchSuite$$anonfun$6.apply$mcV$sp(PatternMatchSuite.scala:103)
	at org.graphframes.PatternMatchSuite$$anonfun$6.apply(PatternMatchSuite.scala:99)
	at org.graphframes.PatternMatchSuite$$anonfun$6.apply(PatternMatchSuite.scala:99)
	at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
	at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
	at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at org.scalatest.Transformer.apply(Transformer.scala:22)
	at org.scalatest.Transformer.apply(Transformer.scala:20)
	at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166)
	at org.graphframes.SparkFunSuite.withFixture(SparkFunSuite.scala:40)
	at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163)
	at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
	at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175)
	at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175)
	at org.scalatest.FunSuite.runTest(FunSuite.scala:1555)
	at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
	at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413)
	at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401)
	at scala.collection.immutable.List.foreach(List.scala:381)
	at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401)
	at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396)
	at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483)
	at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208)
	at org.scalatest.FunSuite.runTests(FunSuite.scala:1555)
	at org.scalatest.Suite$class.run(Suite.scala:1424)
	at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555)
	at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
	at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212)
	at org.scalatest.SuperEngine.runImpl(Engine.scala:545)
	at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212)
	at org.graphframes.PatternMatchSuite.org$scalatest$BeforeAndAfterAll$$super$run(PatternMatchSuite.scala:25)
	at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257)
	at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256)
	at org.graphframes.PatternMatchSuite.run(PatternMatchSuite.scala:25)
	at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:357)
	at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:502)
	at sbt.ForkMain$Run$2.call(ForkMain.java:294)
	at sbt.ForkMain$Run$2.call(ForkMain.java:284)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

@thunterdb thunterdb added the bug label Jan 9, 2017

@felixcheung

This comment has been minimized.

Show comment
Hide comment
@felixcheung

felixcheung Jan 10, 2017

Member

thanks! I didn't think to look for things since tests are passing. Will try to investigate this

Member

felixcheung commented Jan 10, 2017

thanks! I didn't think to look for things since tests are passing. Will try to investigate this

@felixcheung

This comment has been minimized.

Show comment
Hide comment
@felixcheung

felixcheung Jan 10, 2017

Member

Seems like codegen bug; it looks like at least 2 issues:

  1. At L472, inputadapter_value is not defined within scope
  2. inputadapter_value is an InternalRow, for this statement to work
bhj_primitiveA = inputadapter_value;

it should be

bhj_primitiveA = inputadapter_value.getLong(0);
Member

felixcheung commented Jan 10, 2017

Seems like codegen bug; it looks like at least 2 issues:

  1. At L472, inputadapter_value is not defined within scope
  2. inputadapter_value is an InternalRow, for this statement to work
bhj_primitiveA = inputadapter_value;

it should be

bhj_primitiveA = inputadapter_value.getLong(0);
@jkbradley

This comment has been minimized.

Show comment
Hide comment
@jkbradley

jkbradley Mar 27, 2017

Member

This is still an issue. I just reported it: https://issues.apache.org/jira/browse/SPARK-20111

Member

jkbradley commented Mar 27, 2017

This is still an issue. I just reported it: https://issues.apache.org/jira/browse/SPARK-20111

@jkbradley

This comment has been minimized.

Show comment
Hide comment
@jkbradley

jkbradley Mar 27, 2017

Member

Nevermind! Not an issue. Fixed in Spark 2.1.1 and 2.2.0. (I just tested it manually with master.)

Member

jkbradley commented Mar 27, 2017

Nevermind! Not an issue. Fixed in Spark 2.1.1 and 2.2.0. (I just tested it manually with master.)

@jkbradley jkbradley closed this Mar 27, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment