HIVE-21935: Hive Vectorization : degraded performance with vectorize UDF

[mustafaiman]

Change-Id: I5fad847456d8c3319ea07cfe114007ed01b54bbe

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

[rbalamohan]

After reset, can you reinit isNull and isRepeating for columns which are not present in currentDataColumnCount?
(i.e similar to the case mentioned in https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapOperator.java#L706)

[mustafaiman]

I excluded partition column and rowidentifiercolumn from reset. So this is more like the original code now. It only resets the output columns in addition to original implementation.

[abstractdog]

the change makes sense to me, could you confirm that this use-case is covered by a q.test?
I was looking for qtests, I found similar ones but only with struct type (StructColumnVector, which doesn’t inherit MultiValuedColumnVector)
e.g.:
schema_evol_text_vecrow_part_all_complex.q
I think q.test for this use should have the following properties:

1. use text format file for reading
2. force vector/row serde deserialize:

SET hive.vectorized.use.vectorized.input.format=false;
SET hive.vectorized.use.vector.serde.deserialize=true;
SET hive.vectorized.use.row.serde.deserialize=true;

3. use map/list type
4. work on small batches or a large amount of rows <1024 (to force batch reuse = reuse of deserializerBatch)

[pgaref]

Latest changes + test look good to me!

[mustafaiman]

@abstractdog @rbalamohan are you ok with the latest patch?

[abstractdog]

+1
debugged locally, resetVectorizedRowBatchForDeserialize only clears data columns, then skips partition columns if any, and then clears all the remaining columns, so this looks good to me
thanks @mustafaiman for the patch