Skip to content

Commit

Permalink
ARROW-7546: [Java] Use new implementation to concat vectors values in…
Browse files Browse the repository at this point in the history
… batch

Related to [ARROW-7546](https://issues.apache.org/jira/browse/ARROW-7546).

Per discussion #5945 (comment).

In ARROW-7284, we write a simple method to concat vectors. However, ARROW-7073 is about to concat vector values efficiently, after this PR merged, we should use this new implementation in ArrowReader.

Closes #6431 from tianchen92/ARROW-7546 and squashes the following commits:

9bced46 <tianchen> ARROW-7546:  Use new implementation to concat vectors values in batch

Authored-by: tianchen <niki.lj@alibaba-inc.com>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
  • Loading branch information
tianchen92 authored and emkornfield committed Feb 18, 2020
1 parent 38504e3 commit 45a7047
Showing 1 changed file with 3 additions and 16 deletions.
Expand Up @@ -35,7 +35,7 @@
import org.apache.arrow.vector.types.pojo.Field;
import org.apache.arrow.vector.types.pojo.Schema;
import org.apache.arrow.vector.util.DictionaryUtility;
import org.apache.arrow.vector.util.TransferPair;
import org.apache.arrow.vector.util.VectorBatchAppender;

/**
* Abstract class to read Schema and ArrowRecordBatches.
Expand Down Expand Up @@ -224,7 +224,8 @@ protected void loadDictionary(ArrowDictionaryBatch dictionaryBatch) {
if (dictionaryBatch.isDelta()) {
FieldVector deltaVector = vector.getField().createVector(allocator);
load(dictionaryBatch, deltaVector);
concatDeltaDictionary(vector, deltaVector);
VectorBatchAppender.batchAppend(vector, deltaVector);
deltaVector.close();
return;
}

Expand All @@ -242,18 +243,4 @@ private void load(ArrowDictionaryBatch dictionaryBatch, FieldVector vector) {
dictionaryBatch.close();
}
}

/**
* Concat dictionary vector and delta dictionary vector.
*/
private void concatDeltaDictionary(FieldVector vector, FieldVector deltaVector) {
final int valueCount = vector.getValueCount();
final int deltaValueCount = deltaVector.getValueCount();
final TransferPair transferPair = deltaVector.makeTransferPair(vector);
for (int i = 0; i < deltaValueCount; i++) {
transferPair.copyValueSafe(i, valueCount + i);
}
deltaVector.close();
vector.setValueCount(valueCount + deltaValueCount);
}
}

0 comments on commit 45a7047

Please sign in to comment.