This repository has been archived by the owner. It is now read-only.

LocalTransformProcessRecordReader can't handle filter ops #552

AlexDBlack opened this Issue Apr 9, 2018 · 0 comments


None yet
2 participants
Copy link

AlexDBlack commented Apr 9, 2018

Exception in thread "main" java.lang.IndexOutOfBoundsException: index: 0, length: 1 (expected: range(0, 0))
	at io.netty.buffer.ArrowBuf.checkIndexD(
	at io.netty.buffer.ArrowBuf.chk(
	at io.netty.buffer.ArrowBuf.getByte(
	at org.apache.arrow.vector.BaseFixedWidthVector.isSet(
	at org.apache.arrow.vector.BaseFixedWidthVector.isNull(
	at org.datavec.arrow.recordreader.ArrowWritableRecordBatch.get(
	at org.datavec.arrow.recordreader.ArrowWritableRecordBatch.get(
	at org.datavec.api.records.mapper.RecordMapper.copy(
	at org.datavec.transform.basic.BasicDataVecExampleLocal.main(

Reason for the exception: a length 0 output, which I'm pretty sure is due to filtering.

Edit: unit test:

    public void testLocalFilter(){

        List<List<Writable>> in = new ArrayList<>();
        in.add(Arrays.asList(new Text("Keep"), new IntWritable(0)));
        in.add(Arrays.asList(new Text("Remove"), new IntWritable(1)));
        in.add(Arrays.asList(new Text("Keep"), new IntWritable(2)));
        in.add(Arrays.asList(new Text("Remove"), new IntWritable(3)));

        Schema s = new Schema.Builder()
                .addColumnCategorical("cat", "Keep", "Remove")

        TransformProcess tp = new TransformProcess.Builder(s)
                .filter(new CategoricalColumnCondition("cat", ConditionOp.Equal, "Remove"))

        RecordReader rr = new CollectionRecordReader(in);
        LocalTransformProcessRecordReader ltprr = new LocalTransformProcessRecordReader(rr, tp);

        List<List<Writable>> out = new ArrayList<>();

        List<List<Writable>> exp = Arrays.asList(in.get(0), in.get(2));

        assertEquals(exp, out);

@AlexDBlack AlexDBlack added the bug label Apr 9, 2018

@raver119 raver119 added the ETL label Apr 29, 2018

@AlexDBlack AlexDBlack self-assigned this May 8, 2018

AlexDBlack added a commit that referenced this issue May 8, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.