[C# -> R} - Record Batches in reverse order?

[abbotware]

Are 'RecordBatches' being in C# being written in reverse order?

I made a simple test which creates a single row per record batch of 0 to 99 and attempted to read this in R. To my surprise batch(0) in R had the value 99 not 0

This may not seem like a big deal, however when dealing with 'huge' files, its more efficient to use Record Batches / index lookup than attempting to load the entire file into memory.

Having the order consistent within the different language / API seems only to make sense - for now I can work around this by reversing the order before writing.

[abbotware]

sample code:

        var batches = new List<RecordBatch>();
        var memoryAllocator = new NativeMemoryAllocator(alignment: 64);

        for (int i = 0; i < 100; ++i)
        {
            var builder = new RecordBatch.Builder(memoryAllocator);

            var batch = builder.AppendNullableInt32("Int32", new List<int?> { i })
                .Build();

            batches.Add(batch);
        }

        using (var stream = File.OpenWrite("C:\\temp\\batch.order.tests.arrow"))
        using (var writer = new ArrowFileWriter(stream, batches[0].Schema))
        {
            foreach(var b in batches)
            {
                writer.WriteRecordBatchAsync(b).Wait();
            }

            writer.WriteFooterAsync().Wait();
        }

[abbotware]

R Code:

rbfr <- arrow::RecordBatchFileReader("C:\temp\batch.order.tests.arrow")
rbn <- rbfr$get_batch(0)
df <- as.data.frame(rbn)

[wesm]

@abbotware can you use JIRA for bug reports?

[abbotware]

@wesm sure - I think I have an old ASF JIRA... but I haven't used it in years ;-)

[abbotware]

https://issues.apache.org/jira/browse/ARROW-6681