[C#] Large record batch is written with negative buffer length

[asfimport]

I have a 30 gig arrow file with 100 batches.  the largest batch in the file causes get batch to fail - All other batches load fine. in 14.11 the individual batch errors.. in 15.1.1 the batch crashes R studio when it is used

14.1.1

>  rbn <- data_rbfr$get_batch(x)
Error in ipc__RecordBatchFileReader_ReadRecordBatch(self, i) : 
Invalid: negative malloc size
  

15.1.1

rbn <- data_rbfr$get_batch(x)  works!
df <- as.data.frame(rbn) - Crashes R Studio! 

 

Update

I put the data in the batch into a separate file.  The file size is over 2 gigs. 

Using 15.1.1, when I try to load this entire file via read_arrow it also fails.

ar <- arrow::read_arrow("e:\\temp\\file.arrow") 
Error in Table__from_RecordBatchFileReader(batch_reader) :
 Invalid: negative malloc size

 

Reporter: Anthony Abate / @abbotware

Original Issue Attachments:

• image-2019-11-13-16-27-30-641.png
• SingleBatch_String_70000_Rows.ok.rar
• SingleBatch_String_85000_Rows.crash.rar

_{Note: This issue was originally created as ARROW-7156. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
Could you please provide a code example and output/error message? Even without sharing the file, that would help us understand what you're seeing.

[asfimport]

Anthony Abate / @abbotware:
I working on it-... I have to 'downgrade' arrow since 15.x seems even more broken...

[asfimport]

Anthony Abate / @abbotware:
ok updated - I don't know the exact size of the batch - but it can't be coincidence that the largest batch in the file fails to load - i suspect there is some size limitation that was hit

[asfimport]

Anthony Abate / @abbotware:
this is still a problem in 15.1.1 but the failure is different

rbn <- data_rbfr$get_batch(x)  works!
df <- as.data.frame(rbn) - Crashes R Studio!
  

 

 

[asfimport]

Neal Richardson / @nealrichardson:
Searching for the error message, I see

arrow/cpp/src/arrow/memory_pool.cc

Line 285 in a33bd3a

return Status::Invalid("negative malloc size");

which suggests an overflow somewhere.

Could you please clarify what "Crashes R Studio!" means?

https://issues.apache.org/jira/issues/?jql=project%20%3D%20ARROW%20AND%20fixVersion%20%3D%201.0.0%20AND%20component%20%3D%20R%20AND%20text%20~%20%222gb%22 finds 3 known issues about large string columns--could that be involved?

[asfimport]

Anthony Abate / @abbotware:
@nealrichardson- do you know if an individual RecordBatch can exceed 2 gigs (int32 max) ?

This might not be an Arrow C++ issue, but another bug in the C# library that I used to generate the file.

[asfimport]

Anthony Abate / @abbotware:
@nealrichardson- crashes RStudio means just that - instead of an error message

 

[asfimport]

Anthony Abate / @abbotware:
From Event viewer:

 

Faulting application name: rsession.exe, version: 1.2.1335.0, time stamp: 0x5c9d0154
Faulting module name: arrow.dll, version: 0.0.0.0, time stamp: 0x5dc40022
Exception code: 0xc0000005
Fault offset: 0x00000000003e4c05
Faulting process id: 0x8ec
Faulting application start time: 0x01d59a59ff052a76
Faulting application path: C:\software\RStudio\bin\rsession.exe
Faulting module path: C:\Users\aabate\Documents\R\win-library\3.6\arrow\libs\x64\arrow.dll
Report Id: db7e29f8-54ba-40fc-a104-75d3b6f75d0e
Faulting package full name:
Faulting package-relative application ID:

[asfimport]

Anthony Abate / @abbotware:
I uploaded some test files. they deceptively small compressed.. but 2gigs uncompressed

I have a work around for now - just make sure my batches are less than 2 gigs. 

[asfimport]

Neal Richardson / @nealrichardson:
https://arrow.apache.org/docs/format/Columnar.html doesn't say that RecordBatches are limited to 2 GB, but either way it sounds like there's an issue in the C# writer if it's (presumably) overflowing int32 and reporting negative offsets in the file you're trying to read.

[asfimport]

Ben Kietzman / @bkietz:
@abbotware If it's supsected that the c# writer is emitting invalid record batches, could you share the code which generates your test files?

[asfimport]

Ben Kietzman / @bkietz:
I can reproduce this failure with

arrow-file-to-stream SingleBatch_String_85000_Rows.arrow > /dev/null

I can confirm that the buffer length is negative as we read from flatbuffers https://github.com/apache/arrow/blob/bef9a1c/cpp/src/arrow/ipc/message.cc#L159

C# writer does seem to be producing an invalid file.

[asfimport]

Wes McKinney / @wesm:
Until the C# library participates in the integration tests we will continue to be plagued by issues like this.

[asfimport]

Anthony Abate / @abbotware:
@wesm - I should also point out the C# library on Net Framework is not even stable in its current state due to random crashes related to this: ARROW-7040  (I have a local build that fixes is this - so I can make a PR for this one)

Regarding the integration tests - since C# is not using the C++ libs, how do the integration tests work? (i can volunteer some of my time on this, but I may have a lot of questions)

[asfimport]

Wes McKinney / @wesm:
https://github.com/apache/arrow/tree/master/integration#arrow-integration-testing