Skip to content

Commit

Permalink
ARROW-6795: [C#] Fix for reading large (2GB+) files
Browse files Browse the repository at this point in the history
It seems that trying to read larger than 2GB+ files will blow up.

As long as the record batches are less than 2GB (the max size of span) there should be no problem reading a large file

Closes #5412 from abbotware/Fix-For-Large-Files and squashes the following commits:

898d556 <Eric Erhardt> Respond to PR feedback.
a556ac2 <Anthony Abate> fix for reading large (2GB+) files

Lead-authored-by: Anthony Abate <anthony.abate@gmail.com>
Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>
Signed-off-by: Micah Kornfield <emkornfield@gmail.com>
  • Loading branch information
2 people authored and emkornfield committed Oct 18, 2019
1 parent a81db80 commit 3675073
Showing 1 changed file with 4 additions and 9 deletions.
13 changes: 4 additions & 9 deletions csharp/src/Apache.Arrow/Ipc/ArrowFileReaderImplementation.cs
Expand Up @@ -33,11 +33,6 @@ internal sealed class ArrowFileReaderImplementation : ArrowStreamReaderImplement
/// </summary>
private int _recordBatchIndex;

/// <summary>
/// Notes what byte position where the footer data is in the stream
/// </summary>
private int _footerStartPostion;

private ArrowFooter _footer;

public ArrowFileReaderImplementation(Stream stream, MemoryAllocator allocator, bool leaveOpen)
Expand Down Expand Up @@ -77,9 +72,9 @@ protected override async ValueTask ReadSchemaAsync()

await ArrayPool<byte>.Shared.RentReturnAsync(footerLength, async (buffer) =>
{
_footerStartPostion = (int)GetFooterLengthPosition() - footerLength;
long footerStartPostion = GetFooterLengthPosition() - footerLength;
BaseStream.Position = _footerStartPostion;
BaseStream.Position = footerStartPostion;
int bytesRead = await BaseStream.ReadFullBufferAsync(buffer).ConfigureAwait(false);
EnsureFullRead(buffer, bytesRead);
Expand Down Expand Up @@ -110,9 +105,9 @@ protected override void ReadSchema()

ArrayPool<byte>.Shared.RentReturn(footerLength, (buffer) =>
{
_footerStartPostion = (int)GetFooterLengthPosition() - footerLength;
long footerStartPostion = GetFooterLengthPosition() - footerLength;
BaseStream.Position = _footerStartPostion;
BaseStream.Position = footerStartPostion;
int bytesRead = BaseStream.ReadFullBuffer(buffer);
EnsureFullRead(buffer, bytesRead);
Expand Down

0 comments on commit 3675073

Please sign in to comment.