TestValueVector.testFixedVectorReallocation and testVariableVectorReallocation are flaky

[asfimport]

The Travis-ci build has failled several times on these tests.
It looks like they often throw OOME.
stacktrace bellow:

testFixedVectorReallocation(org.apache.arrow.vector.TestValueVector)  Time elapsed: 0.174 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<org.apache.arrow.vector.util.OversizedAllocationException> but was<org.apache.arrow.memory.OutOfMemoryException>
	at java.nio.Bits.reserveMemory(Bits.java:658)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
	at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:155)
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:195)
	at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:62)
	at org.apache.arrow.memory.AllocationManager.<init>(AllocationManager.java:79)
	at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:238)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:220)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:190)
	at org.apache.arrow.vector.UInt4Vector.allocateBytes(UInt4Vector.java:189)
	at org.apache.arrow.vector.UInt4Vector.allocateNew(UInt4Vector.java:171)
	at org.apache.arrow.vector.TestValueVector.testFixedVectorReallocation(TestValueVector.java:106)
testVariableVectorReallocation(org.apache.arrow.vector.TestValueVector)  Time elapsed: 0.148 sec  <<< ERROR!
java.lang.Exception: Unexpected exception, expected<org.apache.arrow.vector.util.OversizedAllocationException> but was<org.apache.arrow.memory.OutOfMemoryException>
	at java.nio.Bits.reserveMemory(Bits.java:658)
	at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
	at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.allocateDirect(UnpooledUnsafeDirectByteBuf.java:108)
	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:69)
	at io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:50)
	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:155)
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:195)
	at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:62)
	at org.apache.arrow.memory.AllocationManager.<init>(AllocationManager.java:79)
	at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:238)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:220)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:190)
	at org.apache.arrow.vector.VarCharVector.allocateNew(VarCharVector.java:364)
	at org.apache.arrow.vector.TestValueVector.testVariableVectorReallocation(TestValueVector.java:163)
Results :
Tests in error: 
  TestValueVector.testFixedVectorReallocation »  Unexpected exception, expected<...
  TestValueVector.testVariableVectorReallocation »  Unexpected exception, expect...

Reporter: Julien Le Dem / @julienledem
Assignee: Jihoon Son / @jihoonson

_{Note: This issue was originally created as ARROW-260. Please see the migration documentation for further details.}

[asfimport]

Julien Le Dem / @julienledem:
from jihoonson: #116 (comment)

I'm also suffering from this error, and found the reason. The BaseValueVector.MAX_ALLOCATION_SIZE value used in TestValueVector is declared as a static variable which means its value is initialized when the BaseValueVector class is loaded. Obviously, changing BaseValueVector.MAX_ALLOCATION_SIZE by setting the "arrow.vector.max_allocation_bytes" property in TestValueVector doesn't work. As a result, this test tries to allocate a very large array of the size of Integer.MAX_VALUE, which causes OOM.

I think that this test looks fragile, and thus we need to make the allocation size small. However, reducing the allocation size causes another problem. Many tests in TestValueVector expect the OversizedAllocationException when value allocation reaches to MAX_VALUE_ALLOCATION. So, reducing the allocation size makes OversizedAllocation tests difficult.

I would like to separate all these OversizedAllocation tests, but not sure this is a right way. Please give me some help.

[asfimport]

Jihoon Son / @jihoonson:
@julienledem, thanks for creating this ticket.

The simplest solution is to add a surefire option -Darrow.vector.max_allocation_bytes=1048576 to limit the max allocation size. This will affect to all unit testings, but I think this is a valid solution because we can avoid the allocation of large memory for a vector which can cause OOM in poor test environments.
Any thoughts on this solution?

[asfimport]

Jacques Nadeau / @jacques-n:
Let's just move the tests that need this functionality to a separate test class that doesn't match the normal surefire pattern (Test... or ...Test) and then run under a separate surefire execution within maven for that one test class.

[asfimport]

Julien Le Dem / @julienledem:
I'd suggest we do that when we need it to keep things simple for now.
That would be needed when we have a unit test that requires to allocate large buffers.
Which would probably make that test not a unit test anymore and we would move is to another target.

[asfimport]

Jacques Nadeau / @jacques-n:
I'm fine with setting the surefire option in the default execution for now.

[asfimport]

Jacques Nadeau / @jacques-n:
Note, it is probably still good to move these three tests into a separate class and put a disclaimer at the top about the parameter.

[asfimport]

Jihoon Son / @jihoonson:
Thanks. I updated my patch. 062e312

[asfimport]

Julien Le Dem / @julienledem:
Resolved in:
e8724f8