ExpandableDirectByteBuffer maximum capacity only 1GB

[spiritlcx]

maximum capacity of ExpandableDirectByteBuffer is only 1GB
I think the reason is that the type of index is int
Do you think it's better to change int to long and expand the capacity?

[mikeb01]

This would require changing all of the accessor methods that take in a index value and switch it from a int to a long. This in turn would break compatibility with the array backed implementations of the DirectBuffer.

[clayly]

This would require changing all of the accessor methods that take in a index value and switch it from a int to a long. This in turn would break compatibility with the array backed implementations of the DirectBuffer.

What do you think, are there other pitfalls besides compatibility that prevent the increase in the maximum size? Only this limitation does not allow using Aeron in our project, where the buffer size must be at least 60 gigabytes.

[mjpt777]

ByteBuffer and byte arrays cannot be larger be larger than int capacity.

[clayly]

ByteBuffer and byte arrays cannot be larger be larger than int capacity.

That is, if Argon would like to support a ring buffer of this size, then the only option is to work with files directly (memory-mapped files, dev/shm, etc.)?

[mjpt777]

ByteBuffer and byte arrays cannot be larger be larger than int capacity.

That is, if Argon would like to support a ring buffer of this size, then the only option is to work with files directly (memory-mapped files, dev/shm, etc.)?

Mapped files in Java still return a ByteBuffer so still not possible. There are techniques that can be used which are unsupported in the JDK. We can provide commercial consulting on that but we do not provide it as free support in Agrona.

[vyazelenko]

First of all as pointed out @mjpt777 the underlying size cannot exceed int (an actual value is Integer.MAX_VALUE - 8). That being said we could allow ExpandableDirectByteBufferto grow until that max value as oppose to 1GB.

If you need more than 2GB of storage you can use org.agrona.concurrent.MappedResizeableBuffer.