Memory mappable ImmutableRoaring64NavigableBitmap

[JervenBolleman]

Is your feature request related to a problem? Please describe.
I would like to use Roaring64NavigableBitmap as an read-only data-structure that may be backed by memory mapped implementations.

Describe the solution you'd like
Add a new implementation in a package *.longlong.buffer immutable as an analogue for the mutable/immutable split of the 32bit options

I am going to try coding this up myself, but would like to know if anyone else is working on this so we could work together.

This might be a time where we can start documenting the on disk format so that it is the same between languages.

[blacelle]

I'm quite convinced nobody is working on this yet.

This might be a time where we can start documenting the on disk format so that it is the same between languages.

This has been discussed most recently in #366
. There is a PR open in #572. I'm not sure of what would be an acceptable strategy (in the vein of #351).

[JervenBolleman]

Ok. I am going to try taking the current on java disk format as guide. and see how that goes :)

[blacelle]

Regarding

I would like to use Roaring64NavigableBitmap as an read-only data-structure that may be backed by memory mapped implementations.

Can this be achieved with org.roaringbitmap.longlong.Roaring64NavigableMap#Roaring64NavigableMap(org.roaringbitmap.BitmapDataProviderSupplier) ?

[JervenBolleman]

I would like to use Roaring64NavigableBitmap as an read-only data-structure that may be backed by memory mapped implementations.

Can this be achieved with org.roaringbitmap.longlong.Roaring64NavigableMap#Roaring64NavigableMap(org.roaringbitmap.BitmapDataProviderSupplier) ?

I have been looking into this but I did not see an obvious way to do it. Because we would need to do a lot of hacking in the deserialize method to get round the issues of the fact that getHighToBitmap has default visibility.

[blacelle]

Because we would need to do a lot of hacking in the deserialize method to get round the issues of the fact that getHighToBitmap has default visibility.

I do not get your point. I suppose a PR would make things clearer.

as an analogue for the mutable/immutable split of the 32bit options

It looks reasonnable to introduce ImmutableRoaring64NavigableBitmap as parent class of Roaring64NavigableBitmap, to follow the same design.

[JervenBolleman]

Because we would need to do a lot of hacking in the deserialize method to get round the issues of the fact that getHighToBitmap has default visibility.
I do not get your point. I suppose a PR would make things clearer.

I get your point after reading more Roaring code :) So using a MutableByteBuffer supplier would allow us to use off-heap buffers. But what I personally need is bytebuffers that are mapped from existing files. I think the only method that we can use to get from a saved on disk status to something accessible from java is the deserialize method that currently exists in Roaring64NavigableMap.
I now have an idea of how this can be hacked in. But a clean supported solution would be off greater benefit :)

(Side note: My problem is I have 37Gb of Roaring64NavigableMaps but only 32Gb of RAM :( so I want much of that the be paged in on demand) Second problem is that at least one of the maps is itself larger than 2gb so I can't just throw in one ByteBuffer :( but have to be a bit smarter about it.

[blacelle]

Working on another matter, I feel relevant to add a way to add(high, RoaringBitmap) into ImmutableRoaring64NavigableBitmap. This would relate the a feature to RoaringBitmap get(high)