Support for bitmaps of size > 2^32 in C

[fsaintjacques]

In OLAP workload, 2^32 rows is quickly reached. This limitation impedes the use of roaring bitmaps as bitmap indices on such system.

It seems the implementation (and the paper) are strongly built on 2^32 assumption. What are the implication of growing the number of buckets vs the size of a bucket?

[lemire]

@fsaintjacques

1. It is "easy" to build a 64-bit data structure on top a 32-bit one. Put the Roaring bitmaps into a tree with 32-bit keys. Samy Chambi has done the work and will have a chapter in his thesis on this topic. There are a few interesting design issues. This work should be published in 2016. Because it can be built on top of a 32-bit data structure, then it is not an issue. It is also possible to build a custom 64-bit data structure, but, again, the 16-bit containers can still prove useful as-is. So supporting 64-bit integers can be done separately without harm.
2. Roaring is used in analytical systems like Druid or Apache Kylin that do scale up to very large data sets, while using only 32 bits. That's because 2^32 is a large number and it often makes sense to partition the problem well before you get to such counts. You probably want to partition for other reasons... such as parallelism.
3. If you do have a practical use case where, for example, hash sets are not an obviously good choice, then I would be very interested.

[lemire]

I'm going to leave this open.

[lemire]

We have several implementations of Roaring over 64-bit integers in separate repositories...

https://bitbucket.org/samytto/lazyroaring64-bits

https://bitbucket.org/samytto/roaring64bits2levels

https://bitbucket.org/samytto/roaringtreemap

... as well as an evaluation...

https://bitbucket.org/samytto/datastructures64-bitints

[madscientist]

All the implementations above appear to be in Java; any news on a 64bit version for C/C++? We have various IDs which are 64bits (32 bits is a lot, yes, but when you're talking 100's of transactions / sec and services running for weeks/months, and being stored durably as well, it goes by pretty quickly...). We'll look into a two-layer approach. You mentioned a year ago that Samy Chambi was working on doing this and might publish something; did that happen and if so is there someplace to get a copy?

Cheers!

[lemire]

@madscientist

any news on a 64bit version for C/C++?

Not that I know. I have marked the issue as "help wanted" just now.

You mentioned a year ago that Samy Chambi was working on doing this and might publish something; did that happen and if so is there someplace to get a copy?

Yes. There is a paper in French with experimental results as well as links to (Java) implementations. http://r-libre.teluq.ca/930/1/Roaring64bits.pdf

[lemire]

We have a relevant PR : #75

[lemire]

With the merger of PR #75, this issue is resolved for the c++ layer.

[lemire]

I think that the issue remains open for C users.