"Frozen" serialization format

[andreigudkov]

Container data is stored in a format identical
to their C memory layout.
Instead of full deserialization, a lightweight readonly
view can be created atop of provided buffer.
This reduces amount of allocations and copying
and is suitable for creating readonly bitmaps from
memory mapped files.

• Added new functions:
  roaring_bitmap_frozen_size_in_bytes()
  roaring_bitmap_frozen_serialize()
  roaring_bitmap_frozen_view()

• copy_on_write was moved into more generic roaring_array_t.flags
  To modify copy_on_write behavior, new functions were added:
  roaring_bitmap_get_copy_on_write()
  roaring_bitmap_set_copy_on_write()

• roaring_bitmap_free() now takes pointer to a constant bitmap

[lemire]

Fantastic. I consider that this is a satisfactory fix for #74

[PerpEternal]

@andreigudkov Hello, why roaring_bitmap_frozen_view input buf must align 32, would it still work if I remove this restriction?

CRoaring/src/roaring.c

Line 3215 in 33da04d

if ((uintptr_t)buf % 32 != 0) {

[andreigudkov]

@andreigudkov Hello, why roaring_bitmap_frozen_view input buf must align 32, would it still work if I remove this restriction?

CRoaring/src/roaring.c

Line 3215 in 33da04d

if ((uintptr_t)buf % 32 != 0) {

If I recall everything correctly, then:

1. All containers of the bitset type are located in the very beginning of the buffer
2. Bitset container operations use AVX2 operations with load_mm256_load_si256 (not loadu). This requires that input is aligned by 32 bytes.

Probably things have changed since frozen format was introduced, and maybe this check is not required anymore.

[lemire]

Bitset container operations use AVX2 operations with load_mm256_load_si256 (not loadu). This requires that input is aligned by 32 bytes.

They do not, at least not currently.

[lemire]

@PerpEternal It would be possible to revisit the alignment requirement, but someone needs to review the code and make sure that it is safe to lift it, and to run tests. If you are interested, don't hesitate. Ideally, we would want unit tests to make sure that it all works out.

[andreigudkov]

@lemire Are you sure that you do not depend on the alignment? I can see that currently on x64 the alignment is either 32 or 64 (when I created frozen format, it was always 32).

CRoaring/src/containers/bitset.c

Line 75 in 8fc18ac

bitset->words = (uint64_t *)roaring_aligned_malloc(

[lemire]

Are you sure that you do not depend on the alignment?

I am sure of nothing. I only wrote that it could be revisited.