Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement roaring_bitmap_internal_validate #400

Open
lemire opened this issue Sep 26, 2023 · 2 comments
Open

Implement roaring_bitmap_internal_validate #400

lemire opened this issue Sep 26, 2023 · 2 comments

Comments

@lemire
Copy link
Member

lemire commented Sep 26, 2023

When deserializing a bitmap, it is possible that the result might be invalid. This could happen because there was data corruption. The deserialization could still generate a bitmap without failure, but the result could be otherwise unusable.

You can avoid such problems by hashing your saved data (e.g., md5sum). But we could could also directly, at some expense, validate the deserialized data.

The C version of Roaring has an interesting function that can be called after deserializing a bitmap, to make sure it is proper:

https://github.com/RoaringBitmap/CRoaring/blob/a103d3811702b9389c538881c9974e9a7a7552af/src/roaring.c#L435

     roaring_bitmap_t *t = roaring_bitmap_portable_deserialize_safe(serializedbytes, expectedsize);
     if(t == NULL) { return EXIT_FAILURE; }
     const char *reason = NULL;
     if (!roaring_bitmap_internal_validate(t, &reason)) {
         return EXIT_FAILURE;
     }

It is not very difficult to implement and could help users who have production data.

@bearrito
Copy link

@lemire Is this issue still valid? I wanted to contribute - the good first issue list seems stale https://github.com/RoaringBitmap/roaring/contribute though

@lemire
Copy link
Member Author

lemire commented Apr 26, 2024

@bearrito Yes. That is a very good first issue.

The C code is well tested and should be similar to what is needed in Go:

https://github.com/RoaringBitmap/CRoaring/blob/a103d3811702b9389c538881c9974e9a7a7552af/src/roaring.c#L435

It would be highly valued by some users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants