Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests and fixes for RoaringBitmap.addOffset #261

Merged
merged 10 commits into from
Jun 24, 2018
Merged

Tests and fixes for RoaringBitmap.addOffset #261

merged 10 commits into from
Jun 24, 2018

Conversation

richardstartin
Copy link
Member

@richardstartin richardstartin commented Jun 22, 2018

This PR shows up some issues in RoaringBitmap.addOffset and fixes some low hanging fruit.

  • There was an array index out of bounds in RunContainer.append when a run container was placed after a bitmap container
  • There was an assertion error when a run container came after a bitmap container and Util.cardinalityInRange was called.
  • An assertion has been added to RoaringArray.append to reject empty containers.
  • A test for equivalence of shifted bitmaps before and after (de)serialization has been added.
  • Tests show some data is lost when the offset is not a multiple of 65536
  • Remove some very slow fuzz tests for parallel aggregation because I can't be bothered to watch them run
  • A test case for the data scenario reported in RoaringBitmap.addOffset causing problems with serialization/deserialization #260 (could not reproduce)

This PR will break because it contains tests which capture bugs but I don't have time to fix them.

@richardstartin richardstartin changed the title Empty containers Tests and fixes for RoaringBitmap.addOffset Jun 22, 2018
@richardstartin
Copy link
Member Author

The tests should all be passing now.

@coveralls
Copy link

coveralls commented Jun 23, 2018

Coverage Status

Coverage increased (+0.1%) to 91.591% when pulling d3d5b31 on richardstartin:empty-containers into 1f99ab6 on RoaringBitmap:master.

@richardstartin
Copy link
Member Author

I've ported these fixes to the buffer implementation just now (I didn't see any evidence of empty containers there either).

@lemire
Copy link
Member

lemire commented Jun 24, 2018

LGTM. Merging.

@lemire lemire merged commit a56d87b into RoaringBitmap:master Jun 24, 2018
@richardstartin richardstartin deleted the empty-containers branch June 25, 2018 08:21
Smallhi pushed a commit to Smallhi/RoaringBitmap that referenced this pull request Jun 14, 2021
* add assert to RoaringArray.append to reject empty input

* remove slow fuzz tests which have good coverage elsewhere

* fix issues with addOffset, add tests to show existence of more issues

* add more failure cases where data is lost

* add cardinality check to demonstrate issue more clearly

* use logical rather than arithmetic shift to solve bitmap concatenation problem

* more failure cases and some fixes

* fix run container concatenation

* demonstrate failure cases exist in buffer implementation

* apply fixes to buffer implementation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants