Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Add `rsdic` combined rank and select data structure #10
I've ported @hillbig's implementation  of Navarro and Providel's data structure  for simultaneously accelerating rank and select queries over a bit vector. This structure is great for select heavy workloads, like wavelet matrices  that currently binary search over ranks. The shape of the implementation is largely the same but with some small changes and Rust idioms.
The testing is all through quickcheck by testing all inputs for
I've also added some basic benchmarks, and the results are already pretty good. This benchmark generates a random 1M bitvector and queries the rank at 1000 different indices.
So, this data structure is faster than Jacobson for rank but slower than rank9, and it's much faster than binary search for my given benchmark.
Then, I added SIMD acceleration for the rank computation, and it's closer to rank9 now. It's behind an experimental feature flag since that whole ecosystem isn't fully settled yet. Here's the benchmarks rerun with the feature and
Not all of the speed up is just from changing
benchmark results: ``` rsdic::rank time: [16.738 us 16.957 us 17.196 us] Found 7 outliers among 100 measurements (7.00%) 7 (7.00%) high mild jacobson::rank time: [21.522 us 21.721 us 21.958 us] Found 3 outliers among 100 measurements (3.00%) 2 (2.00%) high mild 1 (1.00%) high severe rank9::rank time: [7.7177 us 7.7851 us 7.8541 us] Found 5 outliers among 100 measurements (5.00%) 4 (4.00%) high mild 1 (1.00%) high severe rsdic::select0 time: [31.906 us 32.131 us 32.372 us] Found 3 outliers among 100 measurements (3.00%) 3 (3.00%) high mild rsdic::select1 time: [34.378 us 34.956 us 35.641 us] Found 4 outliers among 100 measurements (4.00%) 1 (1.00%) high mild 3 (3.00%) high severe rank9::binsearch::select0 time: [296.19 us 302.05 us 307.71 us] Found 11 outliers among 100 measurements (11.00%) 9 (9.00%) high mild 2 (2.00%) high severe rank9::binsearch::select1 time: [291.32 us 298.60 us 305.48 us] Found 11 outliers among 100 measurements (11.00%) 6 (6.00%) high mild 5 (5.00%) high severe ```
The nice coincidence here is that we represent the small block classes as u8s in a dense buffer and that the start of a large block is always divisible by 1024. So, we can directly load small block classes as a u8x16 vector and then loop through a large block in a single go. There's a few tricks described in the comments to turn indexing into the `ENUM_CODE_LENGTH` table into simple vector operations plus a shuffle. Check them out! Benchmarks after turning on the `simd_acceleration` feature: ``` $ RUSTFLAGS="-C target-cpu=native" cargo bench --features simd_acceleration -- '::rank' rsdic::rank time: [7.9304 ns 7.9607 ns 7.9929 ns] change: [-22.005% -21.462% -20.881%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 4 (4.00%) high mild 3 (3.00%) high severe jacobson::rank time: [13.985 ns 14.057 ns 14.141 ns] change: [-1.9807% -0.6375% +0.6820%] (p = 0.37 > 0.05) No change in performance detected. Found 5 outliers among 100 measurements (5.00%) 1 (1.00%) high mild 4 (4.00%) high severe rank9::rank time: [5.8981 ns 5.9204 ns 5.9443 ns] change: [-3.1620% -1.8780% -0.5792%] (p = 0.00 < 0.05) Change within noise threshold. Found 4 outliers among 100 measurements (4.00%) 2 (2.00%) high mild 2 (2.00%) high severe ``` So, we're not quite at rank9 yet but we're quite close.
- Change raw selection in enum_code to loop over the bits (skipping over zeros with clz) since broadword is really slow - Improve QC tests to randomize blocks and have lengths that aren't multiples of 64 - Fix bug in enum_code::select0 tests
Nope, sorry, I just haven’t had I time to look at it yet! Should get to it quite soon.
On Tue, Nov 19, 2019 at 12:32 PM Sujay Jayakar ***@***.***> wrote: @tov <https://github.com/tov>, ping on this? Let me know if there's anything I can help with. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10?email_source=notifications&email_token=AAFS3BBV5ZUAJHGZ6QHJRTTQUQWKDA5CNFSM4JHFDJDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEPHTPQ#issuecomment-555645374>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFS3BENBVWU4CXL5OXBVBLQUQWKDANCNFSM4JHFDJDA> .
-- Dr. Jesse A. Tov Assistant Professor of Instruction Computer Science McCormick School of Engineering Northwestern University http://www.cs.northwestern.edu/~jesse/
@sujayakar There seems to be a change in behaviour with