Skip to content
This repository has been archived by the owner on Jul 18, 2024. It is now read-only.

opt(bitmap): Lazy cardinality calculations in FastOr #6

Merged
merged 4 commits into from
May 19, 2021
Merged

Conversation

manishrjain
Copy link
Contributor

@manishrjain manishrjain commented May 19, 2021

Calculate cardinality at the end, instead of calculating it for every bitmap container OR. This is equivalent of the lazyOr in official roaring library.

This shows a 20% speed improvement for 50-th %-ile based on the data below, and makes it 3x faster than roaring64.

RealDataFastOr/census-income_srt-32         1.55ms ± 1%    1.00ms ± 1%  -35.15%  (p=0.016 n=5+4)
RealDataFastOr/census-income-32             1.65ms ± 3%    1.20ms ± 9%  -27.13%  (p=0.008 n=5+5)
RealDataFastOr/census1881_srt-32            3.00ms ± 2%    3.06ms ± 2%     ~     (p=0.095 n=5+5)
RealDataFastOr/census1881-32                2.94ms ± 2%    2.96ms ± 7%     ~     (p=1.000 n=5+5)
RealDataFastOr/dimension_003-32             13.7ms ± 2%    12.3ms ± 0%   -9.84%  (p=0.016 n=5+4)
RealDataFastOr/dimension_008-32             4.41ms ± 4%    3.75ms ± 3%  -15.01%  (p=0.008 n=5+5)
RealDataFastOr/dimension_033-32             3.54ms ± 3%    2.65ms ± 1%  -25.16%  (p=0.008 n=5+5)
RealDataFastOr/uscensus2000-32               780µs ± 2%     800µs ± 1%   +2.49%  (p=0.008 n=5+5)
RealDataFastOr/weather_sept_85_srt-32       4.10ms ± 2%    2.73ms ± 2%  -33.40%  (p=0.008 n=5+5)
RealDataFastOr/weather_sept_85-32           6.35ms ± 2%    4.76ms ±10%  -25.11%  (p=0.008 n=5+5)
RealDataFastOr/wikileaks-noquotes_srt-32    1.09ms ± 2%    1.06ms ± 3%   -2.90%  (p=0.008 n=5+5)
RealDataFastOr/wikileaks-noquotes-32        1.07ms ± 1%    0.91ms ± 2%  -15.24%  (p=0.008 n=5+5)

This change is Reviewable

@manishrjain manishrjain merged commit e39ea03 into main May 19, 2021
@manishrjain manishrjain deleted the mrjn/lazy branch May 19, 2021 00:50
@manishrjain manishrjain changed the title opt(bitmap): Allow lazy cardinality calculations opt(bitmap): Lazy cardinality calculations in FastOr May 19, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
1 participant