New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve rolling bloom filter performance and benchmark #7934

Merged
merged 2 commits into from May 9, 2016

Conversation

Projects
None yet
5 participants
@sipa
Member

sipa commented Apr 24, 2016

Added a benchmark for the rolling bloom filter (with parameters corresponding to the tx rejection cache), which showed that on average, adding+checking one item takes around 2.1us, but the refresh action that wipes all old generation items every 60000 iterations takes 65ms, which is very significant.

Thus, this patch also changes the implementation from one that stores 16 2-bit integers in uint32_t's, to one that stores the first bit of 64 2-bit integers in one uint64_t and the second bit in another. This allows for 450x faster refreshing (0.14ms) and 2.2x faster average adding+checking (0.93us).

All benchmarks done on an Intel Core i7-4800MQ CPU, running at 2.6 GHz, with binaries compiled with GCC 5.3.1.

@dcousens

View changes

Show outdated Hide outdated src/bloom.cpp
@dcousens

View changes

Show outdated Hide outdated src/bloom.cpp
uint64_t p1 = data[p], p2 = data[p + 1];
uint64_t mask = (p1 ^ nGenerationMask1) | (p2 ^ nGenerationMask2);
data[p] = p1 & mask;
data[p + 1] = p2 & mask;

This comment has been minimized.

@dcousens

dcousens Apr 26, 2016

Contributor

Nice branch removal :)

@dcousens

dcousens Apr 26, 2016

Contributor

Nice branch removal :)

@dcousens

This comment has been minimized.

Show comment
Hide comment
@dcousens

dcousens Apr 26, 2016

Contributor

light utACK 67d44f8, didn't verify masking operations

Contributor

dcousens commented Apr 26, 2016

light utACK 67d44f8, didn't verify masking operations

@laanwj laanwj referenced this pull request Apr 26, 2016

Closed

Add benchmarks to `bench_bitcoin` #7883

6 of 8 tasks complete

sipa added some commits Apr 24, 2016

More efficient bitsliced rolling Bloom filter
This patch changes the implementation from one that stores 16 2-bit integers
in one uint32_t's, to one that stores the first bit of 64 2-bit integers in
one uint64_t and the second bit in another. This allows for 450x faster
refreshing and 2.2x faster average speed.
@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Apr 28, 2016

Member

Addressed @dcousens's nits.

Member

sipa commented Apr 28, 2016

Addressed @dcousens's nits.

@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell Apr 28, 2016

Member

utACK.

Member

gmaxwell commented Apr 28, 2016

utACK.

@dcousens

This comment has been minimized.

Show comment
Hide comment
@dcousens

dcousens Apr 30, 2016

Contributor

utACK 1953c40

Contributor

dcousens commented Apr 30, 2016

utACK 1953c40

@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell May 5, 2016

Member

ACK. (appears to work.)

Member

gmaxwell commented May 5, 2016

ACK. (appears to work.)

@laanwj laanwj merged commit 1953c40 into bitcoin:master May 9, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request May 9, 2016

Merge #7934: Improve rolling bloom filter performance and benchmark
1953c40 More efficient bitsliced rolling Bloom filter (Pieter Wuille)
aa62b68 Benchmark rolling bloom filter (Pieter Wuille)
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj May 9, 2016

Member

utACK 1953c40

Member

laanwj commented May 9, 2016

utACK 1953c40

@rebroad

This comment has been minimized.

Show comment
Hide comment
@rebroad

rebroad Dec 21, 2016

Contributor

I'd like to understand this code - where do I start?

Contributor

rebroad commented Dec 21, 2016

I'd like to understand this code - where do I start?

uint32_t h = Hash(n, vKey);
put(h, nGeneration);
uint32_t h = RollingBloomHash(n, nTweak, vKey);
int bit = h & 0x3F;

This comment has been minimized.

@dcousens

dcousens Dec 21, 2016

Contributor

I wonder if it is confusing alternating between 0x3f and 63 a few times in this code...

@dcousens

dcousens Dec 21, 2016

Contributor

I wonder if it is confusing alternating between 0x3f and 63 a few times in this code...

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Dec 21, 2016

Member

I'd like to understand this code - where do I start?

In this case it is the theory that is important to understand. With that, the code is pretty straightforward. Google "bloom filters". Most notably the wikipedia page about Bloom filters has a lot of references to CS literature about various kinds of bloom filters, and the article itself may give basic understanding.

Member

laanwj commented Dec 21, 2016

I'd like to understand this code - where do I start?

In this case it is the theory that is important to understand. With that, the code is pretty straightforward. Google "bloom filters". Most notably the wikipedia page about Bloom filters has a lot of references to CS literature about various kinds of bloom filters, and the article itself may give basic understanding.

codablock added a commit to codablock/dash that referenced this pull request Sep 16, 2017

Merge #7934: Improve rolling bloom filter performance and benchmark
1953c40 More efficient bitsliced rolling Bloom filter (Pieter Wuille)
aa62b68 Benchmark rolling bloom filter (Pieter Wuille)

codablock added a commit to codablock/dash that referenced this pull request Sep 19, 2017

Merge #7934: Improve rolling bloom filter performance and benchmark
1953c40 More efficient bitsliced rolling Bloom filter (Pieter Wuille)
aa62b68 Benchmark rolling bloom filter (Pieter Wuille)

codablock added a commit to codablock/dash that referenced this pull request Dec 21, 2017

Merge #7934: Improve rolling bloom filter performance and benchmark
1953c40 More efficient bitsliced rolling Bloom filter (Pieter Wuille)
aa62b68 Benchmark rolling bloom filter (Pieter Wuille)

@dagurval dagurval referenced this pull request Jan 15, 2018

Merged

Bloom filter updates #297

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment