Bloom filters #1795

TheBlueMatt · 2012-09-06T19:49:08Z

Filter tx invs and add MSG_FILTERED_BLOCK to provide blocks as header+vector (not including tx itself)

BitcoinPullTester · 2012-09-08T04:32:03Z

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/bd93f61c2abe5b19e77030fa5859bc370c789a3a for binaries and test log.

gavinandresen · 2012-09-18T23:19:32Z

Very exciting.

Did you see this:
http://matthias.vallentin.net/blog/2011/06/a-garden-variety-of-bloom-filters/

He has a really interesting looking C++11 headers-only implementation here at github that might give you inspiration (e.g. I think he uses boost to do the hashing; he seems to get a ton done in a very small number of lines of code).
https://github.com/mavam/libbf

jgarzik · 2012-09-27T01:09:24Z

implementation ACK, I like it a lot. This seems to match the initial sketch from @mikehearn and myself on IRC.

jgarzik · 2012-10-03T21:25:09Z

Poke, let's get some additional review on this.

mikehearn · 2012-10-24T13:11:43Z

Hey Matt, could you rebase this on top of ultraprune? I'm ready to start merging the bitcoinj side but want to test this with the latest code. Will do a review now-ish.

mikehearn · 2012-10-24T13:45:47Z

ACK on this. Most of my comments are purely cosmetic except for the unit test commit being in the wrong place (it wouldn't compile in the order given).

I guess we'll need a BIP? I'll start on one that documents what we've come up with.

TheBlueMatt · 2012-11-02T20:45:28Z

Thanks mike for the good review (as usual). I updated the code to addresses most of the issues you mentioned.
Changes:

many additional comments
filteradd is now limited to 1MB, any larger and the node gets a Misbehaving(100).
fRelayTxn now defaults to false - this means we will no longer relay tx inv message before receiving the remote peer's version message, if this irks anyone it can be discussed.
More changes to come (sipa's new merkle block stuff, more unit tests and a seed twiddle value for bloom filters)

TheBlueMatt · 2012-11-02T20:58:24Z

@mikehearn there are a few comments which note what the protocol defines as spec (these are obviously up for discussion):

there is no formal definition for filter parameters for a filteradd command if no filter is loaded - it is up to the serving node (if the client doest care, why should the spec determine the parameters?)
as noted above, 1MB limit to filteradd
txes in MSG_FILTERED_BLOCK can be relayed even when a node already has seen it (the current code does this) however it MUST always forward txn that the node has not seen (I think the bip says this, but its not really clearly worded imho, I updated the BIP to make this more clear)

BitcoinPullTester · 2012-11-02T21:12:58Z

Automatic sanity-testing: FAILED BUILD/TEST, see http://jenkins.bluematt.me/pull-tester/1860b72e356f0d4279b25bf1ca562ac00511eded for binaries and test log.

This could happen for one of several reasons:

It chanages paths in makefile.linux-mingw or otherwise changes build scripts in a way that made them incompatible with the automated testing scripts
It does not build on either Linux i386 or Win32 (via MinGW cross compile)
The test suite fails on either Linux i386 or Win32
The block test-cases failed (lookup the first bNN identifier which failed in https://github.com/TheBlueMatt/test-scripts/blob/master/FullBlockTestGenerator.java)

TheBlueMatt · 2012-11-02T21:18:01Z

@BitcoinPullTester hey, I wasnt done yet!

BitcoinPullTester · 2012-11-03T11:23:12Z

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/5be9d251b5901399c6f70b99f1c2bddc4e14e7ec for binaries and test log.

TheBlueMatt · 2012-11-12T00:01:52Z

Added the requested nTweak value and merged sipa's work on partial merkle tree representations. The BIP now needs rewritten and this is ready for second (third?) round reviews.

sipa · 2012-11-18T19:55:34Z

Mostly ACK;

I'd prefer to have the Murmurhash3 implementation separate from CBloomFilter
The 0xFFFFFFFF/(n-1)_i seed value seems intended to result in large bit differences between the different hash function's seeds. Together with the tweak, however, the first and the last now get seeds tweak and tweak-1. I think something simpler like k1_i+k2*n+tweak is better (with k1 and k2 arbitrary large odd 32-bit integers).
I feel uneasy about the arbitrary filter parameters used for the implicitly created filter when sending filteradd without filterload. The server cannot be expected to make a reasonable guess how the client is going to use the filter, and the client still has to track how large the server-side filter grows anyway. I'd prefer to make this simply illegal (DoS 100): filteradd always requires an active filter. Maybe the actual filter data in filterload can be made optional: if it is omitted, it's assumed to be all zeroes (though that would mean the size has to be specified).

BitcoinPullTester · 2012-11-27T18:56:58Z

Automatic sanity-testing: FAILED MERGE, see http://jenkins.bluematt.me/pull-tester/21e780e05c7b18ab318e57bd9e24961a34378e2d for test log.

This pull does not merge cleanly onto current master

BitcoinPullTester · 2012-12-18T09:09:34Z

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/a5a72aa691296af6de92658ff998da120c4b34d8 for binaries and test log.

mikehearn · 2013-01-04T13:27:01Z

src/main.cpp

+
+        // Nodes must NEVER send a data item > 520 bytes (the max size for a script data object,
+        // and thus, the maximum size any matched object can have) in a filteradd message
+        if (vData.size() > 520)


If vData is too large, it will still be added to the filter. There needs to be an else block here.

yep, fixed.

Wouldn't it be nice if that 520 was a constant somewhere or a define? So it could be used for the max size for a script data object and in here.

mikehearn · 2013-01-09T14:34:34Z

Can you add a new feature that lets clients opt out of auto-adding? Sipa points out that it's only necessary for wallets that contain outputs which don't require predictable data in the corresponding inputs, which (for now) is most of them. This should resolve the excessive expansion issues.

gmaxwell · 2013-01-09T14:47:26Z

@mikehearn Kind of an an unfortunate privacy loss for ones who do set it, and it still leaves the feature potentially useless for any who set it. (But I agree on having the option, if for nothing else because it makes the load-balancing lark I suggested viable)

mikehearn · 2013-01-09T15:11:28Z

Well not really, you can still set the FP rate to whatever you want. If all
your outputs are pay-to-address then it's a no-op functionality-wise, the
auto-adding didn't buy you anything. For the case where you have
pay-to-pubkey or other scripts, the client has to refresh the filter every
so often .... it can be implemented in bcj later.

gmaxwell · 2013-01-09T18:09:18Z

Setting that bit or not is still another anonymity set reduction, but I'm splitting hairs there ignore it. I guess I don't have a good feel for how fast the filter fails and needs to be reset. If it's too fast periodic refreshes won't be sufficient (E.g. will the size of the refresh plus the unwanted data end up being more than just not using the filter).

mikehearn · 2013-01-09T19:18:00Z

I think we can explore different algorithms later. The exact choices will
depend on the users privacy preferences and device situation anyway.

On Wed, Jan 9, 2013 at 7:09 PM, Gregory Maxwell notifications@github.comwrote:

Setting that bit or not is still another anonymity set reduction, but I'm
splitting hairs there ignore it. I guess I don't have a good feel for how
fast the filter fails and needs to be reset. If it's too fast periodic
refreshes won't be sufficient (E.g. will the size of the refresh plus the
unwanted data end up being more than just not using the filter).

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1795#issuecomment-12058432.

TheBlueMatt · 2013-01-11T03:44:02Z

Added a nFlags to let the peer pick how/when it wants the filter updated...also added a second two commits to make it possible to match an arbitrary script template as a discussion point (that second part is not yet tested).
Would like comments.

gavinandresen · 2013-01-16T16:19:11Z

Code review notes:

I don't like the entire implementation of CPartialMerkleTree being stuffed into main.h, it should be its own .cpp/.h

Message handling looks good from a potential vulnerability point of view.

NACK on the script.cpp / script.h changes -- they are unused (yes? I don't see where MatchesTemplate is used outside of script.cpp/.h) and I don't see unit tests for the new features like OP_OPCODE.

TheBlueMatt · 2013-01-16T16:58:49Z

@gavinandresen Yes, after discussion I believe we are currently targeting skipping the 2 MatchesTemplate commits for 0.8 and maybe skipping those entirely depending on what we may need in the future.

Note that the default value for fRelayTxes is false, meaning we now no longer relay tx inv messages before receiving the remote peer's version message.

This adds a compact representation for a subset of a merkle tree's nodes.

BitcoinPullTester · 2013-01-16T19:49:05Z

Automatic sanity-testing: PASSED, see http://jenkins.bluematt.me/pull-tester/e1a4f3778cb90ba9f0d4e736752f78dad1703caa for binaries and test log.

TheBlueMatt · 2013-01-16T19:52:54Z

@gavinandresen moved it to main.cpp, is that ok for you?

gavinandresen · 2013-01-16T20:59:45Z

ACK, looks good.

Bloom filters

rebroad · 2013-03-27T09:45:37Z

So.. what was the BIP for these protocol changes? It would be nice if it could be referenced in the pull requests.

rebroad · 2013-03-27T09:51:51Z

src/main.cpp

@@ -2815,6 +2969,10 @@ bool static ProcessMessage(CNode* pfrom, string strCommand, CDataStream& vRecv)
            vRecv >> pfrom->strSubVer;
        if (!vRecv.empty())
            vRecv >> pfrom->nStartingHeight;
+        if (!vRecv.empty())
+            vRecv >> pfrom->fRelayTxes; // set to true after we get the first filter* message


Somehow this doesn't feel like the best way to implement this....

mikehearn · 2013-03-27T10:26:33Z

https://en.bitcoin.it/wiki/BIP_0037 as was discussed on the mailing list. Regardless, this code was already merged and shipped months ago, it's a bit late to be commenting on it now. You can always open up new pull reqs if you want to change how the protocol works.

Bloom filters

mikehearn reviewed Jan 4, 2013
View reviewed changes

TheBlueMatt added 5 commits January 16, 2013 12:48

Add const versions of base_uint.end()/begin(), make size() const.

68feac9

Add MurmurHash3 implementation to hash.h/add hash.cpp.

7ab026f

Add a CBloomFilter class for use as a transaction filter.

bd21612

Bump PROTOCOL_VERSION for filter messages.

133a546

Add a filter field in CNode, add filterload+filteradd+filterclear

422d122

TheBlueMatt and others added 12 commits January 16, 2013 12:48

Replace RelayMessage with RelayTransaction.

269d9c6

Automatically add any matching outputs to a filter during matching.

d3b26f7

Add a CBlock.GetBlockHeader

587f0f8

Add a CMerkleBlock to store merkle branches of filtered txes.

9fb106e

Add test cases for CMerkleBlock and CBloomFilter.

2878c67

Relay CMerkleBlocks when asked for MSG_FILTERED_BLOCK

b02ddbe

Let a node opt out of tx invs before we get a their bloom filter

4c8fc1a

Note that the default value for fRelayTxes is false, meaning we now no longer relay tx inv messages before receiving the remote peer's version message.

Add a nTweak to bloom filters to tweak the seed.

b1f99be

Add CPartialMerkleTree

4bedfa9

This adds a compact representation for a subset of a merkle tree's nodes.

Use CPartialMerkleTree for CMerkleBlock transactions.

21aaf25

Add nFlags to CBloomFilter to make filter updating optional.

e1a4f37

Filter mempool command

c51694e

gavinandresen added a commit that referenced this pull request Jan 17, 2013

Merge pull request #1795 from TheBlueMatt/bloom

91f70a7

Bloom filters

gavinandresen merged commit 91f70a7 into bitcoin:master Jan 17, 2013

rebroad reviewed Mar 27, 2013
View reviewed changes

langerhans mentioned this pull request Jan 3, 2014

Implement bloom filtering for greatly reduced bandwidth usage dogecoin/dogecoin#88

Closed

laudney pushed a commit to reddcoin-project/reddcoin-3.10 that referenced this pull request Mar 19, 2014

Merge pull request bitcoin#1795 from TheBlueMatt/bloom

2481237

Bloom filters

fanquake mentioned this pull request Feb 2, 2015

Add list of implemented BIPs #5708

Merged

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bloom filters #1795

Bloom filters #1795

TheBlueMatt commented Sep 6, 2012

BitcoinPullTester commented Sep 8, 2012

gavinandresen commented Sep 18, 2012

jgarzik commented Sep 27, 2012

jgarzik commented Oct 3, 2012

mikehearn commented Oct 24, 2012

mikehearn commented Oct 24, 2012

TheBlueMatt commented Nov 2, 2012

TheBlueMatt commented Nov 2, 2012

BitcoinPullTester commented Nov 2, 2012

TheBlueMatt commented Nov 2, 2012

BitcoinPullTester commented Nov 3, 2012

TheBlueMatt commented Nov 12, 2012

sipa commented Nov 18, 2012

BitcoinPullTester commented Nov 27, 2012

BitcoinPullTester commented Dec 18, 2012

mikehearn Jan 4, 2013

TheBlueMatt Jan 7, 2013

Diapolo Jan 11, 2013

mikehearn commented Jan 9, 2013

gmaxwell commented Jan 9, 2013

mikehearn commented Jan 9, 2013

gmaxwell commented Jan 9, 2013

mikehearn commented Jan 9, 2013

TheBlueMatt commented Jan 11, 2013

gavinandresen commented Jan 16, 2013

TheBlueMatt commented Jan 16, 2013

BitcoinPullTester commented Jan 16, 2013

TheBlueMatt commented Jan 16, 2013

gavinandresen commented Jan 16, 2013

rebroad commented Mar 27, 2013

rebroad Mar 27, 2013

mikehearn commented Mar 27, 2013

Bloom filters #1795

Bloom filters #1795

Conversation

TheBlueMatt commented Sep 6, 2012

BitcoinPullTester commented Sep 8, 2012

gavinandresen commented Sep 18, 2012

jgarzik commented Sep 27, 2012

jgarzik commented Oct 3, 2012

mikehearn commented Oct 24, 2012

mikehearn commented Oct 24, 2012

TheBlueMatt commented Nov 2, 2012

TheBlueMatt commented Nov 2, 2012

BitcoinPullTester commented Nov 2, 2012

TheBlueMatt commented Nov 2, 2012

BitcoinPullTester commented Nov 3, 2012

TheBlueMatt commented Nov 12, 2012

sipa commented Nov 18, 2012

BitcoinPullTester commented Nov 27, 2012

BitcoinPullTester commented Dec 18, 2012

mikehearn Jan 4, 2013

Choose a reason for hiding this comment

TheBlueMatt Jan 7, 2013

Choose a reason for hiding this comment

Diapolo Jan 11, 2013

Choose a reason for hiding this comment

mikehearn commented Jan 9, 2013

gmaxwell commented Jan 9, 2013

mikehearn commented Jan 9, 2013

gmaxwell commented Jan 9, 2013

mikehearn commented Jan 9, 2013

TheBlueMatt commented Jan 11, 2013

gavinandresen commented Jan 16, 2013

TheBlueMatt commented Jan 16, 2013

BitcoinPullTester commented Jan 16, 2013

TheBlueMatt commented Jan 16, 2013

gavinandresen commented Jan 16, 2013

rebroad commented Mar 27, 2013

rebroad Mar 27, 2013

Choose a reason for hiding this comment

mikehearn commented Mar 27, 2013