Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
BIP 158: Compact Block Filters for Light Clients #12254
This implements the compact block filter construction in BIP 158. The code is not used anywhere in the Bitcoin Core code base yet. The next step towards BIP 157 support would be to create an indexing module similar to
Here is a CSV of filter sizes for blocks in the main chain.
As you can see below, the ratio of filter size to block size drops after the first ~150,000 blocks:
The reason for the relatively large filter sizes is that Golomb-coded sets only achieve good compression with a sufficient number of elements. Empirically, the average element size with 100 elements is 14% larger than with 10,000 elements.
The ratio of filter size to block size is computed without witness data for basic filters. Here is a summary table of filter size ratios for blocks after height 150,000:
@laanwj This is a data structure to be used in a P2P change. I first thought that it shouldn't be tagged "Consensus", but there's an argument to be made for it. It doesn't affect blockchain consensus, but it is kind of a softer P2P consensus change, where network clients (though not other full nodes) may disconnect/ban you if you serve incorrectly computed block filters. I'll let you make the call on the tag.
referenced this pull request
Feb 9, 2018
@jonasschnelli Thanks for reviewing. The test vectors were generated from a Go program I have that cross-validates against the btcsuite implementation. I can easily add any specific testnet blocks to the list of cases. The blocks were chosen to exercise certain edge cases (eg. empty filters, duplicate pushdatas, invalid output scripts), but the vectors aren't commented with which edges cases they exercise. I'll add the comments, because it seems worthwhile.
@Sjors I'd definitely like to see RPC commands to fetch specific filters and filter headers, but I think it makes more sense to do that after adding the filter index, so that the RPC handlers just have to look up a precomputed filter/header. (So basically, in a subsequent PR).
utACK a23681b. Left minor comments (feel free to ignore). Overall code looks very good.
- f154ded streams: Create VectorReader stream interface for vectors. (1/14)
- bdb3419 streams: Unit test for VectorReader class. (2/14)
- faaa4b8 streams: Implement BitStreamReader/Writer classes. (3/14)
- e15e553 streams: Unit tests for BitStreamReader and BitStreamWriter. (4/14)
- a4b03be blockfilter: Declare GCSFilter class for BIP 158 impl. (5/14)
- 515af03 blockfilter: Implement GCSFilter constructors. (6/14)
- 6c1262f blockfilter: Implement GCSFilter Match methods. (7/14)
- e986211 blockfilter: Simple test for GCSFilter construction and Match. (8/14)
- f7a5bdb blockfilter: Construction of basic block filters. (9/14)
- c14e4d5 blockfilter: Serialization methods on BlockFilter. (10/14)
- ab852f9 blockfilter: Additional helper methods to compute hash and header. (11/14)
- 9cf17eb blockfilter: Unit test against BIP 158 test vectors. (12/14)
- 2837e40 blockfilter: Optimization on compilers with int128 support. (13/14)
- a23681b bench: Benchmark GCS filter creation and matching. (14/14)
Even a proof-of-concept PR for that would be useful for review.
@ryanofsky Thank you for the review. I'm going to leave
didn't want to merge it last-minute for 0.17, but it can go in now IMO