Index for BIP 157 block filters #14121

jimpo · 2018-08-31T21:28:20Z

This introduces a new BlockFilterIndex class, which is required for BIP 157 support.

The index is uses the asynchronous BaseIndex infrastructure driven by the ValidationInterface callbacks. Filters are stored sequentially in flat files and the disk location of each filter is indexed in LevelDB along with the filter hash and header. The index is designed to ensure persistence of filters reorganized out of the main chain to simplify the BIP 157 net implementation.

Stats (block height = 565500):

Syncing the index from scratch takes 45m
Total index size is 3.8 GiB

DrahtBot · 2018-08-31T22:47:52Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

leishman · 2018-08-31T23:31:07Z

src/index/blockfilter.cpp

+                std::make_pair(pindex->GetBlockHash(), filter.GetHash()));
+    batch.Write(std::make_pair(DB_FILTER_HEADER, height_key),
+                std::make_pair(pindex->GetBlockHash(), filter.ComputeHeader(prev_header)));
+    return m_db->WriteBatch(batch);


Out of scope for this PR, but is there a reason we can't write a batch of entries for more than a single block? Could we write 100 - 1000 blocks worth of entries in each batch write to speed up the migrations? This introduces some complexity, but perhaps it's worth it?

Could be worth looking into during ThreadSync. As you note though, it's a fairly independent change.

src/index/blockfilter.h

src/index/blockfilter.cpp

gmaxwell · 2018-09-01T00:17:26Z

Storing large records in leveldb is generally a bad idea. Is there a particular reason this doesn't work like the undo data?

jimpo · 2018-09-01T00:47:26Z

@gmaxwell If by that you mean writing the filters sequentially in flat files then indexing the disk positions in LevelDB, I hadn't considered that, but it may be worthwhile. The downside of course is additional complexity. What are you mostly concerned about, read or write performance? I'd want to benchmark reads and writes to determine if the DB value sizes are problematic before making the change. With filters on average being 2% of block size and a LevelDB file size limit of 2 MiB, each file could still store ~200 filters (ignoring keys and overhead and such).

jimpo · 2018-09-03T01:45:53Z

@gmaxwell I put together a (not-production-ready) branch to test your suggestion of writing filters to flat files. In sample size n=1 experiments, I measured that the time to write the entire block index was <1% faster using flat files, and reading 5,000 sequential filters (starting at height 500,000) was 11% slower. The total storage space consumed is nearly the same (3.4 GiB total). Happy to provide the log files/iPython notebooks I used if you'd like.

Given the additional complexity and absence of significantly improved performance, I think writing filters directly into LevelDB is the way to go.

practicalswift · 2018-09-05T07:35:45Z

src/blockfilter.h

@@ -44,19 +54,16 @@ class GCSFilter
 public:

    /** Constructs an empty filter. */
-    GCSFilter(uint64_t siphash_k0 = 0, uint64_t siphash_k1 = 0, uint8_t P = 0, uint32_t M = 0);
+    GCSFilter(const Params& params = Params());


Please make explicit :-)

practicalswift · 2018-09-05T07:37:09Z

src/index/blockfilter.cpp

+
+bool BlockFilterIndex::Rewind(const CBlockIndex* current_tip, const CBlockIndex* new_tip)
+{
+    assert(current_tip->GetAncestor(new_tip->nHeight) == new_tip);


Assertions should not have side effects. Please move GetAncestor outside of assertion :-)

What is the side effect? GetAncestor is a const method.

@jimpo You're right! Forget my comment :-)

practicalswift · 2018-09-05T07:38:03Z

src/blockfilter.h

    uint32_t GetN() const { return m_N; }
-    uint32_t GetM() const { return m_M; }
+    const Params& GetParams() const { return m_params; }


Remove GetParams()? Not used?

No, it's not used, but it feels like there should be a getter.

Unused code is untested code, so I suggest removing it or adding a test for it :-)

practicalswift · 2018-09-07T07:43:57Z

test/functional/rpc_getblockfilter.py

+        self.num_nodes = 2
+        self.extra_args = [["-blockfilterindex"], []]
+
+    def run_test (self):


./test/functional/rpc_getblockfilter.py:21:17: E211 whitespace before '('

Sjors

Concept ACK.

Can you mention -blockfilterindex and getblockfilter in the PR description, as well as perhaps add a release note?

It's nice to be able to quickly delete an index, so having a separate file for each type makes sense to me (as is the case now: indexes/blockindex/basic/...).

Lightly tested on macOS. I get a few mismatching headers compared to the test vectors:

src/bitcoin-cli getblockfilter 00000000fd3ceb2404ff07a785c7fdcc76619edc8ed61bd25134eaa22084366a "basic"
{
  "filter": "0db414c859a07e8205876354a210a75042d0463404913d61a8e068e58a3ae2aa080026",
  "header": "c582d51c0ca365e3fcf36c51cb646d7f83a67e867cb4743fd2128e3e022b700c"
}

000000000000015d6077a411a8f5cc95caf775ccf11c54e27df75ce58d187313 
-> "header": "546c574a0472144bcaf9b6aeabf26372ad87c7af7d1ee0dbfae5e099abeae49c"

0000000000000c00901f2049055e2a437c819d79a3d54fd63e6af796cd7b8a79
-> "header": "0965a544743bbfa36f254446e75630c09404b3d164a261892372977538928ed5

The filters do match.

We should probably include those test vectors. In addition, it would be nice to have script to compare every single block with the btcd RPC, testnet and mainnet.

src/index/blockfilter.cpp

ghost

good idea

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@4368384 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, jasonbcox Reviewed By: #bitcoin_abc, jasonbcox Differential Revision: https://reviews.bitcoinabc.org/D6332

Summary: This is partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@62b7a4f Depends on D6332 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, jasonbcox Reviewed By: #bitcoin_abc, jasonbcox Differential Revision: https://reviews.bitcoinabc.org/D6333

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@ba6ff9a Test Plan: ninja all check-all Reviewers: #bitcoin_abc, jasonbcox Reviewed By: #bitcoin_abc, jasonbcox Differential Revision: https://reviews.bitcoinabc.org/D6335

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@75a76e3 Depends on D6335 and D6333 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, jasonbcox Reviewed By: #bitcoin_abc, jasonbcox Subscribers: jasonbcox Differential Revision: https://reviews.bitcoinabc.org/D6337

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@b5e8200 Depends on D6338 and D6337 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D6339

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@6bcf099 Depends on D6339 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D6340

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@2bc90e4 Depends on D6340 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, Fabien Reviewed By: #bitcoin_abc, Fabien Differential Revision: https://reviews.bitcoinabc.org/D6341

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@accc8b8 Depends on D6341 and D6344 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, majcosta Reviewed By: #bitcoin_abc, majcosta Differential Revision: https://reviews.bitcoinabc.org/D6345

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@ff35105 Depends on D6345 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, jasonbcox Reviewed By: #bitcoin_abc, jasonbcox Subscribers: jasonbcox Differential Revision: https://reviews.bitcoinabc.org/D6346

Summary: Retrieves and returns block filter and header from index. This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@19308c9 Depends on D6346 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, jasonbcox Reviewed By: #bitcoin_abc, jasonbcox Subscribers: jasonbcox Differential Revision: https://reviews.bitcoinabc.org/D6349

Summary: New tests for the case of non-standard OP_RETURN outputs. This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@c7efb65 Depends on D6349 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, jasonbcox Reviewed By: #bitcoin_abc, jasonbcox Differential Revision: https://reviews.bitcoinabc.org/D6351

Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@4368384 Test Plan: ninja all check-all Reviewers: #bitcoin_abc, jasonbcox Reviewed By: #bitcoin_abc, jasonbcox Differential Revision: https://reviews.bitcoinabc.org/D6332

Summary: This is a partial backport of Core [PR14121](bitcoin/bitcoin#14121) - `ba6ff9a6f70139594362b4b4a6b816707bb165c8` Test Plan: ``` ninja all check-all ``` Differential Revision: https://reviews.bitcoinabc.org/D6335

Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>

merge bitcoin#15118, bitcoin#14172, bitcoin#15623, bitcoin#14121, partial bitcoin#13743, partial bitcoin#15280: block filters

…_tests 89e8df1 tests: fix outdate include in blockfilter_index_tests (James O'Beirne) Pull request description: Build is currently failing due to bad merge of bitcoin#15788 and bitcoin#14121. ACKs for commit 89e8df: fanquake: tACK 89e8df1 Tree-SHA512: d3fea861f80d660b4a2827ca7241237311b68de4175d3db938a9a1d538e1325822410c98d84ba0734208af8163fbcc42cf2732788311ea22f3834c95eeb330b8

jimpo force-pushed the bip157-index branch 2 times, most recently from 1eeef83 to 1c20791 Compare August 31, 2018 22:40

leishman reviewed Aug 31, 2018

View reviewed changes

src/index/blockfilter.h Outdated Show resolved Hide resolved

DrahtBot mentioned this pull request Aug 31, 2018

Preserve a format of RPC command definitions #14008

Merged

leishman reviewed Aug 31, 2018

View reviewed changes

src/index/blockfilter.cpp Outdated Show resolved Hide resolved

DrahtBot mentioned this pull request Aug 31, 2018

Refactoring CRPCCommand with enum category #13945

Closed

jimpo force-pushed the bip157-index branch from 1c20791 to b384c3e Compare August 31, 2018 23:59

jimpo commented Sep 1, 2018

View reviewed changes

src/index/blockfilter.cpp Outdated Show resolved Hide resolved

DrahtBot mentioned this pull request Sep 1, 2018

validation: Document where we are intentionally ignoring bool return values from validation related functions #13864

Closed

laanwj added the UTXO Db and Indexes label Sep 1, 2018

jimpo force-pushed the bip157-index branch 3 times, most recently from 5aebbb9 to 9b1e7a6 Compare September 3, 2018 18:28

practicalswift reviewed Sep 5, 2018

View reviewed changes

practicalswift reviewed Sep 7, 2018

View reviewed changes

Sjors reviewed Sep 7, 2018

View reviewed changes

src/index/blockfilter.cpp Outdated Show resolved Hide resolved

ghost approved these changes Sep 8, 2018

View reviewed changes

jimpo mentioned this pull request Sep 8, 2018

Refactor and add tests for BlockFilter construction #14172

Merged

nopara73 mentioned this pull request Feb 28, 2020

blockfilterindex Extensibility Proposal #18221

Closed

romanz mentioned this pull request Oct 24, 2020

electrs improvement proposal romanz/electrs#308

Closed

kwvg added a commit to kwvg/dash that referenced this pull request Aug 2, 2021

bitcoin#14121: Index for BIP 157 block filters

9dbd21b

kwvg added a commit to kwvg/dash that referenced this pull request Aug 2, 2021

bitcoin#14121: Index for BIP 157 block filters

b5d7fc6

kwvg added a commit to kwvg/dash that referenced this pull request Aug 3, 2021

bitcoin#14121: Index for BIP 157 block filters

1111a7f

kwvg added a commit to kwvg/dash that referenced this pull request Aug 8, 2021

bitcoin#14121: Index for BIP 157 block filters

bdd2c2b

UdjinM6 pushed a commit to UdjinM6/dash that referenced this pull request Aug 11, 2021

bitcoin#14121: Index for BIP 157 block filters

3b97c05

kwvg added a commit to kwvg/dash that referenced this pull request Aug 12, 2021

bitcoin#14121: Index for BIP 157 block filters

21c7e57

Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>

UdjinM6 added a commit to dashpay/dash that referenced this pull request Aug 13, 2021

Merge pull request #4314 from kittywhiskers/blockfilters

607bd4b

merge bitcoin#15118, bitcoin#14172, bitcoin#15623, bitcoin#14121, partial bitcoin#13743, partial bitcoin#15280: block filters

bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index for BIP 157 block filters #14121

Index for BIP 157 block filters #14121

jimpo commented Aug 31, 2018 •

edited

Loading

DrahtBot commented Aug 31, 2018 •

edited

Loading

leishman Aug 31, 2018

jimpo Sep 2, 2018

gmaxwell commented Sep 1, 2018

jimpo commented Sep 1, 2018

jimpo commented Sep 3, 2018 •

edited

Loading

practicalswift Sep 5, 2018

practicalswift Sep 5, 2018

jimpo Sep 5, 2018

practicalswift Sep 10, 2018

practicalswift Sep 5, 2018 •

edited

Loading

jimpo Sep 5, 2018

practicalswift Sep 6, 2018

practicalswift Sep 7, 2018

Sjors left a comment •

edited

Loading

ghost left a comment

Index for BIP 157 block filters #14121

Index for BIP 157 block filters #14121

Conversation

jimpo commented Aug 31, 2018 • edited Loading

DrahtBot commented Aug 31, 2018 • edited Loading

Conflicts

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmaxwell commented Sep 1, 2018

jimpo commented Sep 1, 2018

jimpo commented Sep 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

practicalswift Sep 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sjors left a comment • edited Loading

Choose a reason for hiding this comment

ghost left a comment

Choose a reason for hiding this comment

jimpo commented Aug 31, 2018 •

edited

Loading

DrahtBot commented Aug 31, 2018 •

edited

Loading

jimpo commented Sep 3, 2018 •

edited

Loading

practicalswift Sep 5, 2018 •

edited

Loading

Sjors left a comment •

edited

Loading