Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Index for BIP 157 block filters #14121

Merged
merged 12 commits into from Apr 18, 2019
Merged

Index for BIP 157 block filters #14121

merged 12 commits into from Apr 18, 2019

Conversation

@jimpo
Copy link
Contributor

@jimpo jimpo commented Aug 31, 2018

This introduces a new BlockFilterIndex class, which is required for BIP 157 support.

The index is uses the asynchronous BaseIndex infrastructure driven by the ValidationInterface callbacks. Filters are stored sequentially in flat files and the disk location of each filter is indexed in LevelDB along with the filter hash and header. The index is designed to ensure persistence of filters reorganized out of the main chain to simplify the BIP 157 net implementation.

Stats (block height = 565500):

  • Syncing the index from scratch takes 45m
  • Total index size is 3.8 GiB
@jimpo jimpo force-pushed the bip157-index branch 2 times, most recently from 1eeef83 to 1c20791 Aug 31, 2018
@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Aug 31, 2018

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

No conflicts as of last run.

std::make_pair(pindex->GetBlockHash(), filter.GetHash()));
batch.Write(std::make_pair(DB_FILTER_HEADER, height_key),
std::make_pair(pindex->GetBlockHash(), filter.ComputeHeader(prev_header)));
return m_db->WriteBatch(batch);
Copy link
Contributor

@leishman leishman Aug 31, 2018

Out of scope for this PR, but is there a reason we can't write a batch of entries for more than a single block? Could we write 100 - 1000 blocks worth of entries in each batch write to speed up the migrations? This introduces some complexity, but perhaps it's worth it?

Copy link
Contributor Author

@jimpo jimpo Sep 2, 2018

Could be worth looking into during ThreadSync. As you note though, it's a fairly independent change.

src/index/blockfilter.h Outdated Show resolved Hide resolved
src/index/blockfilter.cpp Outdated Show resolved Hide resolved
src/index/blockfilter.cpp Outdated Show resolved Hide resolved
@gmaxwell
Copy link
Contributor

@gmaxwell gmaxwell commented Sep 1, 2018

Storing large records in leveldb is generally a bad idea. Is there a particular reason this doesn't work like the undo data?

@jimpo
Copy link
Contributor Author

@jimpo jimpo commented Sep 1, 2018

@gmaxwell If by that you mean writing the filters sequentially in flat files then indexing the disk positions in LevelDB, I hadn't considered that, but it may be worthwhile. The downside of course is additional complexity. What are you mostly concerned about, read or write performance? I'd want to benchmark reads and writes to determine if the DB value sizes are problematic before making the change. With filters on average being 2% of block size and a LevelDB file size limit of 2 MiB, each file could still store ~200 filters (ignoring keys and overhead and such).

@jimpo
Copy link
Contributor Author

@jimpo jimpo commented Sep 3, 2018

@gmaxwell I put together a (not-production-ready) branch to test your suggestion of writing filters to flat files. In sample size n=1 experiments, I measured that the time to write the entire block index was <1% faster using flat files, and reading 5,000 sequential filters (starting at height 500,000) was 11% slower. The total storage space consumed is nearly the same (3.4 GiB total). Happy to provide the log files/iPython notebooks I used if you'd like.

Given the additional complexity and absence of significantly improved performance, I think writing filters directly into LevelDB is the way to go.

@jimpo jimpo force-pushed the bip157-index branch 3 times, most recently from 5aebbb9 to 9b1e7a6 Sep 3, 2018
@@ -44,19 +54,16 @@ class GCSFilter
public:

/** Constructs an empty filter. */
GCSFilter(uint64_t siphash_k0 = 0, uint64_t siphash_k1 = 0, uint8_t P = 0, uint32_t M = 0);
GCSFilter(const Params& params = Params());
Copy link
Contributor

@practicalswift practicalswift Sep 5, 2018

Please make explicit :-)


bool BlockFilterIndex::Rewind(const CBlockIndex* current_tip, const CBlockIndex* new_tip)
{
assert(current_tip->GetAncestor(new_tip->nHeight) == new_tip);
Copy link
Contributor

@practicalswift practicalswift Sep 5, 2018

Assertions should not have side effects. Please move GetAncestor outside of assertion :-)

Copy link
Contributor Author

@jimpo jimpo Sep 5, 2018

What is the side effect? GetAncestor is a const method.

Copy link
Contributor

@practicalswift practicalswift Sep 10, 2018

@jimpo You're right! Forget my comment :-)

uint32_t GetN() const { return m_N; }
uint32_t GetM() const { return m_M; }
const Params& GetParams() const { return m_params; }
Copy link
Contributor

@practicalswift practicalswift Sep 5, 2018

Remove GetParams()? Not used?

Copy link
Contributor Author

@jimpo jimpo Sep 5, 2018

No, it's not used, but it feels like there should be a getter.

Copy link
Contributor

@practicalswift practicalswift Sep 6, 2018

Unused code is untested code, so I suggest removing it or adding a test for it :-)

self.num_nodes = 2
self.extra_args = [["-blockfilterindex"], []]

def run_test (self):
Copy link
Contributor

@practicalswift practicalswift Sep 7, 2018

./test/functional/rpc_getblockfilter.py:21:17: E211 whitespace before '('

Copy link
Member

@Sjors Sjors left a comment

Concept ACK.

Can you mention -blockfilterindex and getblockfilter in the PR description, as well as perhaps add a release note?

It's nice to be able to quickly delete an index, so having a separate file for each type makes sense to me (as is the case now: indexes/blockindex/basic/...).

Lightly tested on macOS. I get a few mismatching headers compared to the test vectors:

src/bitcoin-cli getblockfilter 00000000fd3ceb2404ff07a785c7fdcc76619edc8ed61bd25134eaa22084366a "basic"
{
  "filter": "0db414c859a07e8205876354a210a75042d0463404913d61a8e068e58a3ae2aa080026",
  "header": "c582d51c0ca365e3fcf36c51cb646d7f83a67e867cb4743fd2128e3e022b700c"
}

000000000000015d6077a411a8f5cc95caf775ccf11c54e27df75ce58d187313 
-> "header": "546c574a0472144bcaf9b6aeabf26372ad87c7af7d1ee0dbfae5e099abeae49c"

0000000000000c00901f2049055e2a437c819d79a3d54fd63e6af796cd7b8a79
-> "header": "0965a544743bbfa36f254446e75630c09404b3d164a261892372977538928ed5

The filters do match.

We should probably include those test vectors. In addition, it would be nice to have script to compare every single block with the btcd RPC, testnet and mainnet.

src/index/blockfilter.cpp Outdated Show resolved Hide resolved
ghost
ghost approved these changes Sep 8, 2018
Copy link

@ghost ghost left a comment

good idea

MarcoFalke added a commit that referenced this issue Oct 17, 2019
f59bbb6 test: Fix bug in blockfilter_index_tests. (Jim Posen)

Pull request description:

  The test case tests a chain reorganization, however the two chains were generated in the same manner and thus produced the same blocks.

  This issue was [pointed out](#14121 (comment)) by MarcoFalke.

ACKs for top commit:
  MarcoFalke:
    Thanks! ACK f59bbb6 (looked at the diff on GitHub, didn't compile, nor run tests)

Tree-SHA512: a2f063ae9312051ffc2a3fcc1116a6a8ac09beeef261bc40aa3ff7270ff4de22a790eb19fec6b15ba1eb46e78f1f317bfd91472d8581b95bb9441a56b102554e
sidhujag added a commit to syscoin/syscoin that referenced this issue Oct 18, 2019
f59bbb6 test: Fix bug in blockfilter_index_tests. (Jim Posen)

Pull request description:

  The test case tests a chain reorganization, however the two chains were generated in the same manner and thus produced the same blocks.

  This issue was [pointed out](bitcoin#14121 (comment)) by MarcoFalke.

ACKs for top commit:
  MarcoFalke:
    Thanks! ACK f59bbb6 (looked at the diff on GitHub, didn't compile, nor run tests)

Tree-SHA512: a2f063ae9312051ffc2a3fcc1116a6a8ac09beeef261bc40aa3ff7270ff4de22a790eb19fec6b15ba1eb46e78f1f317bfd91472d8581b95bb9441a56b102554e
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 3, 2020
Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@4368384

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6332
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 3, 2020
Summary:
This is partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@62b7a4f

Depends on D6332

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6333
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 3, 2020
Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@ba6ff9a

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6335
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 3, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@75a76e3

Depends on D6335 and D6333

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Subscribers: jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6337
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 3, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@b5e8200

Depends on D6338 and D6337

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6339
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 3, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@6bcf099

Depends on D6339

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6340
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 3, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@2bc90e4

Depends on D6340

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6341
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 3, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@accc8b8

Depends on D6341 and D6344

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D6345
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 4, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@ff35105

Depends on D6345

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Subscribers: jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6346
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 4, 2020
Summary:
Retrieves and returns block filter and header from index.

This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@19308c9

Depends on D6346

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Subscribers: jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6349
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Jun 4, 2020
Summary:
New tests for the case of non-standard OP_RETURN outputs.

This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@c7efb65

Depends on D6349

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6351
ftrader added a commit to bitcoin-cash-node/bitcoin-cash-node that referenced this issue Aug 17, 2020
Summary: This is a partial backport of Core [[bitcoin/bitcoin#14121 | PR14121]] : bitcoin/bitcoin@4368384

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, jasonbcox

Reviewed By: #bitcoin_abc, jasonbcox

Differential Revision: https://reviews.bitcoinabc.org/D6332
ftrader pushed a commit to bitcoin-cash-node/bitcoin-cash-node that referenced this issue Aug 17, 2020
Summary: This is a partial backport of Core [PR14121](bitcoin/bitcoin#14121) - `ba6ff9a6f70139594362b4b4a6b816707bb165c8`

Test Plan:
```
ninja all check-all
```

Differential Revision: https://reviews.bitcoinabc.org/D6335
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 2, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 2, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 3, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 8, 2021
UdjinM6 added a commit to UdjinM6/dash that referenced this issue Aug 11, 2021
kittywhiskers added a commit to kittywhiskers/dash that referenced this issue Aug 12, 2021
Co-authored-by: UdjinM6 <UdjinM6@users.noreply.github.com>
UdjinM6 added a commit to dashpay/dash that referenced this issue Aug 13, 2021
Munkybooty added a commit to Munkybooty/dash that referenced this issue Sep 30, 2021
…_tests

89e8df1 tests: fix outdate include in blockfilter_index_tests (James O'Beirne)

Pull request description:

  Build is currently failing due to bad merge of bitcoin#15788 and bitcoin#14121.

ACKs for commit 89e8df:
  fanquake:
    tACK 89e8df1

Tree-SHA512: d3fea861f80d660b4a2827ca7241237311b68de4175d3db938a9a1d538e1325822410c98d84ba0734208af8163fbcc42cf2732788311ea22f3834c95eeb330b8
Munkybooty added a commit to Munkybooty/dash that referenced this issue Oct 7, 2021
…_tests

89e8df1 tests: fix outdate include in blockfilter_index_tests (James O'Beirne)

Pull request description:

  Build is currently failing due to bad merge of bitcoin#15788 and bitcoin#14121.

ACKs for commit 89e8df:
  fanquake:
    tACK 89e8df1

Tree-SHA512: d3fea861f80d660b4a2827ca7241237311b68de4175d3db938a9a1d538e1325822410c98d84ba0734208af8163fbcc42cf2732788311ea22f3834c95eeb330b8
Munkybooty added a commit to Munkybooty/dash that referenced this issue Oct 7, 2021
…_tests

89e8df1 tests: fix outdate include in blockfilter_index_tests (James O'Beirne)

Pull request description:

  Build is currently failing due to bad merge of bitcoin#15788 and bitcoin#14121.

ACKs for commit 89e8df:
  fanquake:
    tACK 89e8df1

Tree-SHA512: d3fea861f80d660b4a2827ca7241237311b68de4175d3db938a9a1d538e1325822410c98d84ba0734208af8163fbcc42cf2732788311ea22f3834c95eeb330b8
Munkybooty added a commit to Munkybooty/dash that referenced this issue Oct 12, 2021
…_tests

89e8df1 tests: fix outdate include in blockfilter_index_tests (James O'Beirne)

Pull request description:

  Build is currently failing due to bad merge of bitcoin#15788 and bitcoin#14121.

ACKs for commit 89e8df:
  fanquake:
    tACK 89e8df1

Tree-SHA512: d3fea861f80d660b4a2827ca7241237311b68de4175d3db938a9a1d538e1325822410c98d84ba0734208af8163fbcc42cf2732788311ea22f3834c95eeb330b8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet