Serve cfcheckpt requests #18877

jnewbery · 2020-05-05T02:58:24Z

Serve cfcheckpt messages if basic block filter index is enabled and -peercfilters is set.

NODE_COMPACT_FILTERS is not signaled to peers, but functionality can be used for testing and serving pre-configured clients.

prusnak · 2020-05-05T08:46:53Z

Seems the lint target fails in the Travis CI.

DrahtBot · 2020-05-05T09:25:36Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

[WIP] Serve BIP 157 compact filters #18876 ([WIP] Serve BIP 157 compact filters by jnewbery)
Make g_chainman internal to validation #18698 (Make g_chainman internal to validation by MarcoFalke)
net: Use mockable time for ping/pong, add tests #18638 (net: Use mockable time for ping/pong, add tests by MarcoFalke)
Add BIP324 encrypted p2p transport de-/serializer (only used in tests) #18242 (Add BIP324 encrypted p2p transport de-/serializer (only used in tests) by jonasschnelli)
Return BlockValidationState from ProcessNewBlock if CheckBlock/AcceptBlock fails #17479 (Return BlockValidationState from ProcessNewBlock if CheckBlock/AcceptBlock fails by jnewbery)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

src/init.cpp

jnewbery · 2020-05-06T03:52:55Z

Note to reviewers: the Travis failures are timeouts and can be ignored. This PR is ready for review.

jkczyz

Concept ACK

src/protocol.h

src/net_processing.cpp

ariard · 2020-05-07T08:51:29Z

test/functional/p2p_cfilters.py

+    wait_until,
+)
+
+class CompactFiltersTest(BitcoinTestFramework):


Is this test verify the following spec requirement "StopHash MUST be known to belong to a block accepted by the receiving peer. This is the case if the peer had previously sent a headers or inv message with any descendent blocks" ?

This point is unclear to me with regards to the receiver requirement, what should we do in case of peer asking for a StopHash non-yet-announced to them. Do we check StopHash inferior ou equal to its pindexBestHeaderSent somewhere ?

I don't think it matters. If a BIP 157 client pre-emptively requests a filter header for a block that the server hasn't yet accepted, then they would be disconnected.

Generally we don't consider block propagation timing to be a privacy/topology leak, since new block events are rare and too expensive to be controlled by an observer.

Requesting a block that the server hasn't accepted yet is logically equivalent to requesting a non-existent block, which is tested in p2p_cfilters.

Generally we don't consider block propagation timing to be a privacy/topology leak, since new block events are rare and too expensive to be controlled by an observer.

I think relying on event scarcity in case of block variance may not hold, so I would rather consider the heavily-optimized block propagation as an obstruction for an attacker to fingerprint topology with confidence.

Requesting a block that the server hasn't accepted yet is logically equivalent to requesting a non-existent block, which is tested in p2p_cfilters.

I think there is logically a difference between accepting a block and header announcement, but AFAIK there is no types of connections for which we don't announce headers so doesn't make a difference in practice, I agree.

jnewbery · 2020-05-07T16:00:16Z

Rebased on master to fix the spurious travis failures, and addressed @jkczyz's review comments. Thanks for the review!

jnewbery · 2020-05-07T16:51:40Z

Addressed @ariard's review comments.

fjahr

Concept ACK

src/net_processing.cpp

src/init.cpp

jonatack

Code review, built/ran tests, and running a mainnet node with -peercfilters=1 -blockfilterindex=1... which took a bit more than an hour to build ~./bitcoin/indexes/blockfilter/basic for the first time, which at current tip height of 629407 takes 5.0 GB of disk space -- not strictly related to this PR, but I wanted to test it. Verified the behavior for "Error: Cannot set -peercfilters without -blockfilterindex." A few non-blocking comments follow below.

src/net_processing.cpp

test/functional/p2p_cfilters.py

src/net_processing.cpp

test/functional/p2p_cfilters.py

fjahr

ACK fccc857

Reviewed code, ran tests, and modified them to see them fail.

Had another nit. And this is also not a blocker for me but I would like to repeat my comment from the PR Review Club that I would prefer that we have a consistent name for this feature for user-facing communication. The connection between -blockfilterindex and -peercfilters does not seem obvious. I am happy with either naming scheme as long as it's consistent.

test/functional/p2p_cfilters.py

jnewbery · 2020-05-08T02:33:24Z

Addressed all review comments from @fjahr and @jonatack . Thanks for the review!

ariard · 2020-05-08T07:30:56Z

Code Review ACK 6691493, changes since last review are switch to int instead of uint32_t, renaming to peerblockfilters, test modifications, better comments.

jonatack

Re-built and tested successfully. A few comments following the change from -peercfilters to -peerblockfilters.

test/functional/p2p_cfilters.py

src/init.cpp

test/functional/test_runner.py

jnewbery · 2020-05-08T13:02:04Z

Thanks for review @jonatack . I've addressed all your comments.

jkczyz · 2020-05-08T14:59:44Z

Thanks for review @jonatack . I've addressed all your comments.

Needs push. Also, please update the flag name in commit messages.

fjahr · 2020-05-10T13:25:20Z

re-ACK 2308385

Only change was fixing merge conflict in init.cpp:

$ git range-diff f54753293fe7355e4280944d766f22054b560ba1..967e2b10a51b831e49abd9171e3f05843e00e76b 5b24f6084ede92d0f493ff416b4726245140b2c1..23083856a551ca13e8b142791c296ecb25cc4e7f

jkczyz · 2020-05-10T18:03:46Z

re-ACK 2308385

Only change required in init.cpp for #16224.

ariard · 2020-05-11T08:11:03Z

re-Code Review ACK 2308385

luke-jr · 2020-05-11T16:07:13Z

This remains Concept NACK since it is harmful to Bitcoin and should not be merged, no matter how well-reviewed.

NicolasDorier · 2020-05-11T16:27:28Z

I also believe neutrino is harmful to bitcoin https://medium.com/@nicolasdorier/neutrino-is-dangerous-for-my-self-sovereignty-18fac5bcdc25

However, I believe it can be useful to whitelisted peers. (I think you should add a permission flag for that)

Concept NACK, except if protected by a permission flag to force its use only to specific trusted peers.

ThomasBucaioni · 2020-05-11T17:03:34Z

All three test suites passed here.

clarkmoody · 2020-05-11T21:54:16Z

Tested ACK 2308385

theStack

ACK 2308385

jnewbery · 2020-05-12T12:45:39Z

This PR now has 6 ACKs:

The PR that this is a subset of also has an ACK:

@MarcoFalke (Serve BIP 157 compact filters #16442 (comment))

And there are 6 additional Concept ACKs for the change:

That's 13 ACKs/Concept ACKs. There does appear to be widespread support to move forwards with this.

@luke-jr @NicolasDorier - your principled objection to BIP 157 has been heard and understood. Thank you for your input!

maflcko · 2020-05-12T12:57:25Z

re-ACK 2308385 🌳

Only change since my previous review #18876 (comment)

Rename setting
Small code and test refactoring

Show signature and timestamp

Signature:

Timestamp of file with hash 286f8e8c18188195dc7b2da1cd9ec5c45e6b4ff0cfa7173476f9af073c383815 -

maflcko · 2020-05-12T13:02:07Z

Concept NACK, except if protected by a permission flag to force its use only to specific trusted peers.

The implementation here is incomplete. The follow-ups in #18876 need to be merged as well. And if another setting is seen to be needed for this to be safely shipped, this should also be done in a follow-up. However, I think we shouldn't block progress here based on Bitcoin Core related implementation details that can easily be fixed up later. There is enough time until the release to hash this out. (See #18947) If by then the implementation is still incomplete, it should be reverted.

Given that, I think this is ready for merge, both code-wise and conceptually.

2308385 [test] Add test for cfcheckpt (Jim Posen) f9e00bb [net processing] Message handling for getcfcheckpt. (Jim Posen) 9ccaaba [init] Add -peerblockfilters option (Jim Posen) Pull request description: Serve cfcheckpt messages if basic block filter index is enabled and `-peercfilters` is set. `NODE_COMPACT_FILTERS` is not signaled to peers, but functionality can be used for testing and serving pre-configured clients. ACKs for top commit: jonatack: Code review re-ACK 2308385 the only change since my review @ 967e2b1 is an update required for bitcoin#16224 that was merged yesterday. fjahr: re-ACK 2308385 jkczyz: re-ACK 2308385 ariard: re-Code Review ACK 2308385 clarkmoody: Tested ACK 2308385 MarcoFalke: re-ACK 2308385 🌳 theStack: ACK bitcoin@2308385 Tree-SHA512: 8c751bbd7d1c31a413096462ae025c3d2f3163c7016cbec472a5f5ec267f8dd19a2dfc4d749876d7409c1db546e6fdd16461c6863effcfa0d3e993edcfa92a08

jonasschnelli · 2020-05-13T13:08:44Z

post merge concept ACK.

However, I think this merge was too fast. A 6 day window for a first concrete step in filter serving after BIP157 is IMO too fast for a reasonable discussion. I also still miss conceptual reviews from long term p2p contributors. There is no rush IMO.

Edit: I probably underestimated the effort that went into #16442 (this PR is mostly recycled code from 16442) which could legitimate a quicker merge.

maflcko · 2020-05-14T16:36:41Z

Edit: I probably underestimated the effort that went into #16442 (this PR is mostly recycled code from 16442) which could legitimate a quicker merge.

Thanks. Just wanted to reply that.

For reference, the permission flag pr is here, btw: #18972

When a node is configured with --blockfilterindex=basic and -peerblockfilters it can serve compact block filters to its peers. This commit adds the configuration option handling. Future commits add compact block serving and service bits signaling. Github-Pull: bitcoin#18877 Rebased-From: 9ccaaba

If -peerblockfilters is configured, handle requests for cfcheckpt. Github-Pull: bitcoin#18877 Rebased-From: f9e00bb

Github-Pull: bitcoin#18877 Rebased-From: 2308385

Summary: ``` Serve cfcheckpt messages if basic block filter index is enabled and -peercfilters is set. NODE_COMPACT_FILTERS is not signaled to peers, but functionality can be used for testing and serving pre-configured clients. ``` Backport of core [[bitcoin/bitcoin#18877 | PR18877]]. Depends on D8461 and D8462. Test Plan: ninja all check-all Reviewers: #bitcoin_abc, deadalnix Reviewed By: #bitcoin_abc, deadalnix Differential Revision: https://reviews.bitcoinabc.org/D8463

merge bitcoin#16787, bitcoin#17474, bitcoin#17687, bitcoin#18165, bitcoin#18877, bitcoin#18960, bitcoin#19010, bitcoin#19044, bitcoin#19070: block filters

jnewbery mentioned this pull request May 5, 2020

[WIP] Serve BIP 157 compact filters #18876

Closed

fanquake added the P2P label May 5, 2020

DrahtBot mentioned this pull request May 5, 2020

net: Use mockable time for ping/pong, add tests #18638

Merged

jnewbery force-pushed the 2020-04-serve-getcfcheckpt branch from 63659fb to 493eb3b Compare May 5, 2020 12:17

This was referenced May 5, 2020

Add BIP324 encrypted p2p transport de-/serializer (only used in tests) #18242

Closed

[WIP] Index for UTXO Set Statistics #18000

Closed

Return BlockValidationState from ProcessNewBlock if CheckBlock/AcceptBlock fails #17479

Closed

jnewbery force-pushed the 2020-04-serve-getcfcheckpt branch from 493eb3b to 6fdf2ea Compare May 5, 2020 20:57

promag reviewed May 5, 2020

View reviewed changes

src/init.cpp Outdated Show resolved Hide resolved

jnewbery force-pushed the 2020-04-serve-getcfcheckpt branch from 6fdf2ea to 18ad397 Compare May 6, 2020 03:41

DrahtBot mentioned this pull request May 6, 2020

Make g_chainman internal to validation #18698

Merged

jnewbery force-pushed the 2020-04-serve-getcfcheckpt branch from 18ad397 to fccc857 Compare May 6, 2020 15:25

jkczyz reviewed May 7, 2020

View reviewed changes

src/protocol.h Show resolved Hide resolved

src/net_processing.cpp Outdated Show resolved Hide resolved

src/net_processing.cpp Outdated Show resolved Hide resolved

src/net_processing.cpp Outdated Show resolved Hide resolved

src/net_processing.cpp Outdated Show resolved Hide resolved

ariard reviewed May 7, 2020

View reviewed changes

jnewbery force-pushed the 2020-04-serve-getcfcheckpt branch from fccc857 to f874115 Compare May 7, 2020 15:59

jnewbery force-pushed the 2020-04-serve-getcfcheckpt branch from f874115 to 2a15f99 Compare May 7, 2020 16:51

fjahr reviewed May 7, 2020

View reviewed changes

src/net_processing.cpp Outdated Show resolved Hide resolved

src/init.cpp Outdated Show resolved Hide resolved

jonatack reviewed May 7, 2020

View reviewed changes

fjahr approved these changes May 7, 2020

View reviewed changes

test/functional/p2p_cfilters.py Outdated Show resolved Hide resolved

jnewbery force-pushed the 2020-04-serve-getcfcheckpt branch from 2a15f99 to 6691493 Compare May 8, 2020 02:32

jonatack reviewed May 8, 2020

View reviewed changes

test/functional/p2p_cfilters.py Outdated Show resolved Hide resolved

src/init.cpp Outdated Show resolved Hide resolved

test/functional/test_runner.py Outdated Show resolved Hide resolved

theStack approved these changes May 12, 2020

View reviewed changes

maflcko merged commit e45fb7e into bitcoin:master May 12, 2020

jnewbery deleted the 2020-04-serve-getcfcheckpt branch May 12, 2020 14:54

luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Jun 9, 2020

[net processing] Message handling for getcfcheckpt.

5b11ad8

If -peerblockfilters is configured, handle requests for cfcheckpt. Github-Pull: bitcoin#18877 Rebased-From: f9e00bb

luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Jun 9, 2020

[test] Add test for cfcheckpt

f91a612

Github-Pull: bitcoin#18877 Rebased-From: 2308385

kwvg added a commit to kwvg/dash that referenced this pull request Aug 22, 2021

bitcoin#18877: Serve cfcheckpt requests

3ad2028

kwvg added a commit to kwvg/dash that referenced this pull request Aug 22, 2021

bitcoin#18877: Serve cfcheckpt requests

7f683d9

kwvg added a commit to kwvg/dash that referenced this pull request Sep 16, 2021

bitcoin#18877: Serve cfcheckpt requests

a82c615

kwvg added a commit to kwvg/dash that referenced this pull request Sep 18, 2021

merge bitcoin#18877: Serve cfcheckpt requests

8ce7543

kwvg added a commit to kwvg/dash that referenced this pull request Sep 19, 2021

merge bitcoin#18877: Serve cfcheckpt requests

5b22d6d

thelazier pushed a commit to thelazier/dash that referenced this pull request Sep 25, 2021

merge bitcoin#18877: Serve cfcheckpt requests

de1edb5

kwvg added a commit to kwvg/dash that referenced this pull request Oct 12, 2021

merge bitcoin#18877: Serve cfcheckpt requests

d34285e

gades pushed a commit to cosanta/cosanta-core that referenced this pull request May 30, 2022

merge bitcoin#18877: Serve cfcheckpt requests

3eaf66d

bitcoin locked as resolved and limited conversation to collaborators Aug 16, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Serve cfcheckpt requests #18877

Serve cfcheckpt requests #18877

jnewbery commented May 5, 2020

prusnak commented May 5, 2020

DrahtBot commented May 5, 2020 •

edited

Loading

jnewbery commented May 6, 2020

jkczyz left a comment

ariard May 7, 2020

jnewbery May 7, 2020

ariard May 8, 2020

jnewbery commented May 7, 2020

jnewbery commented May 7, 2020

fjahr left a comment

jonatack left a comment •

edited

Loading

fjahr left a comment •

edited

Loading

jnewbery commented May 8, 2020

ariard commented May 8, 2020

jonatack left a comment

jnewbery commented May 8, 2020

jkczyz commented May 8, 2020

fjahr commented May 10, 2020

jkczyz commented May 10, 2020

ariard commented May 11, 2020

luke-jr commented May 11, 2020 •

edited

Loading

NicolasDorier commented May 11, 2020 •

edited

Loading

ThomasBucaioni commented May 11, 2020

clarkmoody commented May 11, 2020

theStack left a comment

jnewbery commented May 12, 2020

maflcko commented May 12, 2020

maflcko commented May 12, 2020

jonasschnelli commented May 13, 2020 •

edited

Loading

maflcko commented May 14, 2020

Serve cfcheckpt requests #18877

Serve cfcheckpt requests #18877

Conversation

jnewbery commented May 5, 2020

prusnak commented May 5, 2020

DrahtBot commented May 5, 2020 • edited Loading

Conflicts

jnewbery commented May 6, 2020

jkczyz left a comment

Choose a reason for hiding this comment

ariard May 7, 2020

Choose a reason for hiding this comment

jnewbery May 7, 2020

Choose a reason for hiding this comment

ariard May 8, 2020

Choose a reason for hiding this comment

jnewbery commented May 7, 2020

jnewbery commented May 7, 2020

fjahr left a comment

Choose a reason for hiding this comment

jonatack left a comment • edited Loading

Choose a reason for hiding this comment

fjahr left a comment • edited Loading

Choose a reason for hiding this comment

jnewbery commented May 8, 2020

ariard commented May 8, 2020

jonatack left a comment

Choose a reason for hiding this comment

jnewbery commented May 8, 2020

jkczyz commented May 8, 2020

fjahr commented May 10, 2020

jkczyz commented May 10, 2020

ariard commented May 11, 2020

luke-jr commented May 11, 2020 • edited Loading

NicolasDorier commented May 11, 2020 • edited Loading

ThomasBucaioni commented May 11, 2020

clarkmoody commented May 11, 2020

theStack left a comment

Choose a reason for hiding this comment

jnewbery commented May 12, 2020

maflcko commented May 12, 2020

maflcko commented May 12, 2020

jonasschnelli commented May 13, 2020 • edited Loading

maflcko commented May 14, 2020

DrahtBot commented May 5, 2020 •

edited

Loading

jonatack left a comment •

edited

Loading

fjahr left a comment •

edited

Loading

luke-jr commented May 11, 2020 •

edited

Loading

NicolasDorier commented May 11, 2020 •

edited

Loading

jonasschnelli commented May 13, 2020 •

edited

Loading