Net code updates [Step 2] #2359
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reachable from either place where SetIP is used when our best-guess addrLocal for a peer is IPv4, but the peer tells us it's reaching us at an IPv6 address. In that case, SetIP turns an IPv4 address into an IPv6 address without setting the scopeId, which is subsequently read in GetSockAddr during CNetAddr::ToStringIP and passed to getnameinfo. Fix by ensuring every constructor initializes the scopeId field with something.
We currently do two resolves for dns seeds: one for the results, and one to serve in addrman as the source for those addresses. There's no requirement that the source hostname resolves to the stored identifier, only that the mapping is unique. So rather than incurring the second lookup, combine a private subnet with a hash of the hostname. The resulting v6 ip is guaranteed not to be publicy routable, and has only a negligible chance of colliding with a user's internal network (which would be of no consequence anyway).
In order to prevent mixups, our internal range is never allowed as a resolve result. This means that no user-provided string will ever be confused with an internal address.
This addresss the TODO to avoid resolving twice.
…up MurmurHash3 to be more clear).
mapAlreadyAskedFor does not keep track of which peer has a request queued for a particular tx. As a result, a peer can blind a node to a tx indefinitely by sending many invs for the same tx, and then never replying to getdatas for it. Each inv received will be placed 2 minutes farther back in mapAlreadyAskedFor, so a short message containing 10 invs would render that tx unavailable for 20 minutes. This is fixed by disallowing a peer from having more than one entry for a particular inv in mapAlreadyAskedFor at a time.
…rns. The setAskFor duplicate elimination was too eager and removed entries when we still had no getdata response, allowing the peer to keep INVing and not responding.
furszy
force-pushed
the
2021_net_backports_step_2
branch
from
ff9ecd8
to
74d0482
Compare
Rationale: * delete ptr is a no-op if ptr is nullptr
I am not aware of any reason that we'd try to stop a ban-list timing side-channel and the prior code wouldn't be enough if we were.
Previously if we didn't have any local addresses, GetLocalAddress would return 0.0.0.0 and then we'd swap in a peer's notion of our address in AdvertiseLocal, but then nServices would never get set.
furszy
force-pushed
the
2021_net_backports_step_2
branch
2 times, most recently
from
9ee5c0a
to
fcdf87a
Compare
random-zebra
approved these changes
May 18, 2021
There was a problem hiding this comment.
Code ACK fcdf87a
Note: this is not including bitcoin#15689, might want to update the PR description.
|
Yeah, just removed 15689 mention from the description. If my memory doesn't fail me, i removed it due some pre-requisites that we don't have atm. |
random-zebra
added this to In Progress
in perpetual updating PIVX Core to BTC Core
via automation
This was referenced Jun 4, 2021
random-zebra
added a commit
that referenced
this pull request
Jun 28, 2021
…t framework update) 7b04c0a [Test] Clean duplicate connections creation in mining_pos_coldStaking.py (furszy) 15a799e [Test] MAX_PROTOCOL_MESSAGE_LENGTH PIVXified. (furszy) 0aedf35 [tests] Don't import asyncio to test magic bytes (John Newbery) 9bb5309 Refactor resource exhaustion test (Troy Giorshev) 589a780 Fix "invalid message size" test (Troy Giorshev) 8a0c12b Move size limits to module-global (Troy Giorshev) b3391c5 Remove two unneeded tests (Troy Giorshev) 1404e68 test: Add various low-level p2p tests (furszy) 8aaf7e1 test: Remove fragile assert_memory_usage_stable (MarcoFalke) f68e22c [Test] p2p_invalid_messages.py do not change msg.command in test_command and raise sync_with_ping timeout (furszy) c851c0b Test framework: Wait for verack by default on every new connection (furszy) c02e9a0 [Test] move framework subversion to string (furszy) 3472a39 Replace coroutine with async def in p2p_invalid_messages.py (Elichai Turkel) 33c7b19 net_processing: align msg checksum check properly. (furszy) 7f71b1b Hash P2P messages as they are received instead of at process-time (furszy) 215a638 test: Skip flaky p2p_invalid_messages test on macOS (Fabian Jahr) c11f565 qa: Make swap_magic_bytes in p2p_invalid_messages atomic (MarcoFalke) be979ad test: Fix race in p2p_invalid_messages (MarcoFalke) 6a72f0c qa: Add tests for invalid message headers (MarcoFalke) 51ddd3d Introduce p2p_invalid_messages.py test (furszy) 55a37b5 net: fix missing jump line in "Oversized message from peer.." error. (furszy) 0edfce5 test_node: get_mem_rss fixups (MarcoFalke) 6f21213 tests: add P2PConnection.send_raw_message (James O'Beirne) ae68c6e tests: add utility to assert node memory usage hasn't increased (James O'Beirne) 8469afe test: forward timeouts properly in send_blocks_and_test (James O'Beirne) db28a53 Skip is_closing() check when not available. (Daniel Kraft) be9dacb tests: fixes mininode's P2PConnection sending messages on closing transport (marcoagner) 53599c8 qa: Avoid start/stop of the network thread mid-test (furszy) 688190c [qa] mininode: Expose connection state through is_connected (MarcoFalke) Pull request description: Part of the deep and long net and ser work that I'm doing (and Tor v3 onion addresses support). Friend of #2359. Focused on the end goal of implementing the `p2p_invalid_messages` functional test which validates that invalid msg headers, msg types, oversized payloads and inventory msgs aren't accepted nor can cause a resource exhaustion. And an extra covered scenario, in `p2p_invalid_tx.py`, for the orphan pool overflow. Plus, to get up to the goal, had to work over the functional test framework as well. So.. adapted list: * bitcoin#9045. * bitcoin#13512. * bitcoin#13517. * bitcoin#13715. * bitcoin#13747. * bitcoin#14456. * bitcoin#14522. * bitcoin#14672. * bitcoin#14693. * bitcoin#14812. * bitcoin#15246. * bitcoin#15330. * bitcoin#15697. * bitcoin#16445. * bitcoin#17931. * bitcoin#17469. * bitcoin#18628 (only `p2p_invalid_tx.py` and `p2p_invalid_messages.py`. We don't support the other tests yet). * bitcoin#19177. * bitcoin#19264. ACKs for top commit: random-zebra: utACK 7b04c0a and merging... Tree-SHA512: 48d1f1a6acc24a9abce033fbf1f281ba4392147da39d118a694a09d63c3e0610cc1a29d711b434b16cc0d0e030dd9f1a89843870091b6999b862b9ab18a20679
furszy
added a commit
that referenced
this pull request
Jul 1, 2021
a470c34 Backport attributes.h and connect it to base58.h functions only (practicalswift) 0247f6f util: Don't allow base58-decoding of std::string:s containing non-base58 characters (practicalswift) 70c480c tests: Add tests for base58-decoding of std::string:s containing non-base58 characters (practicalswift) 9d481be Add bounds checks in key_io before DecodeBase58Check (Pieter Wuille) eac71b5 Finish Encode/Decode destination functions move from base58 to key_io. (furszy) 2e9376c Pass a maximum output length to DecodeBase58 and DecodeBase58Check (Pieter Wuille) c93e19f Clean duplicate usage of DecodeSecret & EncodeSecret. (furszy) 4d4160e Stop using CBase58Data for ext keys (furszy) e861cda Backport string ToUpper and ToLower. (furszy) f6c2872 util: Add Join helper to join a list of strings (MarcoFalke) 32c1e42 Add tests for util/vector.h's Cat and Vector (Pieter Wuille) dc42563 Add some general std::vector utility functions (Pieter Wuille) Pull request description: Decoupled from #2411 Tor's v3 addr support, built on top of #2359. This PR finishes the address encoding cleanup, removing the `CBase58`, `CBitcoinSecret`, `CBitcoinExtKey`, and `CBitcoinExtPubKey` classes, in favor of using the KeyIO::Encode/Decode functions. Furthermore, all PIVX-specific address logic is moved to key_io.{h,cpp}, leaving base58.{h,cpp} as a pure utility that implements the base58 encoding/decoding logic. Plus, includes some general utility functions for std::vector and std::string. Adaptation of the following PRs: * bitcoin#11372. * bitcoin#16670. (without faebf62) * bitcoin#16889. * bitcoin#17511. * bitcoin#17721. ACKs for top commit: random-zebra: rebase utACK a470c34 Fuzzbawls: ACK a470c34 Tree-SHA512: 7a3e1ea0f86c7dab960a5761a666dc7eb291d749e1e9cc24583eec2d6114ca47bc6b9ad50c1c7ff2ecba7f3f60100ce7c0ee8522dc3a2f29d6d79cb052187e0d
furszy
added a commit
that referenced
this pull request
Jul 5, 2021
bd4b846 Remove old serialization primitives (Pieter Wuille) 060d62b Convert the last, non-trivial, serialization functions to the new form (furszy) 8c74c09 Convert LimitedString to formatter (Pieter Wuille) f021897 Fix CDiskBlockIndex serialization of dummy fields for old DB versions (random-zebra) 1ee0cb2 Convert CDiskBlockIndex to new serialization. (furszy) 221bf49 Convert wallet to new serialization (furszy) cf06950 Convert to new serialization (step 3) (furszy) dc0fc95 Remove old MESS_VER_STRMESS message version try-catch. (furszy) 35fca11 Convert Qt to new serialization (Pieter Wuille) 3f7826e Add comments to CustomUintFormatter (Pieter Wuille) eccd473 Convert to new serialization (step 2). Focused on object's serializations that doesn't require an special treatment. (furszy) 0f15784 Convert everything except wallet/qt to new serialization (step 1) (Pieter Wuille) 3d3ee64 Convert merkleblock to new serialization (Pieter Wuille) 13577fb Add SER_READ and SER_WRITE for read/write-dependent statements (Russell Yanofsky) 7344c1a Extend CustomUintFormatter to support enums (Russell Yanofsky) c4d6228 Merge BigEndian functionality into CustomUintFormatter (Pieter Wuille) 3765d6c Add static_asserts to ser_X_to_Y() methods (Samer Afach) 806213a Fix a violation of C++ standard rules that unions cannot be switched. (Samer Afach) d6380c4 Add CustomUintFormatter (Pieter Wuille) fd29a50 Make VectorFormatter support stateful formatters (Russell Yanofsky) 4e2afad Convert CCompactSize to proper formatter (Pieter Wuille) bb99030 Get rid of VARINT default argument (Pieter Wuille) e107a0c Convert undo.h to new serialization framework (Pieter Wuille) a926ba3 Make std::vector and prevector reuse the VectorFormatter logic (Pieter Wuille) 1dfddce Add custom vector-element formatter (Pieter Wuille) df4e1ba Add a constant for the maximum vector allocation (5 Mbyte) (Pieter Wuille) c2fdeaf Convert compression.h to new serialization framework (Pieter Wuille) aa35991 Add FORMATTER_METHODS, similar to SERIALIZE_METHODS, but for formatters (Pieter Wuille) 3e38199 Move compressor utility functions out of class (Pieter Wuille) 7376a95 Convert chain to new serialization (Pieter Wuille) bbfc55c Convert VARINT to the formatter/Using approach (Pieter Wuille) 39c58a1 Add a generic approach for (de)serialization of objects using code in other classes (Pieter Wuille) ace3895 Convert addrdb/addrman to new serialization (Pieter Wuille) 6bb135e Introduce new serialization macros without casts (Pieter Wuille) ace7d14 Drop minor GetSerializeSize template (Ben Woosley) f05e692 Drop unused GetType() from CSizeComputer (furszy) 5c36b3d Introduce BigEndian wrapper and use it for netaddress ports (Pieter Wuille) fb3c646 Migrate last FLATDATA calls to use Span. (furszy) 1ef2d90 Support serializing Span<unsigned char> and use that instead of FLATDATA (Pieter Wuille) 8fef544 Add Slice: a (pointer, size) array view that acts like a container (Pieter Wuille) Pull request description: Decoupled from #2411, built on top of #2359. Focused on creating the Span class and updating the serialization framework and every object using it up to latest upstream structure (3-4 years ahead of what we currently are in master). We will be up-to-date with them in the area after finishing with #2411 entirely (there are few more updates to the serialization code that comes down #2411 commits line that cannot cherry-pick here). Adapted the following PRs: * bitcoin#12886. * bitcoin#12916. * bitcoin#13558. * bitcoin#17850. * bitcoin#17896. * bitcoin#12752. * bitcoin#17957. * bitcoin#18021. * bitcoin#18087. * bitcoin#18112 (only from 353f376 that we don't support). * bitcoin#18167. * bitcoin#18317. * bitcoin#19032. ACKs for top commit: random-zebra: ACK bd4b846 Fuzzbawls: ACK bd4b846 Tree-SHA512: fe1b31d0976dff97bfeed0f9efeeb4c6c161277529880ede990c9b3fb0ea680f25b4be01b739f6bf7eeca79fa7687c9c2146c403c96e86bc6b052c6dd88e6930
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Grouped some net updates (plus a miscellaneous one) coming from upstream.
Part of another deep rabbit hole that i'm doing in parallel of the tier two work.
PRs List: