Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BIP 155: addrv2 BIP proposal #766

Merged
merged 4 commits into from Jul 23, 2019
Merged

BIP 155: addrv2 BIP proposal #766

merged 4 commits into from Jul 23, 2019

Conversation

@laanwj
Copy link
Member

@laanwj laanwj commented Feb 27, 2019

Proposal for wider v2 address messages.

Last time this went around (as a github gist) there were some considerations still to be discussed, for lack of a working mailing list I'll post them here:

Considerations

  • ''Client MAY store and gossip address formats that they do not know about'': does it ever make sense to gossip addresses outside a certain overlay network? Say, I2P addresses to Tor? I'm not sure. Especially for networks that have no exit nodes as there is no overlap with the globally routed internet at all.

  • Lower precision of time field? seconds precision seems overkill, and can even be harmful, there have been attacks that exploited high precision timestamps for mapping the current network topology.

    • (gmaxwell) If you care about space time field could be reduced to 16 bits easily. Turn it into a "time ago seen" quantized to 1 hour precision. (IIRC we quantize times to 2hrs regardless).
  • Rolling port into addr, or making the port optional, would make it possible to shave off two bytes for address types that don't have ports (however, all of the currently listed formats have a concept of port.). It could also be an optional data item (see below).

  • (gmaxwell) Optional (per-service) data could be useful for various things:

    • Node-flavors for striping (signalling which slice of the blocks the node has in selective pruning)

    • Payload for is alternative ports for other transports (e.g. UDP ports)

    • If we want optional flags. I guess the best thing would just be a byte to include the count of them, then a byte "type" for each one where the type also encodes if the payload is 0/8/16/32 bits. (using the two MSB of the type to encode the length). And then bound the count of them so that the total is still reasonably sized.

@luke-jr luke-jr changed the title addrv2 BIP proposal BIP 155: addrv2 BIP proposal Mar 29, 2019
@luke-jr
Copy link
Member

@luke-jr luke-jr commented Mar 29, 2019

Assigned BIP 155

Loading

@laanwj
Copy link
Member Author

@laanwj laanwj commented Jul 17, 2019

Thanks!
Filled in the BIP number and fixed the travis issues, this should be ready for merge I think?

Loading

<ref>[https://bitcoin.org/en/developer-reference#addr Bitcoin Developer Reference: addr message]</ref>, with the difference that the
fixed 16-byte IP address is replaced by a network ID and a variable-length address, and the time and services format has been changed to VARINT.

This means that the message contains a serialized <code>std::vector</code> of the following structure:
Copy link
Member

@MarcoFalke MarcoFalke Jul 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this is how bips were written in the past, but I'd slightly prefer not to leak cpp details or Bitcoin Core implementation details such as std::vector (can be replaced by "list" or "vector"), pchCommand (can be replaced by "message type"), and static const int GOSSIP_ADDRV2_VERSION (can be omitted without loss of information).

If someone is eager to look at the cpp implementation, there is already a section that links to it.

Loading

Copy link
Member Author

@laanwj laanwj Jul 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fine with me

Loading

Copy link
Member Author

@laanwj laanwj Jul 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I went with explicit C++ types here for clarity as to how things would be serialized, after all there is not really a spec of the serialization format (and how the certain structures are named, like "how to serialize a list", wouldn't "a list" be vague and ambigious?) besides the implementation

Loading

Copy link
Member

@MarcoFalke MarcoFalke Jul 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loading

Copy link
Member Author

@laanwj laanwj Jul 31, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

string seems to be worse as it implies a human-readable text

Loading

Send <code>addrv2</code> messages only, and exclusively, when the peer has a certain protocol version (or higher):
<source lang="c++">
//! gossiping using `addrv2` messages starts with this version
static const int GOSSIP_ADDRV2_VERSION = 70016;
Copy link
Member

@MarcoFalke MarcoFalke Jul 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't put much thought into this, but could you explain the tradeoffs here a bit? I think there are two ways to announce support for a new p2p feature:

  • Bump the protocol version
  • Send a p2p message to announce your support for the feature

Bumping the minimum protocol version seems to imply that nodes would have to support all protocol features that were introduced prior (like compactblocks, feefilter, ...)

Whereas with a new protocol message, even mobile wallets could make use of the new feature?

Loading

Copy link
Member Author

@laanwj laanwj Jul 31, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it does add a new protocol message (that's the addrv2), but the version tells it that the other side understands them

would you prefer to add another protocol message to signal support, which is always sent after version?

Loading

Copy link
Member

@MarcoFalke MarcoFalke Jul 31, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, you can keep this as is. mobile wallets tend to get their addresses from dns seeds, so this shouldn't matter.

Loading

Copy link
Member

@MarcoFalke MarcoFalke Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In light of yesterday's IRC discussion and on further thought, I think support for addrv2 should be announced by a service bit. This has the following advantages:

  • leech nodes (nodes that only want to consume addrv2 messages, e.g. "SPV clients") can find addrv2 nodes by the service bit and they can expect them to respond to sendaddrv2 messages
  • address relay between address relay nodes (e.g. full nodes/Bitcoin Core nodes) can be made explicit, instead of relying on heuristics such as NODE_NETWORK. See bitcoin/bitcoin#17163 (comment)

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

* address relay between address relay nodes (e.g. full nodes/Bitcoin Core nodes) can be made explicit, instead of relying on heuristics such as `NODE_NETWORK`. See [bitcoin/bitcoin#17163 (comment)](https://github.com/bitcoin/bitcoin/pull/17163#issuecomment-543072194)

To be clear, you mean address relay can be made explicit by an additional service bit, e.g. NODE_ADDR_GOSSIP or something like that?

Loading

Copy link
Member

@MarcoFalke MarcoFalke Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, you mean address relay can be made explicit by an additional service bit, e.g. NODE_ADDR_GOSSIP or something like that?

Yes. Currently we use heuristics such as NODE_NETWORK for that, which is fragile and not future proof. For example, pruned nodes may not have any NODE_NETWORK* bit set. See also bitcoin/bitcoin#17194

Loading

|-
| <code>VARINT</code> (unsigned)
| <code>time</code>
| Time that this node was last seen as connected to the network. A time in Unix epoch time format, up to 64 bits wide.
Copy link
Member

@MarcoFalke MarcoFalke Jul 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would help to explain what "last seen" means, since it appears easy to get wrong. E.g. bitcoin/bitcoin#5860 (comment)

Loading

Copy link
Member Author

@laanwj laanwj Jul 31, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The meaning of this field doesn't change from the current implementation. I'm fine with adding an explanation though, if you supply it, but I'm not sure what to add here.

Loading

@luke-jr luke-jr merged commit 095087b into bitcoin:master Jul 23, 2019
1 check passed
Loading
@laanwj
Copy link
Member Author

@laanwj laanwj commented Jul 31, 2019

Came up during discussion on IRC, need to be explicit about representing IPv4 as network ID IPV4, that the old wrapped representation is no longer allowed:

==Appendix X: IPv4 address encoding==
Clients MUST send IPv4 addresses with this network ID, with the 32-bit address in the address field.

Clients SHOULD ignore IPv4 Mapped Address (::FFFF:0:0/96) addresses on receive if they come with the IPV6 network ID.

(thanks @dongcarl)

Loading

@laanwj
Copy link
Member Author

@laanwj laanwj commented Jul 31, 2019

As for changing the signaling from protocol version-based to sending a message. I think this message, when sent from one peer to another, would signal support for the first peer to receive addrv2 messages from the other one. So something like sendaddrv2? (like bip130)

I wish we had some kind of standard message for this.

I think for simplicy of implementation we'd want to mandate that this should be sent after version but before any other (non-BIP-signalling) messages, so a connection won't switch to v2 suddenly halfway other things.

Edit2: This sendaddrv2 message could also contain wider addresses that doesn't fit in the version message. e.g. addrMe, the TorV3 address that we used to connect to the peer. The version message would contain a dummy address (localhost) then.

Loading

@laanwj
Copy link
Member Author

@laanwj laanwj commented Aug 7, 2019

There was some discussion on IRC that there should be a max cap for network addresses, but it should not be 256 bits, but more something like 512 bytes, to accommodate future extensions but have some kind of limit for DoS. Currently it says:

Field addr has a variable length, with a maximum of 32 bytes (256 bits). Clients SHOULD reject
longer addresses.

Change this to: a maximum of 512 bytes, and the whole message (not just the single address) SHOULD be rejected if it contains any larger item.

Loading

@dongcarl
Copy link

@dongcarl dongcarl commented Aug 22, 2019

@laanwj Just to be clear, addrv2 entries that claims they are IPv6 but come with either the IPv4-prefix, OnionCat-prefix, or CJDNS-prefix should be ignored?

For Bitcoin Core, I'm assuming our weird g_internal_prefix would also be added to that ignore list?

Loading

@dongcarl
Copy link

@dongcarl dongcarl commented Aug 23, 2019

As for changing the signaling from protocol version-based to sending a message. I think this message, when sent from one peer to another, would signal support for the first peer to receive addrv2 messages from the other one. So something like sendaddrv2? (like bip130 )

I wish we had some kind of standard message for this.

I think for simplicy of implementation we'd want to mandate that this should be sent after version but before any other (non-BIP-signalling) messages, so a connection won't switch to v2 suddenly halfway other things.

Edit2: This sendaddrv2 message could also contain wider addresses that doesn't fit in the version message. e.g. addrMe, the TorV3 address that we used to connect to the peer. The version message would contain a dummy address (localhost) then.

It would seems that that addrMe has been deprecated: bitcoin/bitcoin#8740

Loading

@naumenkogs
Copy link
Member

@naumenkogs naumenkogs commented Oct 22, 2019

''Client MAY store and gossip address formats that they do not know about'': does it ever make sense to gossip addresses outside a certain overlay network? Say, I2P addresses to Tor? I'm not sure. Especially for networks that have no exit nodes as there is no overlap with the globally routed internet at all.

I think relaying unknown networks is very important, because the nodes are often multi-homed. So this might be actually useful for somebody who your node thinks is simply ipv4.
Also, I believe it would be easier for an attacker to attack, if we don't relay these at all.
Lastly, depending on the implementation, this specific behavior might leak that you do support Tor, while you look ipv4 to an attacker.

There is no reason to not. If an attacker wants to spend our bandwidth, might as well flood with ipv4, we don't check if the node is alive before forwarding them.

Storing is another thing. With the current rate-limiting, the bandwidth we're spending here is pretty low. But addrman size is bounded, so I wouldn't store everything. Maybe, storing a couple exotic things might make sense if you will be switching in future.

Loading

vasild pushed a commit to vasild/bips that referenced this issue Apr 10, 2020
* Increase the maximum length of an address from 32 to 64 bytes and
  clarify that the entire message should be rejected if it contains
  a longer address.
  (from bitcoin#766 (comment))

* Remove a contradiction about unknown address types - "MUST ignore" VS
  "MAY store and gossip".

* Recommend gossiping addresses for unknown network ID and for networks
  to which the node is not currently connected to.
  (from bitcoin#766 (comment))

* Clarify that the entire message should be rejected if it contains an
  address with unexpected size (e.g. IPv4 address with length 5).
vasild pushed a commit to vasild/bips that referenced this issue Apr 10, 2020
* Increase the maximum length of an address from 32 to 512 bytes and
  clarify that the entire message should be rejected if it contains
  a longer address.
  (from bitcoin#766 (comment))

* Remove a contradiction about unknown address types - "MUST ignore" VS
  "MAY store and gossip".

* Recommend gossiping addresses for unknown network ID and for networks
  to which the node is not currently connected to.
  (from bitcoin#766 (comment))

* Clarify that the entire message should be rejected if it contains an
  address with unexpected size (e.g. IPv4 address with length 5).
@dongcarl
Copy link

@dongcarl dongcarl commented Apr 10, 2020

Following up on a discussion started in #907 (comment)

Was it decided how to signal support for the new addrv2 type of messages? 3 options have been discussed:

  1. Bump the protocol version (to 70016).
    pros: simple to implement and the info will be available right after the handshake, without changing the handshake itself.
    cons: the protocol version is not a bitmap, so if an implementation wants to support addrv2 then it also has to support all features prior to 70016, see also #766 (comment)

  2. Use a new network service bit (NODE_ADDRV2 = (1 << 4)).
    pros: same as 1., see also #766 (comment)
    cons: hmm?

  3. Use a new message sendaddrv2 during the handshake.
    pros: can send wider addresses during handshake that would not otherwise fit in addrMe, but addrMe is deprecated. See #766 (comment)
    cons: complicates and requires changes to the handshake and requires handling if the new message sendaddrv2 arrives after the handshake.

To overgeneralize, I think that all else being the same I prefer the "new message" mechanism over the "service bit" mechanism over the "protocol version" mechanism. However, the "service bit" mechanism is well-suited for making sure that certain specific node configurations can discover useful peers.

The support signaling can be broken down into 2 parts:

Something for my node to signal to peers that it wants addrv2 messages.

This should be indicated with a sendaddrv2 message, perhaps something like I've implemented here: bitcoin/bitcoin@c0414a6#diff-eff7adeaec73a769788bb78858815c91R2099-R2102

Something for my node to signal to peers that it serves addrv2 messages.

This should be indicated with a NODE_ADDRV2 service bit so that both:

  1. Nodes which specify an -onlynet with post-addrv2 networks, and
  2. Nodes (probably light/custom nodes) which only support addrv2

Can discover useful peers during DNS bootstrapping

Why not just use the NODE_ADDRV2 service bit for both signals?

As Marco implies here, if we use the same service bit for both signals, leech nodes (which only really want addrv2 messages) will be seen as signaling that they serve addrv2 messages, which they won't.


I'd love some feedback on whether the above seem reasonable, and perhaps some feedback on wether or not -onlynet makes sense for all the NetworkIDs specified here.

Loading

@vasild
Copy link
Contributor

@vasild vasild commented Apr 13, 2020

I'd love some feedback on whether the above seem reasonable,

What about using a service bit (different one) also for the first capability?

The second capability is actually addr/addrv2 agnostic, isn't it? It means "I will forward addresses via gossip to my peers (as addr or addrv2 messages)". This will fit perfectly into https://github.com/bitcoin/bips/pull/17194. Maybe use a name like NODE_GOSSIP_ADDRESSES to avoid confusion that this is addr or addrv2 specific?

and perhaps some feedback on whether or not -onlynet makes sense for all the NetworkIDs specified here.

ipv4, ipv6, torv2, torv3, i2p, cjdns - I think yes, although it would be strange to use just -onlynet=torv2. Maybe most often people will use -onlynet=torv2 -onlynet=torv3. Currently the options are ipv4, ipv6 and onion, the latter could be extended to do "torv2 or torv3".

Loading

@MarcoFalke
Copy link
Member

@MarcoFalke MarcoFalke commented Apr 13, 2020

What about using a service bit (different one) also for the first capability?

I don't remember what this post to the mailing list said: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2019-October/017428.html, so it might prove my point invalid. But I think it said something that both those properties should be negotiated on a per-link (per-connection) basis. In which case both should be done via a new message type.

If you think of service bits as a "static global" across the whole Bitcoin network, then it makes sense to only use them for things that don't change often and don't change on a per-connection basis. E.g. NODE_WITNESS that indicates you can serialize witness data comes to my mind. Non-consensus related stuff like transaction relay or addr relay can be negotiated on a per-connection basis.

Loading

@dongcarl
Copy link

@dongcarl dongcarl commented Apr 15, 2020

@vasild

The second capability is actually addr/addrv2 agnostic, isn't it? It means "I will forward addresses via gossip to my peers (as addr or addrv2 messages)". This will fit perfectly into bitcoin/bips/pull/17194. Maybe use a name like NODE_GOSSIP_ADDRESSES to avoid confusion that this is addr or addrv2 specific?

Unfortunately, it is not addr/addrv2 agnostic. Think of this scenario: I am a node that is -onlynet=torv3, when doing DNS bootstrapping, if we deploy NODE_GOSSIP_ADDRESSES and I use the service bits to filter for those, it is likely that I always connect to old nodes who have no idea about addrv2, however, if we deploy NODE_ADDRV2 and filter for that, the DNS seeds will be able to help connect me to nodes who at least claim to serve addrv2 messages.

@MarcoFalke

But I think it said something that both those properties should be negotiated on a per-link (per-connection) basis. In which case both should be done via a new message type.

I'm think that the "my node wants addrv2 messages" signaling should be done via a new message type, but not the "my node serves addrv2 messages" signaling. Mostly so that we don't break DNS bootstrapping (as detailed above) for both:

  1. Nodes which specify an -onlynet with post-addrv2 networks, and
  2. Nodes (probably light/custom nodes) which only support addrv2

Perhaps @naumenkogs has more thoughts.

Loading

@vasild
Copy link
Contributor

@vasild vasild commented Apr 15, 2020

The second capability is actually addr/addrv2 agnostic, isn't it? It means "I will forward addresses via gossip to my peers (as addr or addrv2 messages)". This will fit perfectly into bitcoin/bips/pull/17194. Maybe use a name like NODE_GOSSIP_ADDRESSES to avoid confusion that this is addr or addrv2 specific?

Unfortunately, it is not addr/addrv2 agnostic. Think of this scenario: I am a node that is -onlynet=torv3, when doing DNS bootstrapping, if we deploy NODE_GOSSIP_ADDRESSES and I use the service bits to filter for those, it is likely that I always connect to old nodes who have no idea about addrv2, however, if we deploy NODE_ADDRV2 and filter for that, the DNS seeds will be able to help connect me to nodes who at least claim to serve addrv2 messages.

I meant two service bits:

  • NODE_ADDRV2 - support for addrv2.

    • The node can parse addrv2 messages, if it receives them.
    • The node can generate addrv2 messages - if the node relays addresses to his peers and if the given peer has NODE_ADDRV2 then the address will be relayed as addrv2 message. If the peer does not have NODE_ADDRV2 then the address will be relayed encoded in the old addr format.
  • NODE_GOSSIP_ADDRESSES - the node would relay some of the addresses to some of its peers, as it works now. This is addr vs addrv2 agnostic and would fit perfectly to bitcoin/bitcoin#17194 - we will not relay addresses to nodes that did not signal NODE_GOSSIP_ADDRESSES.

So, for example, a multi-homed ipv4 and i2p node may seek to connect to ipv4 nodes that have both capabilities in the hopes to learn more i2p addresses from them.

Loading

vasild pushed a commit to vasild/bips that referenced this issue Jul 16, 2020
* Increase the maximum length of an address from 32 to 512 bytes and
  clarify that the entire message should be rejected if it contains
  a longer address.
  (from bitcoin#766 (comment))

* Remove a contradiction about unknown address types - "MUST ignore" VS
  "MAY store and gossip".

* Recommend gossiping addresses for networks to which the node is not
  currently connected to.
  (from bitcoin#766 (comment))

* Clarify that the entire message should be rejected if it contains an
  address with unexpected size (e.g. IPv4 address with length 5).
@Sjors
Copy link
Member

@Sjors Sjors commented Sep 16, 2020

//! gossiping using addrv2 messages starts with this version
static const int GOSSIP_ADDRV2_VERSION = 70016;

That particular version is now used for WTXID_RELAY_VERSION and bitcoin/bitcoin#19845 proposes to OR a version flag instead:

ADDRV2_FORMAT = 0x20000000;

Loading

@vasild
Copy link
Contributor

@vasild vasild commented Sep 16, 2020

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
7 participants