IPv8 v3.0, IPv6 support and network interface refactoring

[qstokkink]

2021 update: This issue is almost resolved. The base Community class can now handle IPv6. The TunnelCommunity is the only component left in IPv8 that is required to be ported.

---

Old O.P.

Title is slightly suggestive: only one change is needed for all three

Right now, Peer objects have a most up to date address (.address). In the future we will want to support multiple network interfaces (requiring multiple sockets, which probably need to be separately punctured).

The biggest change to our current codebase will be the refactoring of Peer objects to support a set/list of interface specifications. The Peer object will have to be in charge of implementing a send() and puncture() method (opening at least one communication channel), this is currently handled by the Community class.

Pros:

• Overlay programmers have an opaque identity messaging layer, while still having control over the byte format of each message. No more worrying about addresses.
• It just works, all of the network magic is in the IPv8 core.
• More efficient: right now we just register any IPv4 we find as walkable and hope there is some user on the other side. If we bundle the possible IPv4 addresses under one key, one address working excludes the need of checking the other addresses.
• We can store historical addresses per peer and attempt to reconnect to the last known interface.

Cons:

• Need to open a socket for each interface (IPv4, IPv6, NFC?, ..)
• Breaking change to all existing code. EDIT: No breaking change was required
• Even though you really shouldn't need it in the first place, IPv4 addresses can no longer be seen as semi-stable identifiers.
• A lot of work.

[qstokkink]

Here is the class diagram (using n instead of *) of the current state of affairs and the newly proposed design (see my previous post).

Current

Proposed

[qstokkink]

With #830, the first (easy) step is complete: we can now send and receive IPv6 interface packets.

Remaining:

• Refactor the Peer.address so a Peer can have one address per interface (also includes limited changes to Network).
• Refactor existing communities so they can handle IPv6 addresses. This includes both business and puncturing logic.
• Add and support a configuration directive for interface loading.

[qstokkink]

I believe we can maintain full backwards compatibilty by making sure all introduction requests and responses using IPv4 still use the old messages.
For all other interfaces we will define a new introduction request and response message that allow for generic (str, int) tuples as addresses, reserving message ids 233 and 234.
Hereby, old versions should never receive new-style introduction requests and responses.

We can have both types (new and old) of introduction requests and response messages call their respective introduction_request_callback or introduction_response_callback callbacks, leading to an opaque design for overlay programmers.

Edit: pucturing will also have to change.

[qstokkink]

Naively implementing these new messages turns the community.py code into a complete mess. I'll take some time to (hopefully) figure out a good way to combine the old and the new logic.

Some positive news is that, due to the Payload abstraction, most of the logic can stay intact. By adding a new serializer format we can have drop-in support for the new payload types ((varlenH, H) instead of (4s, H)).

[qstokkink]

Update on the current best plan to make sure the new-style introduction messages don't spaghettify the community logic:

Both the IntroductionRequest and IntroductionResponse have unused bits (in the advice and connection) for future use. For both messages I will claim two of these bits:

• Bit 1: "I also support new-style introduction logic"
• Bit 2 (IntroductionResponse): "The introduced peer supports new-style logic"

Naturally, receiving any new-style introduction logic from a peer automatically flags it as supporting new style logic.

This turns most of the connection management that would've had to reside in the community.py into a single flag on Peer objects: new_style_intro.

[qstokkink]

With #913 merged we now move into the final phase: porting the DHTCommunity and the TunnelCommunity.

With that, after an entire year of development, my involvement with this issue is also over.

[qstokkink]

Once the last component has been ported, the configuration should also support specifying the address and port of the IPv6 interface.

py-ipv8/ipv8_service.py

Lines 79 to 80 in 76da077

	UDPIPv4={'port': configuration['port'],
	'ip': configuration['address']})

In turn, the documentation should also be updated:

https://github.com/Tribler/py-ipv8/blob/master/doc/reference/configuration.rst

[Solomon1732]

What is preventing from doing the switch? Asking because if what is left isn't too big of a task then I might be interested in tackling it.

[qstokkink]

@Solomon1732 The only Community left to port is the anonymization Community (the TunnelCommunity). We have assigned this to its original author @egbertbouman as it is quite some complex logic/cryptography that is deeply rooted in IPv4.

If you don't need the anonymization, you can disable it and locally revert this commit and see IPv8 automatically hand off to IPv6.

[Destroyinator69420]

Not to forget more closely implementing the V3 hidden services protocol.

[qstokkink]

[Memo] The test suite can be run using IPv6 fake addresses by changing the following code from "UDPv4Address" to "UDPv6Address" .

py-ipv8/ipv8/test/mocking/endpoint.py

Lines 74 to 76 in fcf2c79

	class AutoMockEndpoint(MockEndpoint):
	ADDRESS_TYPE = "UDPv4Address"