RFC: Introducing AltNet, a pluggable framework for alternative transports

[ariard]

As of 14/03/2022 (chain tip hash : 00000000000000000000628002621832be9281faf72c394e66a8bc8446f80c75), this project is under implementation in separate process relying on the multiprocess framework. See chaincodelabs/libmultiprocess#56 for more information on current state.

This PR proposes to introduce a new subsystem aiming towards providing a generic framework for integrating alternative transport communication channels inside Bitcoin Core. This PR is a functional proof-of-concept using LN transport protocol as a new link layer for header fetching. See #18989 for high-level discussion and proposal merits.

The Design

Previous work has been attempted to increase network security and robustness by increasing communication redundancy. While being optional and fault-tolerant, these attempt don't scale well to integrate a wide-range of alternatives such as domain-fronting, orbital communications, amateur radio, Tor pluggable transports, unlicensed UHF, Bluetooth, I2P, Lightning Noise transport, ... Each of these transports has its own availability, latency, don't bind to the Berkeley sockets model, are unidirectional, require custom peer discovery, are not suited for block traffic, ...

This proposal tries to overcome these issues by laying out a generic "driver" framework with its own new processing stack, thereby encapsulating per-transport oddities. It introduces:

• Per-transport capabilities (TransportCapabilities). Originally declared by each driver, they abstract driver specifities to general flags, like directionality, protocol messages scope. They could be extended with further attributes, like max bandwidth for adjusted packing or "privacy"/"trustness" niceness of the link.
• Abstract driver model (CDriver). Basically, it dissociate driver warmup, message preprocessing, sending and reception. Ideally you would want to load them dynamically.
• "Anycast" header-fetching (AltLogicValidation::BlockHeaderAnomalie). After notification from the watchdog module, it sends a GETHEADERS message to any peers supporting bidirectionality and headers processing.
• Alternative processing stack (AltLogicValidation). Relying on its own threads, it manages drivers and may take action based on static logical capabilites instead of peer state, reducing likelihood of bugs. In the future, one could implement a more robust headers-syncing strategy like reverse-header or compressed headers. It may also leverage a future asynchronous validation interface.

There are many advantages of this kind of design --, it is easy to integrate new drivers binding to an abstract class, it is easy to reason about due to logical capabilities and at the same time provides granular tuning due to the fine-integration with rpc. Peer discovery and policy or bandwidth/memory DoS-protection are all possible driver add-ons as well. One could envision an rpc like sendemergencytx, broadcasting a transaction on any transport protocol available, like a LN's justice transaction after detection of being eclipsed.

This module could be made entirely optional at compile-time with --enable-altnet even if such integration with the build system hasn't been made.

Obviously this is an exploratory PoC, I don't expect code here to be merged. Beyond worthiness of doing this in Rust there are a lot of open questions, including the threading/memory model, interface with existing codebase, build system, resources consumption and fault-tolerance. As next steps, I would favor building on the multiprocess work and hosting this new network stack in its own to avoid any bug contamination from introducing complex driver. I encourage those with opinions on the higher level motivation and design discussion to leave their ideas on the linked issue.

High-level design:

          ___________            ______________________
         |           |          |                      |
         | CWatchdog |----------|  AltLogicValidation  |
         |___________|          |______________________|
                                           |
                                     ______|______
                                    |             |
                                    |  CAltstack  |                                                         
                                    |_____________| 
                                           |
                                       ____|____
                                      |         |
                                      | CDriver |
                                      |_________|
                                           |
                                   ________|__________
                                  |                  |
                                  | ClightningDriver |
                                  |__________________|

The Demo

This PoC features a header-fetching round-trip, leveraging LN connection protocol. Each LN node connects to its peers through an encrypted and authenticated channel. Since your LN peer must have chain access, you could leverage this privileged link to create a redundant chain access.

Using C-lightning custom LN message support and plugin API, we could embed directly inside HEADERS/GETHEADERS and realize a round-trip between 2 disconnected bitcoin nodes through their respective C-lightning nodes.

For this to work, you need to compile and run 2 C-lightning nodes, each with its own data directory and launched with this custom bridge plugin. Once those LN nodes are running, activate bridges by lightning-cli alice_on and lightning-cli bob_on. The plugin doesn't seem to accept CLI argument so I had to hardcode some node addresses.

Then compile 2 bitcoin nodes with this patchset. For one of them you must change the hardcoded port in C-lightning driver and deactivate the watchdog logic. Also, it would be best if you changed SCAN_INTERVAL to something reasonable, for instance 1 min.

High-level setup:

                                             -->

                __________          LN_msg(`GETHEADERS`)                   ________
               |          |                                               |        |
               | CL-Alice |-----------------------------------------------| CL-Bob |
               |__________|                                               |________|           
                    |               LN_msg(`HEADERS`)                         |
                ____|_____                                                 ___|_____
               |          |                 <--                           |         |
               |  bridge  |                                               | bridge  |
               |__________|                                               |_________|
                    |                                                         |
                ____|_____                                                 ___|____
               |          |                                               |        |
               | BC-Alice |                                               | BC-Bob |
               |__________|                                               |________|

                  HEIGHT                                                   HEIGH + 1

Running both nodes, they should automatically connect to the bridge, and after anomaly detection realize a header exchange, if one of them is at least one block forward (run generatetoaddress 1 $ADDR).

Otherwise you may observe past logs:

Alice:

Ê0-05-16T00:55:11Z Boostrapping Altstack
2020-05-16T00:55:11Z init message: Done loading
2020-05-16T00:55:11Z msghand thread start
2020-05-16T00:55:11Z drivers-warmup thread start
2020-05-16T00:55:11Z addcon thread start
2020-05-16T00:55:11Z Registered driver with 0
2020-05-16T00:55:11Z net thread start
2020-05-16T00:55:11Z altstack-processing thread start
2020-05-16T00:55:11Z Clightning - Warmup
2020-05-16T00:55:11Z dnsseed thread start
2020-05-16T00:55:11Z altstack-handle thread start
2020-05-16T00:55:11Z 0 addresses found from DNS seeds
2020-05-16T00:55:11Z dnsseed thread exit
2020-05-16T00:55:11Z Clightning - Bridge connection
2020-05-16T00:55:11Z Registering node_id 0 with driver_id 0
2020-05-16T00:55:11Z drivers-warmup thread exit
2020-05-16T00:55:16Z Block header anomalie detected - notifying subscribers
2020-05-16T00:55:16Z Block header anomalie detected - Anycasting header fetching from 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206
2020-05-16T00:55:16Z Clightning - Preprocessing getheader
2020-05-16T00:55:16Z Cligthning - Bridge send success 71
2020-05-16T00:55:19Z Cligtning - Bridge receive success
2020-05-16T00:55:19Z Receive HEADERS from 0
2020-05-16T00:55:19Z Receiver header 65084efeaffe9929de0a0bb747cf48ac7f9b4edb214b2d1ad2ea468977be8e1e vs tip 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206

Bob:

2020-05-16T00:55:09Z Boostrapping Altstack
2020-05-16T00:55:09Z addcon thread start
2020-05-16T00:55:09Z opencon thread start
2020-05-16T00:55:09Z msghand thread start
2020-05-16T00:55:09Z drivers-warmup thread start
2020-05-16T00:55:09Z Registered driver with 0
2020-05-16T00:55:09Z Clightning - Warmup
2020-05-16T00:55:09Z init message: Done loading
2020-05-16T00:55:09Z dnsseed thread exit
2020-05-16T00:55:09Z altstack-processing thread start
2020-05-16T00:55:09Z altstack-handle thread start
2020-05-16T00:55:09Z Clightning - Bridge connection
2020-05-16T00:55:09Z Registering node_id 0 with driver_id 0
2020-05-16T00:55:09Z drivers-warmup thread exit
2020-05-16T00:55:10Z Leaving InitialBlockDownload (latching to false)
2020-05-16T00:55:17Z Cligtning - Bridge receive success
2020-05-16T00:55:17Z Receive GETHEADERS from 0
2020-05-16T00:55:17Z Fetch header starting from 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206, tip 65084efeaffe9929de0a0bb747cf48ac7f9b4edb214b2d1ad2ea468977be8e1e
2020-05-16T00:55:17Z Sending back 1 headers
2020-05-16T00:55:17Z Clightning - Preprocessing header
2020-05-16T00:55:17Z Cligthning - Bridge send success 84
2020-05-16T00:56:10Z Adding fixed seed nodes as DNS doesn't seem to be available.

[icota]

Exciting stuff but I question the logic of there being a framework just for "alternatives". Shouldn't ipv4/6 and onion be one of many drivers with their own capabilities and trustworthiness?

More work initially but probably easier maintenance down the road and a guarantee that any type of smoke signal protocol driver could be a first-class citizen like ip.

Concept NACK for not being ambitious enough 😉

[DrahtBot]

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

• Use shared pointers only in validation interface #18354 (Protect wallet by using shared pointers by bvbfan)
• Log RPC parameters (arguments) if -debug=rpcparams #16365 (Log RPC parameters (arguments) if -debug=rpcparams by LarryRuane)
• feature: Added ability for users to add a startup command #15367 (feature: Added ability for users to add a startup command by benthecarman)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

[Ayms]

What I want to do is:

bitcoin | node-Tor | bitcoin via stdin/stdout or bitcoin <-> node-Tor via unix sockets
		|
	send/receive
	via Tor or
	node-Tor nodes
		|
	optional: node-Tor RDV point
		|
	node-Tor or Tor nodes
		|
	bitcoin node via Tor exit node
		or
	bitcoin <-> node-Tor process

This is one phase of some funding proposals for the more global Convergence concepts, I will let you know when/if it happens and maybe altnet by that time will have evolved, if too complicate to setup another option is to try this with bitcoin-transactions which implement a part of the bitcoin protocol, and/or to combine both so the node-Tor node above can be a browser also (WebRTC/WS)

[ryanofsky]

What I want to do is:

bitcoin | node-Tor | bitcoin via stdin/stdout or bitcoin <-> node-Tor via unix sockets

This is really interesting. Maybe it would be good to create a specific issue to discuss this here or in the node-Tor github repo.

If I were trying to connect bitcoind <--> node-Tor, I think the simplest approach to start with would be to get existing bitcoind -onion and -listenonion options to work with node-Tor. To do this, no bitcoind changes should be needed and node-Tor should just need to support simple subsets of the tor control and socks5 protocols (if it doesn't already). To avoid the need to use plumbing tools like socat, or to modify bitcoind, it would probably be easiest to just start with plain tcp sockets instead of unix sockets or stdin/stdout.

After this worked, I'd start looking into more changes to streamline and package the functionality better:

1. Changing -onion and -listenonion to support unix sockets or pipes instead of tcp
2. Using RFC: Introducing AltNet, a pluggable framework for alternative transports #18988 (this PR) to write a new altnet driver that can speak socks5/torcontrol to node-Tor. No need to mess with multiprocess: Add basic spawn and IPC support #19160 (multiprocess pr) if the altnet driver is written in c++ or can be part of bitcoin core, but you might use multiprocess: Add basic spawn and IPC support #19160 if you want to write the altnet driver in a different language or more independently of the bitcoin core codebase.
3. Maybe switch from socks5/torcontrol to a custom protocol for bitcoind <--> node-Tor communication. If socks5/torcontrol don't fit well into the altnet framework, maybe do this at the same time as writing the altnet driver.

[Ayms]

This is really interesting. Maybe it would be good to create a specific issue to discuss this here or in the node-Tor github repo.

Thanks, probably @ariard can tell us if this discussion should be moved to other places, I don't find it uninteresting to compare the different approaches

node-Tor does support the socks interface since the very beginning but I don't see the point of using a potentially vulnerable localhost and weak socks protocol for processes that are running on the same machine, we saw not a long time ago that it could lead to security breaches for bitcoin

The Tor project (which is designed for browsing) does not really have the choice today (also for Snowflake), for node-Tor I just don't want to use it, and just don't need it, because browsers for example are not browsing anything but acting as Tor protocol relays/nodes (paradoxically they can't browse anything via WebRTC/WS/xhr interfaces)

So I think that a better approach is pluggable layers with altnet that can be chained including unix sockets and including browser interfaces

What I am proposing can look "ambitious/disruptive or whatever" but I am not far from it and those ideas are not just from yesterday, it's just a matter of time to do it and that's just a part of the Convergence project, 3 with unix sockets looks to be the right option for a universal nodejs driver (not talking about a specific driver for node-Tor, it should work for other nodejs programs), writing it myself... most likely no, I could do it but I can't pretend to be a C/C++ expert for whom it should probably be a piece of cake to do it

[fanquake]

Given that #18987 (which this is based on) has been closed, I'm going to close this for now. I think further discussion of this in issues is potentially going to be more productive.

[ariard]

Still eager to collect feedbacks on the design. Blocker is currently the internal plumbing to communicate across processes. State of the discussion is here : chaincodelabs/libmultiprocess#56