Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Introducing AltNet, a pluggable framework for alternative transports #18988

Closed
wants to merge 25 commits into from

Conversation

ariard
Copy link
Member

@ariard ariard commented May 16, 2020

As of 14/03/2022 (chain tip hash : 00000000000000000000628002621832be9281faf72c394e66a8bc8446f80c75), this project is under implementation in separate process relying on the multiprocess framework. See chaincodelabs/libmultiprocess#56 for more information on current state.

This PR proposes to introduce a new subsystem aiming towards providing a generic framework for integrating alternative transport communication channels inside Bitcoin Core. This PR is a functional proof-of-concept using LN transport protocol as a new link layer for header fetching. See #18989 for high-level discussion and proposal merits.

The Design

Previous work has been attempted to increase network security and robustness by increasing communication redundancy. While being optional and fault-tolerant, these attempt don't scale well to integrate a wide-range of alternatives such as domain-fronting, orbital communications, amateur radio, Tor pluggable transports, unlicensed UHF, Bluetooth, I2P, Lightning Noise transport, ... Each of these transports has its own availability, latency, don't bind to the Berkeley sockets model, are unidirectional, require custom peer discovery, are not suited for block traffic, ...

This proposal tries to overcome these issues by laying out a generic "driver" framework with its own new processing stack, thereby encapsulating per-transport oddities. It introduces:

  • Per-transport capabilities (TransportCapabilities). Originally declared by each driver, they abstract driver specifities to general flags, like directionality, protocol messages scope. They could be extended with further attributes, like max bandwidth for adjusted packing or "privacy"/"trustness" niceness of the link.
  • Abstract driver model (CDriver). Basically, it dissociate driver warmup, message preprocessing, sending and reception. Ideally you would want to load them dynamically.
  • "Anycast" header-fetching (AltLogicValidation::BlockHeaderAnomalie). After notification from the watchdog module, it sends a GETHEADERS message to any peers supporting bidirectionality and headers processing.
  • Alternative processing stack (AltLogicValidation). Relying on its own threads, it manages drivers and may take action based on static logical capabilites instead of peer state, reducing likelihood of bugs. In the future, one could implement a more robust headers-syncing strategy like reverse-header or compressed headers. It may also leverage a future asynchronous validation interface.

There are many advantages of this kind of design --, it is easy to integrate new drivers binding to an abstract class, it is easy to reason about due to logical capabilities and at the same time provides granular tuning due to the fine-integration with rpc. Peer discovery and policy or bandwidth/memory DoS-protection are all possible driver add-ons as well. One could envision an rpc like sendemergencytx, broadcasting a transaction on any transport protocol available, like a LN's justice transaction after detection of being eclipsed.

This module could be made entirely optional at compile-time with --enable-altnet even if such integration with the build system hasn't been made.

Obviously this is an exploratory PoC, I don't expect code here to be merged. Beyond worthiness of doing this in Rust there are a lot of open questions, including the threading/memory model, interface with existing codebase, build system, resources consumption and fault-tolerance. As next steps, I would favor building on the multiprocess work and hosting this new network stack in its own to avoid any bug contamination from introducing complex driver. I encourage those with opinions on the higher level motivation and design discussion to leave their ideas on the linked issue.

High-level design:

          ___________            ______________________
         |           |          |                      |
         | CWatchdog |----------|  AltLogicValidation  |
         |___________|          |______________________|
                                           |
                                     ______|______
                                    |             |
                                    |  CAltstack  |                                                         
                                    |_____________| 
                                           |
                                       ____|____
                                      |         |
                                      | CDriver |
                                      |_________|
                                           |
                                   ________|__________
                                  |                  |
                                  | ClightningDriver |
                                  |__________________|

The Demo

This PoC features a header-fetching round-trip, leveraging LN connection protocol. Each LN node connects to its peers through an encrypted and authenticated channel. Since your LN peer must have chain access, you could leverage this privileged link to create a redundant chain access.

Using C-lightning custom LN message support and plugin API, we could embed directly inside HEADERS/GETHEADERS and realize a round-trip between 2 disconnected bitcoin nodes through their respective C-lightning nodes.

For this to work, you need to compile and run 2 C-lightning nodes, each with its own data directory and launched with this custom bridge plugin. Once those LN nodes are running, activate bridges by lightning-cli alice_on and lightning-cli bob_on. The plugin doesn't seem to accept CLI argument so I had to hardcode some node addresses.

Then compile 2 bitcoin nodes with this patchset. For one of them you must change the hardcoded port in C-lightning driver and deactivate the watchdog logic. Also, it would be best if you changed SCAN_INTERVAL to something reasonable, for instance 1 min.

High-level setup:

                                             -->

                __________          LN_msg(`GETHEADERS`)                   ________
               |          |                                               |        |
               | CL-Alice |-----------------------------------------------| CL-Bob |
               |__________|                                               |________|           
                    |               LN_msg(`HEADERS`)                         |
                ____|_____                                                 ___|_____
               |          |                 <--                           |         |
               |  bridge  |                                               | bridge  |
               |__________|                                               |_________|
                    |                                                         |
                ____|_____                                                 ___|____
               |          |                                               |        |
               | BC-Alice |                                               | BC-Bob |
               |__________|                                               |________|

                  HEIGHT                                                   HEIGH + 1

Running both nodes, they should automatically connect to the bridge, and after anomaly detection realize a header exchange, if one of them is at least one block forward (run generatetoaddress 1 $ADDR).

Otherwise you may observe past logs:

Alice:

Ê0-05-16T00:55:11Z Boostrapping Altstack
2020-05-16T00:55:11Z init message: Done loading
2020-05-16T00:55:11Z msghand thread start
2020-05-16T00:55:11Z drivers-warmup thread start
2020-05-16T00:55:11Z addcon thread start
2020-05-16T00:55:11Z Registered driver with 0
2020-05-16T00:55:11Z net thread start
2020-05-16T00:55:11Z altstack-processing thread start
2020-05-16T00:55:11Z Clightning - Warmup
2020-05-16T00:55:11Z dnsseed thread start
2020-05-16T00:55:11Z altstack-handle thread start
2020-05-16T00:55:11Z 0 addresses found from DNS seeds
2020-05-16T00:55:11Z dnsseed thread exit
2020-05-16T00:55:11Z Clightning - Bridge connection
2020-05-16T00:55:11Z Registering node_id 0 with driver_id 0
2020-05-16T00:55:11Z drivers-warmup thread exit
2020-05-16T00:55:16Z Block header anomalie detected - notifying subscribers
2020-05-16T00:55:16Z Block header anomalie detected - Anycasting header fetching from 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206
2020-05-16T00:55:16Z Clightning - Preprocessing getheader
2020-05-16T00:55:16Z Cligthning - Bridge send success 71
2020-05-16T00:55:19Z Cligtning - Bridge receive success
2020-05-16T00:55:19Z Receive HEADERS from 0
2020-05-16T00:55:19Z Receiver header 65084efeaffe9929de0a0bb747cf48ac7f9b4edb214b2d1ad2ea468977be8e1e vs tip 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206

Bob:

2020-05-16T00:55:09Z Boostrapping Altstack
2020-05-16T00:55:09Z addcon thread start
2020-05-16T00:55:09Z opencon thread start
2020-05-16T00:55:09Z msghand thread start
2020-05-16T00:55:09Z drivers-warmup thread start
2020-05-16T00:55:09Z Registered driver with 0
2020-05-16T00:55:09Z Clightning - Warmup
2020-05-16T00:55:09Z init message: Done loading
2020-05-16T00:55:09Z dnsseed thread exit
2020-05-16T00:55:09Z altstack-processing thread start
2020-05-16T00:55:09Z altstack-handle thread start
2020-05-16T00:55:09Z Clightning - Bridge connection
2020-05-16T00:55:09Z Registering node_id 0 with driver_id 0
2020-05-16T00:55:09Z drivers-warmup thread exit
2020-05-16T00:55:10Z Leaving InitialBlockDownload (latching to false)
2020-05-16T00:55:17Z Cligtning - Bridge receive success
2020-05-16T00:55:17Z Receive GETHEADERS from 0
2020-05-16T00:55:17Z Fetch header starting from 0f9188f13cb7b2c71f2a335e3a4fc328bf5beb436012afca590b1a11466e2206, tip 65084efeaffe9929de0a0bb747cf48ac7f9b4edb214b2d1ad2ea468977be8e1e
2020-05-16T00:55:17Z Sending back 1 headers
2020-05-16T00:55:17Z Clightning - Preprocessing header
2020-05-16T00:55:17Z Cligthning - Bridge send success 84
2020-05-16T00:56:10Z Adding fixed seed nodes as DNS doesn't seem to be available.

Antoine Riard added 25 commits May 16, 2020
CWatchdog is a cross-layer anomalies detection module aims to
cover block issuance, peer management, local clock or net level.

If any anomalie is detected an event is triggered to inform an
application layer or any internal consumer of a future watchdog
interface.

This code is only integrated in next commit.
CWatchdogInterface is consumed by another module or application
willingly to take corrective actions based on anomalies detected.

CWatchdogInterface only callback is BlockHeaderAnomalies, meaning
block header aren't received at a normal rate modulo block variance.
CWatchdog is integrated in NodeContext to make it accessible to any
futute rpc calls. Some application may fine-tune watchdog according
to their requirements.

This code is not scheduled yet.
Run ScanAnomalies to process any heuristics and trigger watchdog
interface callback.

As of present commit, there is no heuristic implemented, callback
is blankly called to exerce watchdog interface.
Insert watchdog LogHeader after a successful ProcessNewBlockHeaders,
therefore updating last time watchdog have seen a step forward
on headers tree construction.

Watchdog loggers should be designed on being invasive-minimal at they
may encumber hot paths.
CAltstack is a secondary network stack for alternative transports.
It aims to provide a pluggable framework to host diverse transports,
like obfuscated traffic, radio-based, protocols piggy-backing, ...

This class introduces a foundation, followed-up in next commits.
This new class helps to communicate identifier payload between low-level
alternative stack drivers and message processing logic.

Add vRecvMsg to CAltstack to serve as a buffer between drivers processing
and messages processing.
CAltstack is integrated in NodeContext to make it accessible to any
future rpc calls. Some application may load alternative drivers
according to their requirements.
CDriver is an abstract interface which must be implemented by each
transport protocol driver to interface with CAltstack.

It contains basic methods to separate warmup, preprocessing, flusing
and receiving.

This is used in next commit.
Add a thread iterating over all drivers and warmup, associating to
each a driver id.
ThreadHandleDrivers is the logical equivalent of CConnman::ThreadSocketHandler,
listen for incoming connections, serve and receive messages per-transport
protocol drivers.
ThreadAltProcessing is the logical equivalent of Cconnman::ThreadAltProcessing,
processing messages to and from drivers.
This new class represents all capabilities supported by driver and must
be inherited to peer processing state.

Capabilities may be relatives to link bidirectionality, protocol
messages supported, privacy niceness, priority.
This new class aims to process protocol messages from alternatives
transports. As those may not be bidirectional, prohibitive, privacy-
leaking or size-scoped, processing logic needs to do different
assumptions based on capabilities.

This is used in next commit.
AltLogicValidation is integrated in NodeContext to make it accessible
to any future rpc calls. Some application may favor protocol messages
differently according to their requirement.
This new class encapsulate a alternative validation-specific state.
It makes available peer capabilites to processing layer.

This design is exploratory, per-peer state may be dropped completely
and processing logic may only rely on drivers capabilities to make
decision.
PushMessage dispatch message to the requested driver.
At GETHEADERS reception if peer capabilities support sending we
reply back with headers.
"Anycast" is a header-fetching strategy where any abstract drivers
supporting header and bidirectionality is send a GETHEADERS, aiming
to receive back a longuer-valid headers chain.
As this whole patch is a PoC, threading and locking model is
non-robust.
At HEADERS reception we print headers received against our
on tip.

This processing may be easily plug to some validation interface
to call ProcessNewBlockHeaders.
ClightningDriver connects through a bridge to a C-lightning node
to exchange headers on the LN transport layer through custom LN
messages.

Plugin uses C-lightning's `sendcustommessage` rpc aand "custommsg" hook.
@icota
Copy link
Contributor

icota commented May 16, 2020

Exciting stuff but I question the logic of there being a framework just for "alternatives". Shouldn't ipv4/6 and onion be one of many drivers with their own capabilities and trustworthiness?

More work initially but probably easier maintenance down the road and a guarantee that any type of smoke signal protocol driver could be a first-class citizen like ip.

Concept NACK for not being ambitious enough 😉

@DrahtBot
Copy link
Contributor

DrahtBot commented May 16, 2020

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@ryanofsky
Copy link
Contributor

ryanofsky commented Jun 19, 2020

@ryanofsky , thanks for all the links, I will look at it in more details when I need it but it still looks a bit complicate, for simple testing, is there a simple way (modif in the bitcoin core code) to redirect/listen the bitcoin protocol via stdin/stdout?

With #10102 you could start unmodified bitcoin-node or bitcoin-wallet binaries via socat if you wanted to control them with the ipc protocol over stdin/stdout:

socat - system:"src/bitcoin-node -ipcfd 4",fdin=4,fdout=4

I could actually tweak the multiprocess spawning code to communicate over stdin/stdout instead of a separate socketpair descriptor, so you wouldn't need socat. But one reason the implementation avoids stdin/stdout is that it makes printf debugging awkward, and one stray printf or -printtoconsole anywhere in the codebase messes up the IPC connection.

Also as mentioned #18988 (comment), with 01c31f9 from branch ryanofsky@ipc-export, you can connect to an already running bitcoin-node through a <datadir>/sockets/node.sock socket, which is probably what you would want to do as an external application, rather than starting a bitcoin process.

Let me know what you're trying to do though, since I'm not really sure what information's relevant here.

@Ayms
Copy link

Ayms commented Jun 19, 2020

What I want to do is:

bitcoin | node-Tor | bitcoin via stdin/stdout or bitcoin <-> node-Tor via unix sockets
		|
	send/receive
	via Tor or
	node-Tor nodes
		|
	optional: node-Tor RDV point
		|
	node-Tor or Tor nodes
		|
	bitcoin node via Tor exit node
		or
	bitcoin <-> node-Tor process

This is one phase of some funding proposals for the more global Convergence concepts, I will let you know when/if it happens and maybe altnet by that time will have evolved, if too complicate to setup another option is to try this with bitcoin-transactions which implement a part of the bitcoin protocol, and/or to combine both so the node-Tor node above can be a browser also (WebRTC/WS)

@ryanofsky
Copy link
Contributor

ryanofsky commented Jun 19, 2020

What I want to do is:

bitcoin | node-Tor | bitcoin via stdin/stdout or bitcoin <-> node-Tor via unix sockets

This is really interesting. Maybe it would be good to create a specific issue to discuss this here or in the node-Tor github repo.

If I were trying to connect bitcoind <--> node-Tor, I think the simplest approach to start with would be to get existing bitcoind -onion and -listenonion options to work with node-Tor. To do this, no bitcoind changes should be needed and node-Tor should just need to support simple subsets of the tor control and socks5 protocols (if it doesn't already). To avoid the need to use plumbing tools like socat, or to modify bitcoind, it would probably be easiest to just start with plain tcp sockets instead of unix sockets or stdin/stdout.

After this worked, I'd start looking into more changes to streamline and package the functionality better:

  1. Changing -onion and -listenonion to support unix sockets or pipes instead of tcp
  2. Using RFC: Introducing AltNet, a pluggable framework for alternative transports #18988 (this PR) to write a new altnet driver that can speak socks5/torcontrol to node-Tor. No need to mess with multiprocess: Add basic spawn and IPC support #19160 (multiprocess pr) if the altnet driver is written in c++ or can be part of bitcoin core, but you might use multiprocess: Add basic spawn and IPC support #19160 if you want to write the altnet driver in a different language or more independently of the bitcoin core codebase.
  3. Maybe switch from socks5/torcontrol to a custom protocol for bitcoind <--> node-Tor communication. If socks5/torcontrol don't fit well into the altnet framework, maybe do this at the same time as writing the altnet driver.

@Ayms
Copy link

Ayms commented Jun 19, 2020

This is really interesting. Maybe it would be good to create a specific issue to discuss this here or in the node-Tor github repo.

Thanks, probably @ariard can tell us if this discussion should be moved to other places, I don't find it uninteresting to compare the different approaches

node-Tor does support the socks interface since the very beginning but I don't see the point of using a potentially vulnerable localhost and weak socks protocol for processes that are running on the same machine, we saw not a long time ago that it could lead to security breaches for bitcoin

The Tor project (which is designed for browsing) does not really have the choice today (also for Snowflake), for node-Tor I just don't want to use it, and just don't need it, because browsers for example are not browsing anything but acting as Tor protocol relays/nodes (paradoxically they can't browse anything via WebRTC/WS/xhr interfaces)

So I think that a better approach is pluggable layers with altnet that can be chained including unix sockets and including browser interfaces

What I am proposing can look "ambitious/disruptive or whatever" but I am not far from it and those ideas are not just from yesterday, it's just a matter of time to do it and that's just a part of the Convergence project, 3 with unix sockets looks to be the right option for a universal nodejs driver (not talking about a specific driver for node-Tor, it should work for other nodejs programs), writing it myself... most likely no, I could do it but I can't pretend to be a C/C++ expert for whom it should probably be a piece of cake to do it

ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Jun 29, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Jul 7, 2020
multiprocess: Add basic spawn and IPC support

This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Jul 10, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Aug 10, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Aug 11, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Aug 17, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Aug 24, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Aug 26, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Sep 1, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Sep 24, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Oct 1, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Oct 1, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Oct 2, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Oct 21, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node`, `bitcoin-gui`, and `bitcoin-wallet` executables built with
`--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The LocalInit::spawnProcess
and IpcProcess::serve methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Nov 25, 2020
multiprocess: Add basic spawn and IPC support

This commit adds basic process spawning and IPC method call support to
`bitcoin-node` executables built with `--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The Ipc::spawnProcess and
Ipc::serveProcess methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Nov 25, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node` executables built with `--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The Ipc::spawnProcess and
Ipc::serveProcess methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Dec 8, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node` executables built with `--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The Ipc::spawnProcess and
Ipc::serveProcess methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Dec 8, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node` executables built with `--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The Ipc::spawnProcess and
Ipc::serveProcess methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Dec 11, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node` executables built with `--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The Ipc::spawnProcess and
Ipc::serveProcess methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
ryanofsky added a commit to ryanofsky/bitcoin that referenced this issue Dec 18, 2020
This commit adds basic process spawning and IPC method call support to
`bitcoin-node` executables built with `--enable-multiprocess`.

These changes are used in bitcoin#10102 to let
node, gui, and wallet functionality run in different processes, and in PRs
after that to allow gui and wallet processes to be started and stopped
independently and connect to the node over a socket.

These changes can also be used to implement new functionality outside the
bitcoin-node process like external indexes or pluggable transports
(bitcoin#18988). The Ipc::spawnProcess and
Ipc::serveProcess methods added here are entry points for spawning a child
process and serving a parent process, and being able to make bidirectional,
multithreaded method calls between the processes. A simple example of this is
implemented in the next commit "Add echoipc RPC method and test."
laanwj added a commit to bitcoin-core/gui that referenced this issue Apr 27, 2021
…pport

84934bf multiprocess: Add echoipc RPC method and test (Russell Yanofsky)
7d76cf6 multiprocess: Add comments and documentation (Russell Yanofsky)
ddf7ecc multiprocess: Add bitcoin-node process spawning support (Russell Yanofsky)
10afdf0 multiprocess: Add Ipc interface implementation (Russell Yanofsky)
745c9ce multiprocess: Add Ipc and Init interface definitions (Russell Yanofsky)
5d62d7f Update libmultiprocess library (Russell Yanofsky)

Pull request description:

  This PR is part of the [process separation project](https://github.com/bitcoin/bitcoin/projects/10).

  ---

  This PR adds basic process spawning and IPC method call support to `bitcoin-node` executables built with `--enable-multiprocess`[*].

  These changes are used in bitcoin/bitcoin#10102 to let node, gui, and wallet functionality run in different processes, and extended in bitcoin/bitcoin#19460 and bitcoin/bitcoin#19461 after that to allow gui and wallet processes to be started and stopped independently and connect to the node over a socket.

  These changes can also be used to implement new functionality outside the `bitcoin-node` process like external indexes or pluggable transports (bitcoin/bitcoin#18988). The `Ipc::spawnProcess` and `Ipc::serveProcess` methods added here are entry points for spawning a child process and serving a parent process, and being able to make bidirectional, multithreaded method calls between the processes. A simple example of this is implemented in commit "Add echoipc RPC method and test."

  Changes in this PR aside from the echo test were originally part of #10102, but have been split and moved here for easier review, and so they can be used for other applications like external plugins.

  Additional notes about this PR can be found at https://bitcoincore.reviews/19160

  [*] Note: the `--enable-multiprocess` feature is still experimental, and not enabled by default, and not yet supported on windows. More information can be found in [doc/multiprocess.md](https://github.com/bitcoin/bitcoin/blob/master/doc/multiprocess.md)

ACKs for top commit:
  fjahr:
    re-ACK 84934bf
  ariard:
    ACK 84934bf. Changes since last ACK fixes the silent merge conflict about `EnsureAnyNodeContext()`. Rebuilt and checked again debug command `echoipc`.

Tree-SHA512: 52a948b5e18a26d7d7a09b83003eaae9b1ed2981978c36c959fe9a55abf70ae6a627c4ff913a3428be17400a3dace30c58b5057fa75c319662c3be98f19810c6
@fanquake
Copy link
Member

fanquake commented May 12, 2022

Given that #18987 (which this is based on) has been closed, I'm going to close this for now. I think further discussion of this in issues is potentially going to be more productive.

@fanquake fanquake closed this May 12, 2022
@ariard
Copy link
Member Author

ariard commented May 12, 2022

Still eager to collect feedbacks on the design. Blocker is currently the internal plumbing to communicate across processes. State of the discussion is here : chaincodelabs/libmultiprocess#56

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet