Multiprocess tracking issue

[ryanofsky]

This issue will be updated to reflect the current state of bitcoin core multiprocess support.

---

PRs for review related to building & releasing:

• init: reserve file descriptors for IPC connections #34978
• ipc: Support for windows support #35084
• fuzz: add ipc round-trip fuzz target #35118
• ipc: add windows support #32387
• Upstream libmultiprocess pulls

PRs for review related to mining:

• mining: add getMemoryLoad() and track template non-mempool memory footprint #33922
• mining: add getTransactions(ByWitnessID) IPC methods #34020
• mining: add reason/debug to submitSolution and unify with submitBlock #34672
• ipc: add bitcoin-mine test program #30437
• node: add BlockTemplateCache #33421
• mining: add reason/debug to submitSolution and unify with submitBlock #34672
• ipc: support per-address max-connections options on -ipcbind #35037
• mining: pr 33966 followups (disentangle miner startup defaults) #35403
• Tracking issue Mining interface tracking issue #33777

PRs for review related wallet & gui & cli:

• multiprocess: Add capnp wrapper for Chain interface #29409
• Multiprocess bitcoin #10102
• multiprocess: Add bitcoin-wallet -ipcconnect option #19460
• multiprocess: Add bitcoin-gui -ipcconnect option #19461
• POC: IPC tracing interface #32898

PRs for review related to bitcoin wrapper

• contrib: zsh and fish completions for bitcoin wrapper executable #33747
• contrib: Add zsh completion scripts #34906

Issues:

• RFC: Add multiprocess fuzz target #23015
• upstream: capnp V2 doesn't support compilation with GCC (yet?) #32669
• Assertion hit on shutdown of bitcoin-node with connected mining interface client #33387
• Header-only support for waitNext() #33756
• Block template memory management (for IPC clients) #33899
• Memory leak when using IPC mining interface #33940
• i386 segv in #0 0xf7a49ce0 in kj::operator*(kj::_::Stringifier, kj::Exception const&) (/lib/i386-linux-gnu/libkj-1.1.0.so+0x38ce0) #34732
• ci: subtree lint scope #34928
• multiprocess: expose existing interfaces or design new ones? #34981
• RFC: IPC based tracing interface (alternative to eBPF/USDT) #35142
• ci: spurious output / errors in IWYU job #35361
• IPC docs bitcoin-core/bitcoincore.org#1247

Branches:

• drop-libmultiprocess - @willcl-ark proof of concept showing tradeoffs of using libmultiprocess, how to use plain capnproto for new interfaces

PRs / issues closed:

• [qt] Don't call method on null WalletModel object #10242
• [qt] Move some WalletModel functions into CWallet #10295
• Make feebumper class stateless #10600
• Refactor: separate gui from wallet and node #10244
• Add AssertLockHeld assertions in CWallet::ListCoins #10605
• Avoid using numeric_limits for sequence numbers and lock times #14636
• Refactor: Start to separate wallet from node #14437
• Remove uses of chainActive and mapBlockIndex in wallet code #14711
• Suggested wallet code cleanups from #14711 #15342
• Move ParseConfirmTarget from rpc/mining to rpc/util #15373
• Remove wallet -> node global function calls #15288
• Suggested interfaces::Chain cleanups from #15288 #15531
• Refactor: separate wallet from node #10973
• Replace Connman and BanMan globals with NodeContext local #16839
• gui: Set CConnman byte counters earlier to avoid uninitialized reads #17906
• interfaces: Describe and follow some code conventions #18278
• Multiprocess build support #16367
• refactor: Add ChainClient setMockTime, getWallets methods #17999
• depends: Add --sysroot option to mac os native compile flags #18743
• gui: Avoid wallet tryGetBalances calls in WalletModel::pollBalanceChanged #18587
• Multiprocess build support #18677
• refactor: Move node and wallet code out of src/interfaces #20494
• depends: Set CMAKE_INSTALL_RPATH for native packages #20046
• Remove pointer cast in CRPCTable::dumpArgMap #21035
• MOVEONLY: Move common init code to init/common #21732
• multiprocess: Add basic spawn and IPC support #19160
• refactor: Make SetupServerArgs callable without NodeContext #22216
• refactor: Rearrange fillPSBT arguments #22214
• multiprocess: Add ipc::Context and ipc::capnp::Context structs #22218
• Unregister wallet notifications before unloading wallets bitcoin-core/gui#360
• refactor: Add FoundBlock.found member #22215
• refactor: Avoid wallet code writing node settings file #22217
• refactor: remove ::vpwallets and related global variables #19101
• MOVEONLY: Expose BanMapToJson / BanMapFromJson #22848
• multiprocess: Start using init makeNode, makeChain, etc methods #22219
• multiprocess: Make interfaces::Chain::isTaprootActive non-const #23003
• multiprocess: Add new bitcoin-gui, bitcoin-qt, bitcoin-wallet init implementations #23006
• multiprocess: add interfaces::ExternalSigner class #23004
• multiprocess: Delay wallet client construction #23005
• multiprocess: Add interfaces::Node::broadCastTransaction method #23499
• multiprocess compatibility updates #28721
• depends: bump libmultiprocess to fix capnproto deprecation warnings #28907
• doc: Add multiprocess design doc #28978
• multiprocess: Add basic type conversion hooks #28921
• serialization: Support for multiple parameters #28929
• multiprocess: Add -ipcbind option to bitcoin-node #30509
• multiprocess: Add IPC wrapper for Mining interface #30510
• Prune mining interface #31196
• refactor: mining interface 30955 followups #31197
• depends: Fix build with MULTIPROCESS=1 in Guix environment #30940
• Mining interface: getCoinbaseMerklePath() and submitSolution() #30955
• Update libmultiprocess library #31105
• Add waitFeesChanged() to Mining interface #31003
• Add destroy to BlockTemplate schema #31288
• refactor: Fix "modernize-use-starts-ends-with" clang-tidy warning #31480
• Drop script_pub_key arg from createNewBlock #31318
• Prune mining interface #31196
• depends: Update libmultiprocess library before converting to subtree #31740
• ci: intermittent issue in rpc_misc.py node0 stderr terminate called after throwing an instance of 'kj::ExceptionImpl' [15:12:14.943] what(): mp/proxy.cpp:242: disconnected: write(m_post_fd, &buffer, 1): Broken pipe #31151
• multiprocess: build failure on Alpine with depends & DEBUG=1 #31455
• Pass custom DEP_OPTS and CONFIG_FLAGS to guix-build #31763
• multiprocess: Lock CapnpProtocol::m_loop with mutex #31815
• build: disable bitcoin-node if daemon is not built #31834
• depends: Update libmultiprocess library to fix CI failures #31945
• Add waitNext() to BlockTemplate interface #31283
• Rename bitcoin-wallet? #31827
• Add checkblock RPC and checkBlock() to Mining interface #31564
• test, refactor: Add TestNode.binaries to hold binary paths #31866
• cmake: Avoid fuzzer "multiple definition of `main'" errors #31992
• multiprocess: Add libmultiprocess git subtree #31741
• ci: build multiprocess on most jobs #30975
• Have createNewBlock() wait for tip, make rpc handle shutdown during long poll and wait methods #31785
• doc: Document WITH_EXTERNAL_LIBMULTIPROCESS build option better #32271
• interfaces: refactor: move Mining and BlockTemplate implementation to miner #32378
• multiprocess: Add bitcoin wrapper executable #31375
• build: Add resource file and manifest to bitcoin.exe #32634
• Update libmultiprocess subtree to fix clang-tidy errors #32641
• depends: fix multiprocess build on OpenBSD (apply capnp patch, correct SHA256SUM command) #32690
• Add checkBlock() to Mining interface #31981
• depends: capnp 1.2.0 #32760
• refactor: modernize deprecated ipc headers #32781
• cmake: Install internal binaries to <prefix>/libexec/ #31679
• Add bitcoin-{node,gui} to release binaries for IPC #31802
• ipc: Handle unclean shutdowns better #32345
• build: Enable ENABLE_IPC option by default #33190
• miner: clamp options instead of asserting #33222
• doc: follow-ups to "Add bitcoin-{node,gui} to release binaries for IPC" #33233
• Update libmultiprocess subtree to fix build issues #33241
• depends: native_libmultiprocess fails to build on OpenBSD #33219
• build: set ENABLE_IPC to OFF when fuzzing #33235
• build: suggest -DENABLE_IPC=OFF when missing capnp #33290
• build: suggest -DENABLE_IPC=OFF when missing capnp #33315
• Add functional test for IPC interface #33201
• Update libmultiprocess subtree to improve build and logs #33322
• build: indefinite mpgen hang on Ubuntu 22.04 #33176
• doc: fix LIBRARY_PATH comment #33308
• build: clang build broken on Debian Bookworm #33279
• clang-tidy: disable clang-analyzer UndefinedBinaryOperatorResult warning #33281
• clang-tidy: Disable UndefinedBinaryOperatorResult check in src/ipc #33312
• ci: tidy job is producing output #33256
• guix: strip binaries in libexec #33342
• contrib: add bitcoin binary to gen-manpages #33348
• mining: log failed blocks in submitSolution() #33372
• test: Add submitblock test in interface_ipc #33380
• Update libmultiprocess subtree to fix intermittent mptest hang #33412
• intermittent timeout in mptest unit test #33244
• intermittent Segmentation fault in mptest IPC path (callFnAsyncParams) #33277
• mining: add applySolution() to interface #33374
• doc: remove unrelated bitcoin-wallet binary from libbitcoin_ipc description #33459
• multiprocess: Don't require bitcoin -m argument when IPC options are used #33229
• Test interface_ipc.py fails with Duplicate ID error when libmultiprocess is installed system-wide #33417
• test: Avoid interface_ipc.py Duplicate ID errors #33420
• multiprocess: ipc_tests fail on *BSD #31618
• Update libmultiprocess subtree to support reduced logging #33518
• Update libmultiprocess subtree in 30.x branch #33519
• multiprocess: Fix high overhead from message logging #33517
• [v30.0] createNewBlock never returns #33647
• Release Schedule for 30.0 #32275
• RFC: Multiprocess binaries and packaging options #30983
• Mining Interface doesn't allow for Bitcoin Core to create blocks when it wants #31109
  • caused by Add waitFeesChanged() to Mining interface #31003
  • resolved by Add waitNext() to BlockTemplate interface #31283
• RFC: Adding bitcoin-{node,gui} binaries for IPC in 30.0 release #31756
• How to backport libmultiprocess changes? #33439
• Stratum v2 via IPC Mining Interface tracking issue #31098
• test: ipc: resolve symlinks in which capnp #33749
• ci: Update Clang in "tidy" job #33445
• interfaces: enable cancelling running waitNext calls #33676
• RFC: Cancelling waitNext calls in the IPC mining interface #33575
• mining: check witness commitment in submitBlock #33745
• init: Fix Ctrl-C shutdown hangs during wait calls #33511
• [v30.0rc3]bitcoin-node aborts with mining IPC interface usage #33554
• interface_ipc functional test failing in CI #33884
• test: Fix race condition in IPC interface block progation test #33880
• mining: add requestedOutputs field, e.g. for merged mining #33890
• bitcoin-node is unkillable after mining IPC connection is established #33463
• contrib: fix manpage generation #33996
• cmake: Move IPC tests to ipc/test #33774
• doc: document capnproto and libmultiprocess deps in 29.x #33623
• consider adding a new interface RawTxFeed on Mining IPC #34030
• test: interface_ipc.py minor fixes and cleanup #34003
• depends: capnp 1.3.0 #34102
• mining: getCoinbase() returns struct instead of raw tx #33819
• Update libmultiprocess subtree to avoid occasional rpc_misc.py timeout #34363
• Intermittent issue in rpc_misc.py: test_framework.authproxy.JSONRPCException: 'echoipc' RPC took longer than 1200.000000 seconds. #34187
• 29.x: using a local libmultiprocess install will no-longer work #33576
• test: interface_ipc.py might start skipping if installed capnp version changes #34016
• mining, ipc: omit dummy extraNonce from coinbase #32420
• mining: pass missing context to createNewBlock() and checkBlock() #33936
• test: split interface_ipc.py #34452
• mining: Break compatibility with existing IPC mining clients #34568
• ipc, test: Add tests for unclean disconnect and thread busy behavior #34284
• mining: fix -blockreservedweight shadows IPC option #33965
• mining: add cooldown to createNewBlock() immediately after IBD #34184
• Mining IPC createNewBlock should not return before IBD is over #33994
• ipc mining: Prevent `Assertion m_node.chainman' failed`` errors on early startup #34661
• Update libmultiprocess subtree to be more stable with rust IPC client #34422
• should concurrent IPC requests directed to the same thread cause a crash? #33923
• crash during IPC cleanup after interruptWait() is called #34250
• doc: Improve dependencies.md IPC documentation #34706
• test: avoid interface_ipc.py race and null pointer dereference #34715
• src/ipc/libmultiprocess/src/mp/proxy.cpp:416:44: runtime error: member access within null pointer of type 'mp::Waiter' #34711
• ipc: AddressSanitizer: SEGV nptl/pthread_mutex_lock.c:80:23 in __pthread_mutex_lock #34756
• Improved CustomBuildField for std::optional in IPC/libmultiprocess #34797
• IPC server crashes on early client connection (assertion failure in proxy.h and interfaces.cpp) #34664
• doc: fix process name typo in multiprocess.md #34818
• depends: capnp 1.4.0 #34825
• rfc: virtio-vsock for RPC and IPC #33897
• refactor: generalize IPC ParseAddress to use sockaddr_storage #34845
• test: Ignore error message give from python because of PYTHON_GIL #33795
• test: Add IPC wake-up test and reuse mining context #34727
• ipc: crash on submitSolution call with invalid coinbase #33341
• contrib: Add zsh completion scripts #33402
• Update libmultiprocess subtree to fix race conditions on disconnects #34804
• bitcoin-cli: Add -ipcconnect option #32297
• IPC via TCP Sockets #32802
• Update libmultiprocess subtree in 31.x branch to fix race conditions on disconnects #34952
• ipc: AddressSanitizer: heap-use-after-free in capnp::CallContext<ipc::capnp::messages::BlockTemplate::GetBlockParams, ipc::capnp::messages::BlockTemplate::GetBlockResults>::getParams() #34777
• mptest: mp::CancelMonitor: AddressSanitizer: stack-use-after-return #34782
• spawn_tests: failed: expected WIFEXITED(status) && WEXITSTATUS(status) == 0 #34975
• mptest hangs, when run in a loop #34014
• contrib: Add bash completion for new bitcoin command #33385
• refactor: replace memcpy with std::copy and add size asserts in IPC capnp serialization #35010
• Update libmultiprocess subtree to fix test timeout #34977
• test: remove macOS REDUCE_EXPORTS exception workaround #35014
• macOS + REDUCE_EXPORTS: Cap'n Proto misidentifies kj::Exception as unknown non-KJ exception #34723
• [31.x] Backports + rc3 #35028
• ipc: support per-address max-connections options on -ipcbind #35036
• mining: always pad scriptSig at low heights, drop include_dummy_extranonce #34860
• refactor: disentangle miner startup defaults from runtime options #33966
• mining: add always_add_coinbase_commitment option #35393
• build: exclude libmultiprocess test target from compile commands #35401
• mining: add submitBlock to IPC Mining interface #34644

Links:

• Documentation
• Project board
• Wiki
• Presentation (code shown)
• Working group
• List of working groups