[qt] Add abstraction layer for accessing node and wallet functionality from gui

[ryanofsky]

This is based on #10600. The non-base commits are listed below.

This change:

1. Creates abstract Node & Wallet interfaces in src/ipc/interfaces.h with implementations in src/ipc/local/interfaces.cpp
2. Updates Qt code to call the new interfaces. This largely consists of diffs of the form:

-    InitLogging();
-    InitParameterInteraction();
+    ipcNode.initLogging();
+    ipcNode.initParameterInteraction();

This change allows followup PR #10102 (makes bitcoin-qt control bitcoind over an IPC socket) to work without any significant updates to Qt code (the only change in Qt code there is ryanofsky/bitcoin@ab0afba#diff-8c9d79ba40bf702f01685008550ac100 which switches the Qt IPC protocol from LOCAL to CAPNP)

Other benefits of this change:

• It provides a single place (src/ipc/interfaces.h) to describe the interface between GUI and daemon code.
• It can make better GUI testing possible, because Node and Wallet objects have virtual methods that can be overloaded for mocking.

Other notes:

• There's a short src/ipc/README.md describing the design of the new code and some ways it can be extended.
• I used python scripts hide-globals.py and replace-syms.py to identify all the places where Qt code was accessing libbitcoin global variables and calling functions accessing those global variables.
• These changes were originally part of #10102. Thanks to @JeremyRubin for the suggestion of splitting them out.

This is based on #10600. The non-base commits are:

• d81553869e Remove direct bitcoin calls from qt/bitcoin.cpp
• f15c66668d Remove direct bitcoin calls from qt/optionsmodel.cpp
• de1c51a4c5 Remove direct bitcoin calls from qt/bitcoingui.cpp
• bf5402d478 Remove direct bitcoin calls from qt/utilitydialog.cpp
• dfa9f2c66e Remove direct bitcoin calls from qt/splashscreen.cpp
• 027e602555 Remove direct bitcoin calls from qt/clientmodel.cpp
• 8d848cb63e Remove direct bitcoin calls from qt/intro.cpp
• 54e9432a56 Remove direct bitcoin calls from qt/peertablemodel.cpp
• c08ce5bbfc Remove direct bitcoin calls from qt/bantablemodel.cpp
• fec4a06f04 Remove direct bitcoin calls from qt/rpcconsole.cpp
• 6a5652d604 Remove direct bitcoin calls from qt/optionsdialog.cpp
• 8ef6f040f4 Remove most direct bitcoin calls from qt/walletmodel.cpp
• b5887682f9 Remove direct bitcoin calls from qt/coincontroldialog.cpp
• c6461dc72d Remove direct bitcoin calls from qt/addresstablemodel.cpp
• 250293f015 Remove direct bitcoin calls from qt/paymentserver.cpp
• ae06c3a2ab Remove direct bitcoin calls from qt transaction table files
• c1fbdd5ca9 Remove direct bitcoin access from qt/guiutil.cpp
• 7b91ed1757 Remove direct bitcoin calls from qt/sendcoinsdialog.cpp
• 5e49ef7b42 Add src/ipc/README.md

[laanwj]

ClientModel and WalletModel were already meant as abstraction layer for accessing the core from the GUI. What is your rationale for adding another layer?

[ryanofsky]

ClientModel and WalletModel were already meant as abstraction layer for accessing the core from the GUI. What is your rationale for adding another layer?

ClientModel and WalletModel might have been intended to be an abstraction layer, but they are not functioning like one. There are libbitcoin functions and global variables accessed all over Qt code right now. With this change, all of these calls (there are around 200 of them) are stripped out of Qt code and moved into a one file: src/ipc/local/interfaces.cpp.

[jonasschnelli]

I once did a similar thing,.. but stopped at some point and now I know why.
It's an impressive code change and I kinda like a central point (your interfaces.cpp) where communication between the node, the wallet and the GUI happens.

I also agree with @laanwj that the clientmodel (node) and the walletmodal (wallet) are originally though to be that layer.
Though, there are many violations AFAIK.

What would be the downsides of using the exiting layers (clientmodel / walletmodel) better?

[ryanofsky]

What would be the downsides of using the exiting layers (clientmodel / walletmodel) better?

If you look at the ClientModel class, you can see it is doing a lot more work than the ipc::local::Node class is. Similarly with WalletModel and ipc::local::Wallet. The ipc classes are just simple shims around low-level node and wallet functionality, while Qt objects implement higher level logic specific to our current GUI. I think ClientModel and WalletModel classes are still useful after this change. They will just have 1 job instead of 2. Instead of serving as both abstraction layers and MVC model classes, they will serve only as MVC model classes.

Also, and in more concrete terms, the reason these interfaces live outside the src/qt directory is that with #10102, they need to be accessed not only by bitcoin-qt but also by bitcoind (specifically inside the StartServer function in src/ipc/server.cpp which is called here: ryanofsky/bitcoin@ab0afba#diff-6e30027c2045842fe842430d98d099fb

[jonasschnelli]

The general IPC interface makes sense to me. The main problem I see for any type of low latency IPC/RPC is the missing asynchronity.
Take getWalletTxDetails. This IPC call may take 2-3 seconds depending on the communication protocol and database you are using. Ideally the GUI is design to handle it asynchronous (like an RPC call) otherwise this will lead to GUI thread freezes. Not sure if this would be solvable as a generic part in the IPC layer of if the wallet/GUI logic must handle it.

[ryanofsky]

The main problem I see for any type of low latency IPC/RPC is the missing asynchronity.

Not sure if you saw the comments about this in the other pr starting here: #10102 (comment)

These changes are orthogonal to event processing / blocking issues in the UI. If UI blocked before, it will still block after these changes, if UI didn't block before, it won't start blocking now because of these changes. If remote calls are too slow because of socket/serialization overhead, we can process UI events in the background while they are being made. There are many ways to accomplish this, with one possible way described in that comment above. If anything, having calls get funnelled through an IPC framework makes it easier, not harder to add more asynchronicity.

[ryanofsky]

Also would point out that Node and Wallet interfaces in ipc/interfaces.h were mainly designed with goal of changing existing Qt code as little as possible. They aren't in any way set in stone, and I would expect them to evolve over time. Probably some calls will get consolidated, others will get broken up, calls that currently return big chunks of data will be made streaming, etc.

[jonasschnelli]

Thinking again and discussing this with @sipa / @laanwj, I think we should use the existing client-/walletmodal as node/wallet abstraction (including a possible IPC abstraction).

What's missing in the first place are better asynchronous messaging between the GUI and the wallet/node.

IMO using a thread with queue for general node/wallet communication (and eventual additional threads for procedures that usually take longer) seems after a low hanging fruit with direct benefits.

Using QT slots/signals for all(most?) communication would be required anyways and would be beneficial even without IPC and therefor should be done first.

[ryanofsky]

What's missing in the first place are better asynchronous messaging between the GUI and the wallet/node.

Again I think this is (and should be) an independent issue, but if you want to flesh out some more concrete suggestions and I would be happy to hear them.

IMO using a thread with queue for general node/wallet communication (and eventual additional threads for procedures that usually take longer) seems after a low hanging fruit with direct benefits.

This is exactly what the change I was suggesting in #10102 (comment) does.

[laanwj]

Using QT slots/signals for all(most?) communication would be required anyways and would be beneficial even without IPC and therefor should be done first.

This was my point too. Making the GUI asynchronous would avoid ever hard-freezing the GUI. Modern operating systems assume that an application has crashed if its GUI thread is unresponsive. This is a priority for improving user experience. For example: Currently, if e.g. a transaction is sent while the cs_main lock is held the entire thing hangs for a moment. Ideally it would display a modal dialog with a status, or progress animation instead. There are similar issues at startup.

Sure, this is only partially related to IPC work: When the GUI already would communicate with Qt signals and slots with the core backend (similar to how RPCConsole and RPCThread communicate, for example), it could be mostly oblivious whether this backend exists in-process or communicates over a pipe.

Although it's laudable that you're working on this, it looks to me that what you are doing currently is simply replicating what we do now but replacing direct core calls with IPC calls. The drawback is that it calcifies some things that shouldn't have been designed that way in the first place (e.g. into multiple abstraction layers), making it harder to improve later.

[ryanofsky]

The drawback is that it calcifies some things

Could you be more concrete about this? I don't see how it is true. Direct calls before are still direct calls now. If we want to follow the RPCConsole / RPCExecutor model in other parts of Qt code, I don't see how any of the changes I've made for IPC make this more difficult.

[ryanofsky]

I had a look at discussion in IRC (https://botbot.me/freenode/bitcoin-core-dev/msg/84348426/)

With respect, what I think you guys are missing on the WalletModel/ClientModel topic is that the ipc::local::WalletImpl and ipc::local::NodeImpl classes in ipc/local/interfaces.cpp are only temporarily being created and invoked within the bitcoin-qt process. In the next PR they are created and run in the bitcoind process instead of bitcoin-qt. That's the reason these classes do not reside in the src/qt directory and one reason why they don't really substitute for the WalletModel/ClientModel classes. See my previous comment for details and a code pointer: #10244 (comment).

However, I do see that it is kind of silly to have cases where Qt code calls a WalletModel/ClientModel method that merely forwards to a WalletImpl/NodeImpl method. I can easily clear this up by inlining these WalletModel/ClientModel methods, which would make the classes more lean.

Also, if this PR will be too difficult to review because of its size (https://botbot.me/freenode/bitcoin-core-dev/msg/84348447/), I can easily decompose it into smaller PRs that could be gradually merged. It is already broken up into separate commits, and many of the individual commits could be further broken up (right now they try to group together related changes).

[ryanofsky]

@laanwj and @jonasschnelli can you let me know if you still see issues with this approach?

On the Qt model class issue, I pulled out a bunch of model methods that were just wrapping IPC calls so it should be clearer what actual work walletmodel.cpp and clientmodel.cpp are doing. I also added a little blurb to the IPC README describing what the distinction between Qt model classes and the IPC interface classes is supposed to be.

On the asynchronous GUI issue, I created #10504 for more discussion, but think that issue is mostly unrelated to the changes in this PR, except as far as some changes here might potentially make it easier to identify blocking calls and make them asynchronous.

[laanwj]

Concept ACK on the approach, moving things to interfaces instead of global calls is usually good as it makes it easier to see what the interface to the core is.

[jonasschnelli]

Concept ACK.
I think we could try to rebase and get this in once 0.15 has been split off