ZMQ: add publishers for wallet transactions. #10554

somdoron · 2017-06-08T13:21:56Z

There is no way to only get real time notifications of transaction that affect the wallet.
You have to do that manually by enabling zmqrawtx and filter out transactions.

I'm suggesting adding two new publisers, both for hash and raw wallet transactions.

Also topic will indicate if transaction came from mempool or block so developers can handle the transaction accordingly without a RPC round trip to bitcoind.

Tests and documentation are updated.

ryanofsky

utACK 76d3e3bb9e2d5fd5e0c570489cc377ce95cc370f. Change looks good. Left some minor comments.

ryanofsky · 2017-06-08T14:19:40Z

src/validationinterface.h

@@ -52,6 +53,8 @@ struct CMainSignals {
    boost::signals2::signal<void (const CBlockIndex *, const CBlockIndex *, bool fInitialDownload)> UpdatedBlockTip;
    /** Notifies listeners of a transaction having been added to mempool. */
    boost::signals2::signal<void (const CTransactionRef &)> TransactionAddedToMempool;
+    /** Notifies listeners of a transaction having been added to the wallet. */
+    boost::signals2::signal<void (const CTransactionRef &, uint256 hashBlock)> TransactionAddedWallet;


Probably s/TransactionAddedWallet/TransactionAddedToWallet/ for consistency

ryanofsky · 2017-06-08T14:20:36Z

src/validationinterface.h

@@ -52,6 +53,8 @@ struct CMainSignals {
    boost::signals2::signal<void (const CBlockIndex *, const CBlockIndex *, bool fInitialDownload)> UpdatedBlockTip;
    /** Notifies listeners of a transaction having been added to mempool. */
    boost::signals2::signal<void (const CTransactionRef &)> TransactionAddedToMempool;
+    /** Notifies listeners of a transaction having been added to the wallet. */
+    boost::signals2::signal<void (const CTransactionRef &, uint256 hashBlock)> TransactionAddedWallet;


Throughout the PR, should use const uint256& instead of uint256 arg type to avoid unnecessary copies.

ryanofsky · 2017-06-08T14:21:55Z

doc/zmq.md

@@ -75,6 +77,11 @@ notification `-zmqpubhashtx` the topic is `hashtx` (no null
 terminator) and the body is the hexadecimal transaction hash (32
 bytes).

+For wallet transaction notifications (both hash and tx), the topic also indicate if the transaction came from a block or mempool.


Maybe wrap these lines for consistency with rest of the file.

laanwj · 2017-06-08T14:41:41Z

Concept ACK - however some comments:

This should respect ENABLE_WALLET, for compiling without wallet support
It should respect runtime -wallet=0
Wallet events should not go through CValidationInterface, it is for validation events only. To do this correctly, zmq should subscribe to wallet events through CWallet directly.

somdoron · 2017-06-08T23:05:28Z

@ryanofsky fixed the type, wrapped the docs and using const uint256 &hashBlock
@laanwj using wallet directly (without going through CValidationInterface) and respecting ENABLE_WALLET. Respecting -wallet=0 by checking if pwalletMain is not null.

somdoron · 2017-06-10T20:39:46Z

@ryanofsky @laanwj rebased the pull request on top of @jnewbery pull request #10555 and now all tests are passing

sipa · 2017-06-12T22:52:35Z

Does ZMQ have any authentication? I believe originally nothing wallet-related was exposed through it, as there may at least by privacy issues from publishing this.

somdoron · 2017-06-13T06:41:29Z

zeromq does have authentication, but it is not being used within bitcoind.
Do you think it is needed here? my only counter argument is that it should be exposed to internal network and trusted peers only.

But if you feel that it is needed I can add authentication for the wallet publishers.

laanwj · 2017-06-13T07:17:07Z

Do you think it is needed here? my only counter argument is that it should be exposed to internal network and trusted peers only.

We could just add a warning (to the option help enabling this) that the API is unauthenticated, and thus wallet notifications should not be used when the zmq endpoints are accessible to other users - they can be restricted by other means, e.g. binding locally, binding to UNIX socket, firewall, etc.

But if you feel that it is needed I can add authentication for the wallet publishers.

I'd insist on that only when adding control of the wallet to the ZMQ interface.

sipa · 2017-06-13T08:16:16Z

I have no strong opinion on the need for authentication; I just wanted to bring up that I believed that was the reason for not having wallet specific notifications in ZMQ before.

maflcko · 2017-06-18T12:27:42Z

Needs rebase.

somdoron · 2017-06-20T17:31:36Z

@MarcoFalke rebased

maflcko · 2017-06-20T17:52:58Z

src/zmq/zmqnotificationinterface.cpp

@@ -110,6 +116,12 @@ bool CZMQNotificationInterface::Initialize()
 void CZMQNotificationInterface::Shutdown()
 {
    LogPrint(BCLog::ZMQ, "zmq: Shutdown notification interface\n");
+
+#ifdef ENABLE_WALLET
+    if (pwalletMain)


nit: missing braces { and } for the if block, also this need adjustment for multiwallet.

somdoron · 2017-06-20T20:14:56Z

@MarcoFalke fixed and rebased

ryanofsky

utACK 2c393441b8dc55322625670a7aaf37e2df583e29, but I definitely think you should consider making CWallet::TransactionAddedToWallet static as described below to make init and shutdown less fragile.

Changes since previous review were moving the signal from validation interface to wallet, passing txids by const reference, wrapping markdown documentation, rebasing the test.

ryanofsky · 2017-06-21T16:17:45Z

src/wallet/wallet.h

@@ -1069,6 +1069,9 @@ class CWallet : public CCryptoKeyStore, public CValidationInterface
    boost::signals2::signal<void (CWallet *wallet, const uint256 &hashTx,
            ChangeType status)> NotifyTransactionChanged;

+    boost::signals2::signal<void (const CTransactionRef &ptxn,


I think you should make this a static variable to make the zmq code simpler and less fragile and remove the dependency on ::vpwallets. Advantages of making this static:

It will let you get rid of the ConnectToWalletSignals method and all the ifdefed code around it and simply connect to the signal in CZMQNotificationInterface::Initialize consistent with you how currently disconnect in CZMQNotificationInterface::Shutdown

If will make it possible in the future for new wallets to be added to ::vpwallets at runtime without having to modify zmq code and make it register for new notifications.

thanks, will make the change

ryanofsky

utACK 123c3e75e96e56a78387353d10dd05b8d73e51c1. Change since last review was a new commit making the wallet signal static.

ryanofsky · 2017-06-22T15:42:45Z

src/wallet/wallet.cpp

+ * Signal when transactions are added to wallet
+ */
+boost::signals2::signal<void (const CTransactionRef &ptxn,
+const uint256 &blockHash)> CWallet::TransactionAddedToWallet = boost::signals2::signal<void (const CTransactionRef &ptxn,


You should be able to shorten this by getting rid of the assignment (it will just call the default constructor which is fine).

fixed and rebased last commit. Do you want me to rebase everything into one commit?

fixed and rebased last commit. Do you want me to rebase everything into one commit?

No preference, either way seems fine.

ryanofsky

utACK d358230d10b00ef7e1a284d1b855fa1604bf1d28. Only change since last review was removing unneeded init assignment.

luke-jr · 2017-09-02T04:06:00Z

doc/zmq.md

+be added. Because zeromq is using prefix matching for topics
+you can subscribe to `rawwallettx` (or `hashwallettx`) to get
+both notifications. If you only want one type of notification
+subscribe to either `rawwallettx-mempool` or `rawwallettx-block`.


No documentation on how to determine which wallet this transaction involves (is it even possible?)

It was actually submitted before the multi-wallet was merged. Wallet name (or identifier) can be part of the topic.

luke-jr · 2017-09-02T04:07:17Z

src/wallet/wallet.cpp

+/*
+ * Signal when transactions are added to wallet
+ */
+boost::signals2::signal<void (const CTransactionRef &ptxn, const uint256 &blockHash)> CWallet::TransactionAddedToWallet;


IMO blockHash should be nullptr when not in a block.

luke-jr · 2017-09-02T04:07:30Z

src/wallet/wallet.cpp

+/*
+ * Signal when transactions are added to wallet
+ */
+boost::signals2::signal<void (const CTransactionRef &ptxn, const uint256 &blockHash)> CWallet::TransactionAddedToWallet;


Probably CWalletTx should be passed instead of CTransactionRef?

actually if CWalletTX is used the blockhash is not needed anymore. I will take a look.

ryanofsky · 2017-10-12T18:05:46Z

@somdoron are you still working on this? IIUC, luke's improvements could be implemented later without breaking backwards compatibility, if they are what's holding this up.

somdoron · 2017-10-17T08:12:25Z

@ryanofsky yes, will rebase today and send a PR

There is no way to get real time notification of transactions that only affect the wallet. You have to do that manually by enabling zmqrawtx and filter out transactions. I'm suggesting adding two new publisers, both for hash and raw wallet transactions. Also topic will indicate if transaction came from mempool or block so developers can handle the transaction accordingly without a RPC round trip to bitcoind.

In order to avoid the registration for wallet signals. Also enable dynamic addition of wallets without register them with ZMQ

somdoron · 2017-10-22T05:38:31Z

@ryanofsky rebased and all tests passed. Doesn't yet include @luke-jr comments.

jonasschnelli · 2017-10-23T01:22:48Z

src/zmq/zmqpublishnotifier.cpp

+    uint256 hash = transaction.GetHash();
+    LogPrint(BCLog::ZMQ, "zmq: Publish hashwallettx %s\n", hash.GetHex());
+    char data[32];
+    for (unsigned int i = 0; i < 32; i++)


~~memcpy?~~

Nevermind. I see we do reverse the byte order here...

jonasschnelli · 2017-10-23T01:26:59Z

I'm still unsure about this, if wtxs (protected by http auth) should be something we broadcast via ZMQ.
Alternative would be HTTP based long polling (#7949).

Also, how does this handle multiwallet? Should we somehow integrate the wallet identifier in the notifications?

jnewbery

Apologies for all the style nits. Please see https://github.com/bitcoin/bitcoin/blob/6157e8ce3937af3f46d3e7dd922d19d6dc272145/doc/developer-notes.md for the full style guide.

I agree with the other reviewers that this needs a bit more work for multiwallet. Can you either:

add documentation saying that multiwallet is supported but that notifications will not indicate which wallet the notification is for; or
add the wallet name to the notification topic

Both are fine. You could do (1) now and a future PR could add the name of the wallet to the notification.

Can you also update the zmq_test.py to verify that this notifications are received for all wallets when multiwallet is being used? I have a branch here that does that: https://github.com/jnewbery/bitcoin/tree/pr10554.1 . Feel free to take that and squash into your commit if you like it.

Also, I think this could use some extra documentation warning users about security implications:

zmq is unauthenticated
notifications are received for all wallets and can't be enabled/disabled on a per-wallet basis.

jnewbery · 2017-10-23T13:57:42Z

src/zmq/zmqnotificationinterface.cpp

@@ -10,6 +10,10 @@
 #include "streams.h"
 #include "util.h"

+#ifdef ENABLE_WALLET
+#include "../wallet/wallet.h"


There only seem to be a couple of other relative includes in the codebase. Other files use:

#include "wallet/wallet.h"

jnewbery · 2017-10-23T14:00:08Z

src/zmq/zmqnotificationinterface.cpp

+    for (std::list<CZMQAbstractNotifier*>::iterator i = notifiers.begin(); i!=notifiers.end(); )
+    {
+        CZMQAbstractNotifier *notifier = *i;
+        if (notifier->NotifyWalletTransaction(tx, hashBlock))


style nit: braces on same line please.

jnewbery · 2017-10-23T14:00:33Z

src/zmq/zmqnotificationinterface.cpp

+        {
+            i++;
+        }
+        else


style nit:

} else {

jnewbery · 2017-10-23T14:04:25Z

src/zmq/zmqpublishnotifier.cpp

+
+    const char *command;
+
+    if (!hashBlock.IsNull())


nit: braces please.

jnewbery · 2017-10-23T14:04:44Z

src/zmq/zmqpublishnotifier.cpp

+    uint256 hash = transaction.GetHash();
+    LogPrint(BCLog::ZMQ, "zmq: Publish hashwallettx %s\n", hash.GetHex());
+    char data[32];
+    for (unsigned int i = 0; i < 32; i++)


nit: braces please

jnewbery · 2017-10-23T14:13:25Z

src/zmq/zmqabstractnotifier.cpp

@@ -20,3 +20,7 @@ bool CZMQAbstractNotifier::NotifyTransaction(const CTransaction &/*transaction*/
 {
    return true;
 }
+
+bool CZMQAbstractNotifier::NotifyWalletTransaction(const CTransaction &transaction, const uint256 &hashBlock){


nit: function brace on newline please.

jnewbery · 2017-10-23T14:35:40Z

test/functional/zmq_test.py

@@ -22,16 +23,17 @@ def __init__(self, socket, topic):
        import zmq
        self.socket.setsockopt(zmq.SUBSCRIBE, self.topic)

-    def receive(self):
+    def receive(self, specific_topic = None):


nit: no spaces around the = for named arguments.

jnewbery · 2017-10-23T14:52:35Z

test/functional/test_framework/util.py

@@ -37,6 +37,10 @@ def assert_equal(thing1, thing2, *args):
    if thing1 != thing2 or any(thing1 != arg for arg in args):
        raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args))

+def assert_not_equal(thing1, thing2):


You've added this but not used it. If it's not being used, can you remove it from this PR?

ryanofsky

utACK ed4fd26. Only change since last review is rebase. Agree with John it'd be good to document the multiwallet limitation if you don't want to add wallet names to the notifications right now. Also I don't see any reason not to squash the two commits. Would make understanding the PR a little simpler.

Github-Pull: bitcoin#10554 Rebased-From: d6448d4

ryanofsky · 2018-01-09T20:20:48Z

For some reason travis is failing with contrib/devtools/lint-python.sh: 10: contrib/devtools/lint-python.sh: flake8: not found. It's possible a rebase might fix this.

promag · 2018-01-10T15:51:09Z

Alternative would be HTTP based long polling (#7949).

I tend to agree with @jonasschnelli alternative.

breitsmiley · 2018-05-04T21:35:12Z

What about this PR? Is it going to be added?

jnewbery · 2018-05-04T21:44:57Z

No update from PR author @somdoron since October 2017. Are you still working on this @somdoron ?

If not, I suggest we close this as 'Up for grabs'

somdoron · 2018-05-05T06:16:22Z

You can close.

I actually used the code in production as part of payments solution.

fanquake added the RPC/REST/ZMQ label Jun 8, 2017

ryanofsky reviewed Jun 8, 2017

View reviewed changes

somdoron force-pushed the zmq_wallet_tx branch from 76d3e3b to 9068773 Compare June 8, 2017 22:58

somdoron force-pushed the zmq_wallet_tx branch 4 times, most recently from f4f1370 to 09c6899 Compare June 10, 2017 19:50

somdoron force-pushed the zmq_wallet_tx branch from 09c6899 to 3c00508 Compare June 20, 2017 17:19

maflcko reviewed Jun 20, 2017

View reviewed changes

somdoron force-pushed the zmq_wallet_tx branch from 3c00508 to 2c39344 Compare June 20, 2017 19:37

ryanofsky reviewed Jun 21, 2017

View reviewed changes

somdoron force-pushed the zmq_wallet_tx branch from 3842e02 to 123c3e7 Compare June 22, 2017 14:38

ryanofsky reviewed Jun 22, 2017

View reviewed changes

somdoron force-pushed the zmq_wallet_tx branch 2 times, most recently from da7ff89 to d358230 Compare June 22, 2017 16:39

ryanofsky reviewed Jun 22, 2017

View reviewed changes

luke-jr reviewed Sep 2, 2017

View reviewed changes

somdoron force-pushed the zmq_wallet_tx branch from d358230 to 6b4224a Compare October 19, 2017 09:02

somdoron added 2 commits October 19, 2017 16:43

ZMQ: Making CWallet::TransactionAddedToWallet static

ed4fd26

In order to avoid the registration for wallet signals. Also enable dynamic addition of wallets without register them with ZMQ

somdoron force-pushed the zmq_wallet_tx branch from 6b4224a to ed4fd26 Compare October 19, 2017 13:43

jonasschnelli reviewed Oct 23, 2017

View reviewed changes

jnewbery reviewed Oct 23, 2017

View reviewed changes

ryanofsky reviewed Oct 31, 2017

View reviewed changes

luke-jr pushed a commit to bitcoinknots/bitcoin that referenced this pull request Nov 6, 2017

QA/ZMQ: Check for rawwallettx

e26209d

Github-Pull: bitcoin#10554 Rebased-From: d6448d4

ajtowns mentioned this pull request Dec 10, 2017

Add Travis check for unused Python imports #11835

Merged

TheBlueMatt mentioned this pull request Dec 19, 2017

How to synchronize access to wallet notifications? #11942

Closed

ajtowns mentioned this pull request Jan 16, 2018

[tests] Rename functional tests #11774

Merged

jnewbery closed this May 5, 2018

jnewbery added the Up for grabs label May 5, 2018

luke-jr mentioned this pull request Jan 26, 2020

wip: zmq: Support -zmqpubwallettx #17878

Closed

promag mentioned this pull request May 8, 2020

ZMQ notices just for wallet-affecting TXes? #13427

Closed

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

ZMQ: add publishers for wallet transactions. #10554

ZMQ: add publishers for wallet transactions. #10554

Conversation

somdoron commented Jun 8, 2017

ryanofsky left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laanwj commented Jun 8, 2017 • edited

somdoron commented Jun 8, 2017 • edited

somdoron commented Jun 10, 2017

sipa commented Jun 12, 2017

somdoron commented Jun 13, 2017

laanwj commented Jun 13, 2017 • edited

sipa commented Jun 13, 2017

maflcko commented Jun 18, 2017

somdoron commented Jun 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

somdoron commented Jun 20, 2017

ryanofsky left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanofsky left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanofsky left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanofsky commented Oct 12, 2017

somdoron commented Oct 17, 2017

somdoron commented Oct 22, 2017

jonasschnelli Oct 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonasschnelli commented Oct 23, 2017

jnewbery left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ryanofsky left a comment

Choose a reason for hiding this comment

ryanofsky commented Jan 9, 2018

promag commented Jan 10, 2018

breitsmiley commented May 4, 2018

jnewbery commented May 4, 2018

somdoron commented May 5, 2018

laanwj commented Jun 8, 2017 •

edited

somdoron commented Jun 8, 2017 •

edited

laanwj commented Jun 13, 2017 •

edited

ryanofsky left a comment •

edited

ryanofsky left a comment •

edited

ryanofsky left a comment •

edited

jonasschnelli Oct 23, 2017 •

edited