Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
Already on GitHub? Sign in to your account
ZMQ: add publishers for wallet transactions. #10554
Conversation
fanquake
added
the
RPC/REST/ZMQ
label
Jun 8, 2017
| @@ -75,6 +77,11 @@ notification `-zmqpubhashtx` the topic is `hashtx` (no null | ||
| terminator) and the body is the hexadecimal transaction hash (32 | ||
| bytes). | ||
| +For wallet transaction notifications (both hash and tx), the topic also indicate if the transaction came from a block or mempool. |
| @@ -52,6 +53,8 @@ struct CMainSignals { | ||
| boost::signals2::signal<void (const CBlockIndex *, const CBlockIndex *, bool fInitialDownload)> UpdatedBlockTip; | ||
| /** Notifies listeners of a transaction having been added to mempool. */ | ||
| boost::signals2::signal<void (const CTransactionRef &)> TransactionAddedToMempool; | ||
| + /** Notifies listeners of a transaction having been added to the wallet. */ | ||
| + boost::signals2::signal<void (const CTransactionRef &, uint256 hashBlock)> TransactionAddedWallet; |
ryanofsky
Jun 8, 2017
Contributor
Probably s/TransactionAddedWallet/TransactionAddedToWallet/ for consistency
| @@ -52,6 +53,8 @@ struct CMainSignals { | ||
| boost::signals2::signal<void (const CBlockIndex *, const CBlockIndex *, bool fInitialDownload)> UpdatedBlockTip; | ||
| /** Notifies listeners of a transaction having been added to mempool. */ | ||
| boost::signals2::signal<void (const CTransactionRef &)> TransactionAddedToMempool; | ||
| + /** Notifies listeners of a transaction having been added to the wallet. */ | ||
| + boost::signals2::signal<void (const CTransactionRef &, uint256 hashBlock)> TransactionAddedWallet; |
ryanofsky
Jun 8, 2017
Contributor
Throughout the PR, should use const uint256& instead of uint256 arg type to avoid unnecessary copies.
|
Concept ACK - however some comments:
|
somdoron
commented
Jun 8, 2017
•
Edited 1 time
-
somdoron
Jun 8, 2017
-
Jun 8, 2017
|
@ryanofsky fixed the type, wrapped the docs and using |
somdoron
commented
Jun 10, 2017
|
@ryanofsky @laanwj rebased the pull request on top of @jnewbery pull request #10555 and now all tests are passing |
|
Does ZMQ have any authentication? I believe originally nothing wallet-related was exposed through it, as there may at least by privacy issues from publishing this. |
somdoron
commented
Jun 13, 2017
|
zeromq does have authentication, but it is not being used within bitcoind. But if you feel that it is needed I can add authentication for the wallet publishers. |
We could just add a warning (to the option help enabling this) that the API is unauthenticated, and thus wallet notifications should not be used when the zmq endpoints are accessible to other users - they can be restricted by other means, e.g. binding locally, binding to UNIX socket, firewall, etc.
I'd insist on that only when adding control of the wallet to the ZMQ interface. |
|
I have no strong opinion on the need for authentication; I just wanted to bring up that I believed that was the reason for not having wallet specific notifications in ZMQ before. |
|
Needs rebase. |
somdoron
commented
Jun 20, 2017
|
@MarcoFalke rebased |
| @@ -110,6 +116,12 @@ bool CZMQNotificationInterface::Initialize() | ||
| void CZMQNotificationInterface::Shutdown() | ||
| { | ||
| LogPrint(BCLog::ZMQ, "zmq: Shutdown notification interface\n"); | ||
| + | ||
| +#ifdef ENABLE_WALLET | ||
| + if (pwalletMain) |
MarcoFalke
Jun 20, 2017
Member
nit: missing braces { and } for the if block, also this need adjustment for multiwallet.
somdoron
commented
Jun 20, 2017
|
@MarcoFalke fixed and rebased |
ryanofsky
reviewed
Jun 21, 2017
•
utACK 2c39344, but I definitely think you should consider making CWallet::TransactionAddedToWallet static as described below to make init and shutdown less fragile.
Changes since previous review were moving the signal from validation interface to wallet, passing txids by const reference, wrapping markdown documentation, rebasing the test.
| @@ -1069,6 +1069,9 @@ class CWallet : public CCryptoKeyStore, public CValidationInterface | ||
| boost::signals2::signal<void (CWallet *wallet, const uint256 &hashTx, | ||
| ChangeType status)> NotifyTransactionChanged; | ||
| + boost::signals2::signal<void (const CTransactionRef &ptxn, |
ryanofsky
Jun 21, 2017
Contributor
I think you should make this a static variable to make the zmq code simpler and less fragile and remove the dependency on ::vpwallets. Advantages of making this static:
- It will let you get rid of the
ConnectToWalletSignalsmethod and all the ifdefed code around it and simply connect to the signal inCZMQNotificationInterface::Initializeconsistent with you how currently disconnect inCZMQNotificationInterface::Shutdown - If will make it possible in the future for new wallets to be added to ::vpwallets at runtime without having to modify zmq code and make it register for new notifications.
ryanofsky
reviewed
Jun 22, 2017
•
utACK 123c3e7. Change since last review was a new commit making the wallet signal static.
| + * Signal when transactions are added to wallet | ||
| + */ | ||
| +boost::signals2::signal<void (const CTransactionRef &ptxn, | ||
| +const uint256 &blockHash)> CWallet::TransactionAddedToWallet = boost::signals2::signal<void (const CTransactionRef &ptxn, |
ryanofsky
Jun 22, 2017
Contributor
You should be able to shorten this by getting rid of the assignment (it will just call the default constructor which is fine).
somdoron
Jun 22, 2017
fixed and rebased last commit. Do you want me to rebase everything into one commit?
ryanofsky
Jun 22, 2017
Contributor
fixed and rebased last commit. Do you want me to rebase everything into one commit?
No preference, either way seems fine.
ryanofsky
reviewed
Jun 22, 2017
•
utACK d358230. Only change since last review was removing unneeded init assignment.
| +be added. Because zeromq is using prefix matching for topics | ||
| +you can subscribe to `rawwallettx` (or `hashwallettx`) to get | ||
| +both notifications. If you only want one type of notification | ||
| +subscribe to either `rawwallettx-mempool` or `rawwallettx-block`. |
luke-jr
Sep 2, 2017
Member
No documentation on how to determine which wallet this transaction involves (is it even possible?)
somdoron
Sep 3, 2017
It was actually submitted before the multi-wallet was merged. Wallet name (or identifier) can be part of the topic.
| +/* | ||
| + * Signal when transactions are added to wallet | ||
| + */ | ||
| +boost::signals2::signal<void (const CTransactionRef &ptxn, const uint256 &blockHash)> CWallet::TransactionAddedToWallet; |
| +/* | ||
| + * Signal when transactions are added to wallet | ||
| + */ | ||
| +boost::signals2::signal<void (const CTransactionRef &ptxn, const uint256 &blockHash)> CWallet::TransactionAddedToWallet; |
somdoron
Sep 4, 2017
actually if CWalletTX is used the blockhash is not needed anymore. I will take a look.
|
@somdoron are you still working on this? IIUC, luke's improvements could be implemented later without breaking backwards compatibility, if they are what's holding this up. |
somdoron
commented
Oct 17, 2017
|
@ryanofsky yes, will rebase today and send a PR |
somdoron
added some commits
Jun 8, 2017
somdoron
commented
Oct 22, 2017
|
@ryanofsky rebased and all tests passed. Doesn't yet include @luke-jr comments. |
| + uint256 hash = transaction.GetHash(); | ||
| + LogPrint(BCLog::ZMQ, "zmq: Publish hashwallettx %s\n", hash.GetHex()); | ||
| + char data[32]; | ||
| + for (unsigned int i = 0; i < 32; i++) |
|
I'm still unsure about this, if wtxs (protected by http auth) should be something we broadcast via ZMQ. Also, how does this handle multiwallet? Should we somehow integrate the wallet identifier in the notifications? |
jnewbery
reviewed
Oct 23, 2017
Apologies for all the style nits. Please see https://github.com/bitcoin/bitcoin/blob/6157e8ce3937af3f46d3e7dd922d19d6dc272145/doc/developer-notes.md for the full style guide.
I agree with the other reviewers that this needs a bit more work for multiwallet. Can you either:
- add documentation saying that multiwallet is supported but that notifications will not indicate which wallet the notification is for; or
- add the wallet name to the notification topic
Both are fine. You could do (1) now and a future PR could add the name of the wallet to the notification.
Can you also update the zmq_test.py to verify that this notifications are received for all wallets when multiwallet is being used? I have a branch here that does that: https://github.com/jnewbery/bitcoin/tree/pr10554.1 . Feel free to take that and squash into your commit if you like it.
Also, I think this could use some extra documentation warning users about security implications:
- zmq is unauthenticated
- notifications are received for all wallets and can't be enabled/disabled on a per-wallet basis.
| @@ -20,3 +20,7 @@ bool CZMQAbstractNotifier::NotifyTransaction(const CTransaction &/*transaction*/ | ||
| { | ||
| return true; | ||
| } | ||
| + | ||
| +bool CZMQAbstractNotifier::NotifyWalletTransaction(const CTransaction &transaction, const uint256 &hashBlock){ |
| @@ -10,6 +10,10 @@ | ||
| #include "streams.h" | ||
| #include "util.h" | ||
| +#ifdef ENABLE_WALLET | ||
| +#include "../wallet/wallet.h" |
jnewbery
Oct 23, 2017
Member
There only seem to be a couple of other relative includes in the codebase. Other files use:
#include "wallet/wallet.h"
| + for (std::list<CZMQAbstractNotifier*>::iterator i = notifiers.begin(); i!=notifiers.end(); ) | ||
| + { | ||
| + CZMQAbstractNotifier *notifier = *i; | ||
| + if (notifier->NotifyWalletTransaction(tx, hashBlock)) |
| + { | ||
| + i++; | ||
| + } | ||
| + else |
| + uint256 hash = transaction.GetHash(); | ||
| + LogPrint(BCLog::ZMQ, "zmq: Publish hashwallettx %s\n", hash.GetHex()); | ||
| + char data[32]; | ||
| + for (unsigned int i = 0; i < 32; i++) |
| + | ||
| + const char *command; | ||
| + | ||
| + if (!hashBlock.IsNull()) |
| @@ -37,6 +37,10 @@ def assert_equal(thing1, thing2, *args): | ||
| if thing1 != thing2 or any(thing1 != arg for arg in args): | ||
| raise AssertionError("not(%s)" % " == ".join(str(arg) for arg in (thing1, thing2) + args)) | ||
| +def assert_not_equal(thing1, thing2): |
jnewbery
Oct 23, 2017
Member
You've added this but not used it. If it's not being used, can you remove it from this PR?
| @@ -22,16 +23,17 @@ def __init__(self, socket, topic): | ||
| import zmq | ||
| self.socket.setsockopt(zmq.SUBSCRIBE, self.topic) | ||
| - def receive(self): | ||
| + def receive(self, specific_topic = None): |
ryanofsky
reviewed
Oct 31, 2017
utACK ed4fd26. Only change since last review is rebase. Agree with John it'd be good to document the multiwallet limitation if you don't want to add wallet names to the notifications right now. Also I don't see any reason not to squash the two commits. Would make understanding the PR a little simpler.
added a commit
to bitcoinknots/bitcoin
that referenced
this pull request
Nov 6, 2017
somdoron commentedJun 8, 2017
There is no way to only get real time notifications of transaction that affect the wallet.
You have to do that manually by enabling zmqrawtx and filter out transactions.
I'm suggesting adding two new publisers, both for hash and raw wallet transactions.
Also topic will indicate if transaction came from mempool or block so developers can handle the transaction accordingly without a RPC round trip to bitcoind.
Tests and documentation are updated.