add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc}#4333
add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc}#4333moneromooo-monero wants to merge 1 commit intomonero-project:masterfrom
Conversation
moneromooo-monero
force-pushed
the
notify
branch
from
0417813 to
5774db7
Compare
|
If anyone has comments about security here, please say so. |
|
Firstly, I would have thought this kind of notification would be either via 0mq or registering a callback URL via the JSON-RPC interface. I like this take on how to provide a notification, but given users are generally using the RPC, it would be nicer if this was part of the RPC as opposed to a different mechanism. Secondly, if a wallet is spammed with micro txs, is there not the risk of a DDoS due to the forking and starting processes? Lastly, I really don't get what is wrong with polling. It works fine. A poll every 2 minutes to get new blocks / txs is perfectly simple and what people use currently. This PR seems to just be adding something else that needs maintaining when there is no real need. |
|
I made that one because I found somewhere saying Bitcoin was doing it that way, and people have been asking for soimeting like Bitcoin. |
|
@jtgrassie polling is not a viable solution at scale. If one have to manage a number of wallets then it will just become a resource drain. Obviously launching new application is a middle-ground between nothing and 0mq/rpc, but hey - that definitely helps! |
Actually polling is almost certainly less resource intensive than forking, launching a new process and then the process doing something (likely making a TCP request). With polling, you only have to perform 1 RPC request to check for a new block every 2 minutes and then for say 10 wallets, thats 10 TCP requests to check for new transactions. The overhead of a TCP response is far lower than N forks and the new process launches (N would be as high as the number of incoming transactions on all wallets as opposed to a deterministic amount of polling TCP request/responses). The most efficient method would be 0mq or a registered RPC callback, hence whilst this option maybe useful (and I have nothing against it being added), it's probably the least efficient option. |
|
@jtgrassie Agreed 0mq would be ideal, but if you need to take action on an unconfirmed transaction as soon as possible, polling will be more frequent than every 2 minutes. |
|
I think adding pool txes might be useful. And also a reorg notification maybe. It starts to become complicated... |
|
@jtgrassie but with polling one will have to query every single wallet, while push-on-update will launch only minimum necessary amount notification application instances. PUB/SUB over 0mq feels like an optimal solution, JSON RPC callback is the second best imo. @danrmiller block mining time in Monero is 2 mins afaik. What will be the reason to poll more often? @monero-project indeed. thanks for giving it a try. |
It does indeed! With the architecture of this PR for notifications, we could end up with a lot of command line parameters. Sorry moneromoo but this looks a bit rushed and ill thought out.
2 minute pooling is for the new block check, not necessarily for incoming txs (which could be higher frequency if needed). Not sure why anyone would want to know about unconfirmed txs anyway. Mining pool servers (which need to know about new blocks and txs) simply poll. Online wallets do the same and polling works fine in both these use cases. I just cant see this yet as anything other than fixing a problem (somewhat inefficiently) that doesn't exist, rather than improving on an existing mechanism (e.g. adding a 0mq or RPC registered callback solution - which actually would be an improvement to polling). |
Yes of course, but with multiple wallets actively receiving transactions, you could be spawning many more processes (which requires much more overhead) than multiple TCP requests. At least with polling you know exactly how many requests are needed as it's fixed to the amount of wallets. Spawning an unknown number of processes is harder to manage. Remember, this PR notifies via first forking, then launching a user supplied process, then (optionally) waiting for that process to end. That's a lot of overhead vs cheap TCP polling. |
|
@jtgrassie I agree that it does start to get complicated, and agree the better solution is 0mq or RPC registered callback. I'm just mentioning that the 2 cases you mention, mining pools and online wallets, both frequently personally tell me they have great pain around polling. Another use case I am personally familiar with where polling is not meeting the users' needs is web shopping, people who have paid you want to see you acknowledge that payment as soon as possible, and there does reach a point where polling hurts. Anyway, again I'm not disagreeing, but want to clarify that polling is not working fine for several projects. |
Which pools / wallets? I know of no pool implementation that polling is a problem for example.
Polling for a purchase is simple though. You can poll seconds after the tx (where it will be unconfirmed) and later to check it's confirmed (so you can ship product). The later check would be based on how many block confirmations the seller is comfortable with - it's deterministic (X * 2 minutes).
What projects? I know of none that don't work fine with polling. Please specify. My hunch is projects that the developers can't wrap their head around how best to poll. The mining pool servers operate fine, payment processors, online wallets... please clarify. I still cannot see how adding something worse than polling makes sense to add when we all agree on a method that would be better (0mq / RPC callback). |
I'll let you know on IRC so I'm not publicly opening people up to criticism who haven't directly involved themselves in it.
True, I take your point, yes. Thanks. |
moneromooo-monero
force-pushed
the
notify
branch
from
5774db7 to
d446041
Compare
It now does. |
Lets trigger
Right. Mining pools don't have any problems polling the daemon. |
|
You always push at least a block after you pop N (at least without user command to pop manually). It was for the wallet that it might be interesting, if a payment gets reverted. |
|
Anyone else has an opinion/argument about this ? If not, it'll get in soon. |
moneromooo-monero
force-pushed
the
notify
branch
from
d446041 to
da43b03
Compare
src/common/notify.cpp
Outdated
There was a problem hiding this comment.
Notify::Notify(const std::string &spec)
{
boost::split(args, spec, boost::is_any_of(" "));
...There was a problem hiding this comment.
The main problem is that it won't handle paths containing spaces.
There was a problem hiding this comment.
Yes, I indicated in the commit message the tokenizing is dumb. I don't mind that. It can always be improved later, and you can use a symlink if it really is a bother.
There was a problem hiding this comment.
Hmm, actually looks like I didn't, having just looked. But the intent was there :D Proper tokenizing is not trival.
src/wallet/wallet2.cpp
Outdated
src/cryptonote_core/blockchain.cpp
Outdated
There was a problem hiding this comment.
auto notify = m_block_notify;
if (notify)
{
notify->notify(epee::string_tools::pod_to_hex(id).c_str());
}
There was a problem hiding this comment.
Any reason why? shared_ptr has an operator bool which checks if a pointer is set.
There was a problem hiding this comment.
Thread safety.
PS: edited
There was a problem hiding this comment.
How does get have any more thread safety than operator bool? Both are thread safe.
There was a problem hiding this comment.
Please, check the code snippet once again, i updated it as i noted in the previous reply.
There was a problem hiding this comment.
Other functions are making use of CRITICAL_REGION_LOCAL (locking).
There was a problem hiding this comment.
Haha, don't be so full of butthurt. Shit happens.
Sure, i will point to any problem i will identify in the code.
That is what is called code review, FYI.
There was a problem hiding this comment.
Not butthurt at all. Genuinely pleased things get reviewed and debated before they get merged - it's critically important. That's why I submit code, participate in code reviews of others and support the community in all ways I can.
In the case of this class, it turns out it uses those CRITICAL_... macros for locking to get thread safety. For consistency, this variable should ideally follow suit I guess.
There was a problem hiding this comment.
If we are goin' to keep shared_ptr and add a mutex, there won't be any point to use shared_ptr exactly in this case then. Will have to replace shared_ptr with unique_ptr and use a mutex to maintain unique_ptr's thread-safety.
And that's obviously would be a self-made shared_ptr parody. That's what shared_ptr actually do.
There was a problem hiding this comment.
I wonder if the copy operator is atomic...
moneromooo-monero
force-pushed
the
notify
branch
from
da43b03 to
da34601
Compare
src/common/exec.cpp
Outdated
There was a problem hiding this comment.
What happens if ctrl-c is pressed in this loop? If SIGTERM is caught, is this potentially stuck until a -9 kill? I don't think there is a generic shutdown flag to check sadly, so I'm not sure what to do about this.
There was a problem hiding this comment.
On further thought - the wait variable is never set to true. Should that option and code supporting it be dropped, until there is a need for it?
There was a problem hiding this comment.
If it happens during the syscall, you should get EINTR back. If not, it might loop indeed. wait is never used currently, it just seemed like a useful thing to have. It can be removed if it's the general consensus.
moneromooo-monero
force-pushed
the
notify
branch
from
da34601 to
c06f07a
Compare
Those take a command line of the form "A [B]", with A being the name (and optional path, if not in the caller's CWD, but fully qualified path is recommended, avoids possible security issues) to a program, and optional arguments. Any occurence of the two character string "%s" will be replaced by the hash of the block or transaction which triggered the notification. Tokenization is barebones. If you want things like pipes, calls to paths with spaces, etc, then use a script (though exec time will suffer). block-notify is called when a new block is added onto the chain. tx-notify is called when a new transaction happens with the wallet as source and/or destination. It is the notification program's responsibility to determine what to do in those cases. Note that this is asynchronous, so it is very possible that: - the notification programs will be run out of order - several events happen before the notification for the first one A Windows port would be nice if someone wants to make one.
moneromooo-monero
force-pushed
the
notify
branch
from
c06f07a to
7340300
Compare
| get_difficulty_for_next_block(); // just to cache it | ||
| invalidate_block_template_cache(); | ||
|
|
||
| std::shared_ptr<tools::Notify> block_notify = m_block_notify; |
There was a problem hiding this comment.
@moneromooo-monero @jtgrassie
Regarding #4333 (comment)
Had to revisit the standard once again. Unfortunately, yes, std::shared_ptr copy constructor is not atomic. There are std::atomic_load and std::atomic_store overloads for this case in c++11.
But they are going to be deprecated in c++20 and the semantic will be changed to std::atomic<std::shared_ptr<T>>. http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2017/p0718r2.html#3.3
I bet we don't want to deal with different compilation issues tied to these changes.
We have no choice but to introduce a kind of manual locking.
There was a problem hiding this comment.
This is set once at program start though. In practice it can't race.
7340300 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
7340300 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
|
How to use this new config parameter? But nothing happened. echo "hi" >> /root/lyl |
|
@DinoStray The parameter cannot contain spaces right now (see above comment). If just using a bash script, you don't need the |
|
@jtgrassie |
|
I have alot of defunct threads with notify (one for each block): monero.conf: /usr/local/bin/block-monero.sh: Notification script working as well, my api is receiving notifications, but i have no ideas how to fix those annoying defuncts. I also tried to add --block-notify='/usr/local/bin/block-monero.sh %s', and --block-notify='/bin/bash /usr/local/bin/block-monero.sh' with same results. |
|
@moneromooo-monero |
|
I have an application that gets notified about new transactions using --tx_notify. |
|
@KraxCZ I can confirm this. Please open an issue. |
11 participants
Those take a command line of the form "A [B]", with A being the
name (and optional path, if not in the caller's CWD, but fully
qualified path is recommended, avoids possible security issues)
to a program, and optional arguments. Any occurence of the two
character string "%s" will be replaced by the hash of the block
or transaction which triggered the notification.
block-notify is called when a new block is added onto the chain.
tx-notify is called when a new transaction happens with the
wallet as source and/or destination.
It is the notification program's responsibility to determine what
to do in those cases.
Note that this is asynchronous, so it is very possible that:
A Windows port would be nice if someone wants to make one.