Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
zmq: mempool notifications #7753
I was planning to submit a pull request to change this kind of notifications. The current solution is not very elegant and doesn't scale. The command line arguments should be clean.
So if you don't mind I'll try to explain here the idea using this example:
Here the notify argument creates one notification pipeline composed by one source of events (mempool), one filter element to transform the event in json and one sink of events (zmqpub), all connected with the operator
Is this design, zmq elements are concrete types of sinks. But we could have more, like log to file or pub to nsq queue.
We could have chain and wallet sources for instance. We could have filters to transform to other types.
@promag I'm all for making it more flexible, but what you're proposing sounds like the gstreamer syntax and that's IMO even harder to use than the current arguments. I've needed to do a few media pipelines in my PhD research but never managed to really master it.
Our basic idea with the ZMQ notifications is that one notification system exists. From there on everything (e.g. forward to the notification system of your choice) can be handled by external software. I don't want too much notification-related complexity in core.
Aside, this is the wrong place to discuss this. Let's focus on the software change here. Please create a new issue.
Some ideas / conceptual questions:
What if you use the following sequence:
You may have some duplicates in the first run, e.g., adds which already show up in the mempool requested, removes which were already removed, but I think due to the idempotent nature of the events (just add and remove) the overall state will be the same?
Missing notifications are a possibiltiy with zmq. But I'd say this is a matter of sizing buffers appropriately, and dedicating a thread to listening to zmq requests.
To reliably detect missing notifications we could add sequence numbers to the events, as I think zmq guarantees that events from a single source are delivered in order.
Then again if we do this we probably want it for all the event types... I actually suggested it for the first zmq proposal.
I thought about sending a generic stats packet with every mempool event, but as you can receive tons of them I don't want to make the individual events too big and include things you can compute yourself. But no problems with just the size / # transactions.
Absolutely, I'll open source it soon, will be part of a bc-monitor repo. Eventually I hope to make a tool like tor-arm (just renamed to nyx IIRC) which allows watching all aspects of a node.
Ok added two commits:
Add notification signals to make it possible to subscribe to mempool changes: - NotifyEntryAdded(const CTxMemPoolEntry &)> - NotifyEntryRemoved(const CTxMemPoolEntry &, MemPoolRemovalReason)> Also add a mempool removal reason enumeration, which is passed to the removed notification based on why the transaction was removed from the mempool.
Replace factories list with calls to a function. This simplifies the code (every notifier is only created once anyway), and makes it possible to pass arguments to notifier contructors.
Add notifications when transactions enter or leave the mempool. These can be enabled with `pubmempool`: - `mempooladded`: a transaction was added to the mempool - `mempoolremoved`: a transaction was removed from the mempool This allows third-party software to keep track of what is in the mempool in real time.