Add ZeroMQ notifications #6103

Merged
merged 4 commits into from Sep 16, 2015

Conversation

Projects
None yet
@promag
Member

promag commented May 4, 2015

Continues Johnathan Corgan and Jeff Garzik's work. Supercedes #4594 and #5303.
As discussed in #6072 and #5328.

src/zmqports.cpp
+
+static void zmqError(const char *str)
+{
+ LogPrint("ZMQ error: %s, errno=%s\n", str, zmq_strerror(errno));

This comment has been minimized.

@ajweiss

ajweiss May 4, 2015

Contributor

This entire file needs to be retabbed.

@ajweiss

ajweiss May 4, 2015

Contributor

This entire file needs to be retabbed.

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

L39: s/LogPrint/LogPrintf otherwise you start introducing a debug category with string identifier "ZMQ error: %s, errno=%s\n".

@jonasschnelli

jonasschnelli May 5, 2015

Member

L39: s/LogPrint/LogPrintf otherwise you start introducing a debug category with string identifier "ZMQ error: %s, errno=%s\n".

This comment has been minimized.

@promag

promag May 5, 2015

Member

Fixed in upcoming commit

@promag

promag May 5, 2015

Member

Fixed in upcoming commit

@ajweiss

This comment has been minimized.

Show comment
Hide comment
@ajweiss

ajweiss May 4, 2015

Contributor

Are you planning on supplying an RPC/regression test for this?

Contributor

ajweiss commented May 4, 2015

Are you planning on supplying an RPC/regression test for this?

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 4, 2015

Member

That work was in previous PR. I can't see a use case where this is useful. Maybe @jmcorgan have something to say about that, otherwise I would remove this RPC call.

Member

promag commented May 4, 2015

That work was in previous PR. I can't see a use case where this is useful. Maybe @jmcorgan have something to say about that, otherwise I would remove this RPC call.

src/ui_interface.h
@@ -95,6 +97,12 @@ class CClientUIInterface
/** New block has been accepted */
boost::signals2::signal<void (const uint256& hash)> NotifyBlockTip;
+
+ /** New block has been accepted */
+ boost::signals2::signal<void (const CBlock& block)> NotifyAcceptBlock;

This comment has been minimized.

@sipa

sipa May 4, 2015

Member

As mentioned in one of the previous PRs, block acceptance is really not interesting or useful. It does not mean the block is part of the best chain, or even that it is valid. Use the NotifyBlockTip signal place, which indicates a new tip of the best chain was accepted.

@sipa

sipa May 4, 2015

Member

As mentioned in one of the previous PRs, block acceptance is really not interesting or useful. It does not mean the block is part of the best chain, or even that it is valid. Use the NotifyBlockTip signal place, which indicates a new tip of the best chain was accepted.

This comment has been minimized.

@promag

promag May 5, 2015

Member

Fixed in upcoming commit

@promag

promag May 5, 2015

Member

Fixed in upcoming commit

@ajweiss

This comment has been minimized.

Show comment
Hide comment
@ajweiss

ajweiss May 4, 2015

Contributor

As far as I know, no RPC tests for ZMQ support have yet been PR'd.

I think there might be some confusion here. When I say RPC tests, I mean the Python regression tests that live qa/rpc-tests tree. They're pretty much only called RPC tests because of history and the primary way they interface with bitcoind. They exercise and verify many large bits of functionality and are always useful! Case in point, when REST support was originally added, it had some pretty serious bugs that weren't caught because there was no RPC test. See PR #5379 for an example of how RPC tests for the REST interface were implemented.

I suppose RPC tests could be PR'd separately, but I bet this will go in faster and with more enthusiasm if tests are added.

Contributor

ajweiss commented May 4, 2015

As far as I know, no RPC tests for ZMQ support have yet been PR'd.

I think there might be some confusion here. When I say RPC tests, I mean the Python regression tests that live qa/rpc-tests tree. They're pretty much only called RPC tests because of history and the primary way they interface with bitcoind. They exercise and verify many large bits of functionality and are always useful! Case in point, when REST support was originally added, it had some pretty serious bugs that weren't caught because there was no RPC test. See PR #5379 for an example of how RPC tests for the REST interface were implemented.

I suppose RPC tests could be PR'd separately, but I bet this will go in faster and with more enthusiasm if tests are added.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 4, 2015

Member

@promag Nice that you have started recycle/update/rebase this!
I have plans to test this during this week and therefore would be willing to write some RPC'like tests.

Member

jonasschnelli commented May 4, 2015

@promag Nice that you have started recycle/update/rebase this!
I have plans to test this during this week and therefore would be willing to write some RPC'like tests.

doc/zmq.md
+Currently, the ZeroMQ facility only needs to have the ZeroMQ endpoint
+specified:
+
+ $ bitcoind -zmqpub=tcp://127.0.0.1/28332

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

this should be: bitcoind -zmqpub=tcp://127.0.0.1:28332 (dash colon instead slash before port)
Edit: dash / colon

@jonasschnelli

jonasschnelli May 5, 2015

Member

this should be: bitcoind -zmqpub=tcp://127.0.0.1:28332 (dash colon instead slash before port)
Edit: dash / colon

This comment has been minimized.

@promag

promag May 5, 2015

Member

fixed in upcoming commit

@promag

promag May 5, 2015

Member

fixed in upcoming commit

src/zmqports.cpp
+ }
+
+ uiInterface.NotifyAcceptBlock.connect(ZMQPublishBlock);
+ uiInterface.NotifyRelayTx.connect(ZMQPublishTransaction);

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

Does it make sense to use the uiInterface CClientUIInterface as signaling layer for zmq? Or would it be better to introduce a custom signaling class/struct?

@jonasschnelli

jonasschnelli May 5, 2015

Member

Does it make sense to use the uiInterface CClientUIInterface as signaling layer for zmq? Or would it be better to introduce a custom signaling class/struct?

This comment has been minimized.

@promag

promag May 5, 2015

Member

I believe this discussion should happen in a different PR.

@promag

promag May 5, 2015

Member

I believe this discussion should happen in a different PR.

src/rpcserver.cpp
@@ -282,6 +282,9 @@ static const CRPCCommand vRPCCommands[] =
{ "network", "getpeerinfo", &getpeerinfo, true },
{ "network", "ping", &ping, true },
+ /* ZMQ notification */
+ { "zmq", "getzmqurl", &getzmqurl, true },

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

What's the use case of getzmqurl? Why should someone need to get the ZMQ URL over RPC?

@jonasschnelli

jonasschnelli May 5, 2015

Member

What's the use case of getzmqurl? Why should someone need to get the ZMQ URL over RPC?

This comment has been minimized.

@promag

promag May 5, 2015

Member

Removed in upcoming commit

@promag

promag May 5, 2015

Member

Removed in upcoming commit

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

I didn't say it has to be removed. I just don't see a usecase for this. Maybe @jgarzik can roll this up?

@jonasschnelli

jonasschnelli May 5, 2015

Member

I didn't say it has to be removed. I just don't see a usecase for this. Maybe @jgarzik can roll this up?

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

Or maybe something for @jmcorgan

@jonasschnelli

jonasschnelli May 5, 2015

Member

Or maybe something for @jmcorgan

src/zmqports.cpp
+
+ // Serialize block
+ CDataStream ss(SER_NETWORK, PROTOCOL_VERSION);
+ ss.reserve(1000000); // FIXME use defined constant

This comment has been minimized.

@promag

promag May 5, 2015

Member

I've changed this to use zeromq multipart messages. Changed in upcoming commit

@promag

promag May 5, 2015

Member

I've changed this to use zeromq multipart messages. Changed in upcoming commit

src/zmqports.cpp
+ LOCK(cs_main);
+
+ CBlockIndex* pblockindex = mapBlockIndex[hash];
+ if(!ReadBlockFromDisk(blk, pblockindex))

This comment has been minimized.

@promag

promag May 5, 2015

Member

I believe this is the right way of getting a block.

@promag

promag May 5, 2015

Member

I believe this is the right way of getting a block.

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

I'm planning to add the option -zmqformat=raw|json. This way a subscriber doesn't need to rely on 3rd libs to process the messages. What do you think?
Pending on #6108

Member

promag commented May 5, 2015

I'm planning to add the option -zmqformat=raw|json. This way a subscriber doesn't need to rely on 3rd libs to process the messages. What do you think?
Pending on #6108

src/zmqports.h
+// Global state
+extern bool fZMQPub;
+
+#if ENABLE_ZMQ

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

Is the reason why the #ifdefs are here instead of wrapping the code-part in init.cpp to lower the #ifdef in init.cpp? Maybe also something for @jgarzik to answer.

@jonasschnelli

jonasschnelli May 5, 2015

Member

Is the reason why the #ifdefs are here instead of wrapping the code-part in init.cpp to lower the #ifdef in init.cpp? Maybe also something for @jgarzik to answer.

This comment has been minimized.

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

@promag: Thanks. Yes this makes sense.

@jonasschnelli

jonasschnelli May 5, 2015

Member

@promag: Thanks. Yes this makes sense.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 5, 2015

Member

Would be nice if you could cherry-pick / pull-in this RPC/ZMQ test case:
jonasschnelli@91334e7

Member

jonasschnelli commented May 5, 2015

Would be nice if you could cherry-pick / pull-in this RPC/ZMQ test case:
jonasschnelli@91334e7

src/init.cpp
@@ -1022,6 +1029,11 @@ bool AppInit2(boost::thread_group& threadGroup)
BOOST_FOREACH(string strDest, mapMultiArgs["-seednode"])
AddOneShot(strDest);
+ if (mapArgs.count("-zmqpub"))
+ {
+ ZMQInitialize(mapArgs["zmqpub"]);

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

typo: should be ZMQInitialize(mapArgs["-zmqpub"]); (mind the dash)

@jonasschnelli

jonasschnelli May 5, 2015

Member

typo: should be ZMQInitialize(mapArgs["-zmqpub"]); (mind the dash)

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 5, 2015

Member

@promag: I would not add JSON support and keep ZMQ as clean and low-level as possible. Keeping it bin only can make things faster and with JSON there is always a risks of things get handled different during enc-/decoding of JSON streams depending on the library you use.

Member

jonasschnelli commented May 5, 2015

@promag: I would not add JSON support and keep ZMQ as clean and low-level as possible. Keeping it bin only can make things faster and with JSON there is always a risks of things get handled different during enc-/decoding of JSON streams depending on the library you use.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 5, 2015

Member

Tested reviewed leak-checked ACK.
nits: usage of uiInterface as "signal layer"

Member

jonasschnelli commented May 5, 2015

Tested reviewed leak-checked ACK.
nits: usage of uiInterface as "signal layer"

src/zmqports.cpp
+ CBlock blk;
+ {
+ LOCK(cs_main);
+

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

nit: L128 has whitespace.

@jonasschnelli

jonasschnelli May 5, 2015

Member

nit: L128 has whitespace.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 5, 2015

Member

nit:
signaling disconnect is missing

diff --git a/src/zmqports.cpp b/src/zmqports.cpp
index d083292..bd1229c 100644
--- a/src/zmqports.cpp
+++ b/src/zmqports.cpp
@@ -155,6 +155,9 @@ void ZMQShutdown()

         zmq_ctx_destroy(zmqContext);
         zmqContext = 0;
+
+        uiInterface.NotifyBlockTip.disconnect(ZMQPublishBlock);
+        uiInterface.NotifyRelayTx.disconnect(ZMQPublishTransaction);
     }

     fZMQPub = false;
Member

jonasschnelli commented May 5, 2015

nit:
signaling disconnect is missing

diff --git a/src/zmqports.cpp b/src/zmqports.cpp
index d083292..bd1229c 100644
--- a/src/zmqports.cpp
+++ b/src/zmqports.cpp
@@ -155,6 +155,9 @@ void ZMQShutdown()

         zmq_ctx_destroy(zmqContext);
         zmqContext = 0;
+
+        uiInterface.NotifyBlockTip.disconnect(ZMQPublishBlock);
+        uiInterface.NotifyRelayTx.disconnect(ZMQPublishTransaction);
     }

     fZMQPub = false;
@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

@jonasschnelli Regarding the output formats, I don't think having JSON is that bad, it is just an option.
Actually I think we could support 3 formats for now:

  • network: currently supported
  • json: needs BlockToUni to convert to JSON (similar to RPC getblock)
  • hash: just send the block hash or txid (this allows a subscriber to queue the notification to process it later)
Member

promag commented May 5, 2015

@jonasschnelli Regarding the output formats, I don't think having JSON is that bad, it is just an option.
Actually I think we could support 3 formats for now:

  • network: currently supported
  • json: needs BlockToUni to convert to JSON (similar to RPC getblock)
  • hash: just send the block hash or txid (this allows a subscriber to queue the notification to process it later)
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 5, 2015

Member

@promag I agree with 1 (p2p) and 3 (hash). IMO the ZMQ interface is a push/notify interface (for now). If someone like to get JSON output he could do a RPC request with received data. Adding JSON format over ZMQ level would just encourage user "to go the wrong way".
Adding a way of only receiving hashes over ZMQ would be good.
But i'm ready to hear other opinions.

Member

jonasschnelli commented May 5, 2015

@promag I agree with 1 (p2p) and 3 (hash). IMO the ZMQ interface is a push/notify interface (for now). If someone like to get JSON output he could do a RPC request with received data. Adding JSON format over ZMQ level would just encourage user "to go the wrong way".
Adding a way of only receiving hashes over ZMQ would be good.
But i'm ready to hear other opinions.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa May 5, 2015

Member

Specifically for blocks, I'm not sure that pushing full blocks is even useful, unless we make it push all new blocks (including intermediary reorged ones). The interesting information is "there is a new block chain tip, and this is it".

Member

sipa commented May 5, 2015

Specifically for blocks, I'm not sure that pushing full blocks is even useful, unless we make it push all new blocks (including intermediary reorged ones). The interesting information is "there is a new block chain tip, and this is it".

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

@sipa consider the following use case: a subscriber wants to know if there are transactions (in the new block chain tip) with outputs targeting a set of addresses.

Member

promag commented May 5, 2015

@sipa consider the following use case: a subscriber wants to know if there are transactions (in the new block chain tip) with outputs targeting a set of addresses.

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

I'm not sure if publishing a block of 20MB is problematic.

Member

promag commented May 5, 2015

I'm not sure if publishing a block of 20MB is problematic.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa May 5, 2015

Member
Member

sipa commented May 5, 2015

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

@sipa right, so publishing the new tip hash and having the consumer walk the block chain would be more appropriate/secure.

Member

promag commented May 5, 2015

@sipa right, so publishing the new tip hash and having the consumer walk the block chain would be more appropriate/secure.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 5, 2015

Member

For now i see the ZMQ extension as a notification-only one-way interface. I guess it's more optimized if a user gets informed about a new block hash (or new mempool tx hash) and then he decides if he likes to load the block with or without tx details over REST or RPC.

Member

jonasschnelli commented May 5, 2015

For now i see the ZMQ extension as a notification-only one-way interface. I guess it's more optimized if a user gets informed about a new block hash (or new mempool tx hash) and then he decides if he likes to load the block with or without tx details over REST or RPC.

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

Agree on network binary format and hash format? Other publish strategies can be discussed later after this PR.

Member

promag commented May 5, 2015

Agree on network binary format and hash format? Other publish strategies can be discussed later after this PR.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 5, 2015

Member

@promag: i think you need to get rid of commit 4542855

Member

jonasschnelli commented May 5, 2015

@promag: i think you need to get rid of commit 4542855

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

@jonasschnelli rebased
missing updated doc and test to reflect multipart message

Member

promag commented May 5, 2015

@jonasschnelli rebased
missing updated doc and test to reflect multipart message

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

Should we use CValidationInterface instead of CClientUIInterface?

Member

promag commented May 5, 2015

Should we use CValidationInterface instead of CClientUIInterface?

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa May 5, 2015

Member
Member

sipa commented May 5, 2015

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

Pending questions:

  1. support network binary format and hash format?
  2. class name CZMQPublisher : public CValidationInterface?
  3. use CValidationInterface::SetBestChain instead of CClientUIInterface::NotifyBlockTip?
Member

promag commented May 5, 2015

Pending questions:

  1. support network binary format and hash format?
  2. class name CZMQPublisher : public CValidationInterface?
  3. use CValidationInterface::SetBestChain instead of CClientUIInterface::NotifyBlockTip?
@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa May 5, 2015

Member
  1. Hash is enough for me. Binary makes sense if we make sure the full set of newly accepted blocks is announced.
  2. Sounds good.
  3. SetBestChain won't work, as it is only called after the best chain on disk is changed (which is infrequently). I'll write a patch to add the signals you need to CValidationInterface.
Member

sipa commented May 5, 2015

  1. Hash is enough for me. Binary makes sense if we make sure the full set of newly accepted blocks is announced.
  2. Sounds good.
  3. SetBestChain won't work, as it is only called after the best chain on disk is changed (which is infrequently). I'll write a patch to add the signals you need to CValidationInterface.
@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag May 5, 2015

Member

@sipa At the moment CValidationInterface is very related to CWallet, but it shouldn't right?

Member

promag commented May 5, 2015

@sipa At the moment CValidationInterface is very related to CWallet, but it shouldn't right?

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa May 5, 2015

Member

@promag It used to be called CWalletInterface. We renamed it because it shouldn't be used exclusively for wallets.

Member

sipa commented May 5, 2015

@promag It used to be called CWalletInterface. We renamed it because it shouldn't be used exclusively for wallets.

@laanwj laanwj added the Feature label May 5, 2015

src/zmqports.h
+
+protected: // CValidationInterface
+ void SyncTransaction(const CTransaction &tx, const CBlock *pblock);
+ void UpdatedBlockTip(const CBlock &block);

This comment has been minimized.

@promag

promag May 5, 2015

Member

waiting @sipa patch, for now assuming UpdatedBlockTip exists in CValidationInterface.

@promag

promag May 5, 2015

Member

waiting @sipa patch, for now assuming UpdatedBlockTip exists in CValidationInterface.

src/zmqports.h
+ enum Format
+ {
+ HashFormat,
+ NetworkFormat

This comment has been minimized.

@promag

promag May 5, 2015

Member

Not sure if this is OK or NETWORK_FORMAT. Same for HashFormat.

@promag

promag May 5, 2015

Member

Not sure if this is OK or NETWORK_FORMAT. Same for HashFormat.

src/zmqports.cpp
+ // Send multipart message
+ int rc = Publish("BLK", 3, ZMQ_SNDMORE);
+ if (rc==0)
+ Publish(hash.begin(), hash.size(), 0);

This comment has been minimized.

@promag

promag May 5, 2015

Member

@jonasschnelli is this OK?

This comment has been minimized.

@jonasschnelli

jonasschnelli May 5, 2015

Member

LGTM.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 10, 2015

Member
Member

sipa commented Sep 10, 2015

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik Sep 10, 2015

Contributor

In general, agree w/ @sipa, just the hash on the first iteration.

Longer term, it is forseeable that some folks looking for low latency may prefer full blocks versus added round-trips. Such solutions would have a fast path - zero round trip - and a slower path fallback in case the full block was insufficient, and further block-locator style queries are needed.

Contributor

jgarzik commented Sep 10, 2015

In general, agree w/ @sipa, just the hash on the first iteration.

Longer term, it is forseeable that some folks looking for low latency may prefer full blocks versus added round-trips. Such solutions would have a fast path - zero round trip - and a slower path fallback in case the full block was insufficient, and further block-locator style queries are needed.

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 10, 2015

Member

So it's the same: a client gets the full block, check if the parent is there, if not retrieve and repeat.

Member

promag commented Sep 10, 2015

So it's the same: a client gets the full block, check if the parent is there, if not retrieve and repeat.

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 10, 2015

Member

@jgarzik also agree. I just don't agree that having just the hash option leads the subscriber to take reorgs into account.
Edit: with just the hash, the subscriber could as well retrieve the tip and miss some blocks without recognising it.

Member

promag commented Sep 10, 2015

@jgarzik also agree. I just don't agree that having just the hash option leads the subscriber to take reorgs into account.
Edit: with just the hash, the subscriber could as well retrieve the tip and miss some blocks without recognising it.

@jtimon

This comment has been minimized.

Show comment
Hide comment
@jtimon

jtimon Sep 10, 2015

Member
Member

jtimon commented Sep 10, 2015

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 10, 2015

Member

@promag If implemented correctly, there is no difference. You can present it as an optimization: "This informs you about the new chain tip, and as courtesy, we already give you the last block immediately so you don't need to fetch it". But I think the current interface gives the impression that it will relay all validated blocks, which is wrong and leads to incorrect understanding. The documentation even says so literally.

Member

sipa commented Sep 10, 2015

@promag If implemented correctly, there is no difference. You can present it as an optimization: "This informs you about the new chain tip, and as courtesy, we already give you the last block immediately so you don't need to fetch it". But I think the current interface gives the impression that it will relay all validated blocks, which is wrong and leads to incorrect understanding. The documentation even says so literally.

+connection of a corresponding ZeroMQ subscriber port in receiving
+software; it is not authenticated nor is there any two-way protocol
+involvement. Therefore, subscribers should validate the received data
+since it may be out of date, incomplete or even invalid.

This comment has been minimized.

@sipa

sipa Sep 10, 2015

Member

The subscriber has no way to verify that the published block is invalid - it would need a full node for that, and that's exactly what the ZMQ interface is for.

@sipa

sipa Sep 10, 2015

Member

The subscriber has no way to verify that the published block is invalid - it would need a full node for that, and that's exactly what the ZMQ interface is for.

This comment has been minimized.

@promag

promag Sep 10, 2015

Member

Yeah, reviewing and updating the doc.

@promag

promag Sep 10, 2015

Member

Yeah, reviewing and updating the doc.

This comment has been minimized.

@sipa

sipa Sep 10, 2015

Member

Thanks!

@sipa

sipa Sep 10, 2015

Member

Thanks!

@jtimon

This comment has been minimized.

Show comment
Hide comment
@jtimon

jtimon Sep 10, 2015

Member

Then only the documentation needs to be fixed?
On Sep 10, 2015 6:02 PM, "Pieter Wuille" notifications@github.com wrote:

@promag https://github.com/promag If implemented correctly, there is no
difference. You can present it as an optimization: "This informs you about
the new chain tip, and as courtesy, we already give you the last block
immediately so you don't need to fetch it". But I think the current
interface gives the impression that it will relay all validated blocks,
which is wrong and leads to incorrect understanding. The documentation even
says so literally.


Reply to this email directly or view it on GitHub
#6103 (comment).

Member

jtimon commented Sep 10, 2015

Then only the documentation needs to be fixed?
On Sep 10, 2015 6:02 PM, "Pieter Wuille" notifications@github.com wrote:

@promag https://github.com/promag If implemented correctly, there is no
difference. You can present it as an optimization: "This informs you about
the new chain tip, and as courtesy, we already give you the last block
immediately so you don't need to fetch it". But I think the current
interface gives the impression that it will relay all validated blocks,
which is wrong and leads to incorrect understanding. The documentation even
says so literally.


Reply to this email directly or view it on GitHub
#6103 (comment).

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 10, 2015

Member

I think many people don't grasp the difference between relaying blocks and chain tip updates, and the fact that this pull request was written with documentation that confuses the two is an indication of that. Don't make the simplest way of using it the wrong one.

Member

sipa commented Sep 10, 2015

I think many people don't grasp the difference between relaying blocks and chain tip updates, and the fact that this pull request was written with documentation that confuses the two is an indication of that. Don't make the simplest way of using it the wrong one.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 10, 2015

Member

Anyway, it seems my opinion is controversial here. I'm fine with merging if the documentation is fixed, but I would really prefer to not have a mechanism that makes it this easy to misunderstand.

Member

sipa commented Sep 10, 2015

Anyway, it seems my opinion is controversial here. I'm fine with merging if the documentation is fixed, but I would really prefer to not have a mechanism that makes it this easy to misunderstand.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 10, 2015

Member

Also, sorry for hammering on this one detail. I'm very happy with the pull request overall and would like to see it merged soon.

Member

sipa commented Sep 10, 2015

Also, sorry for hammering on this one detail. I'm very happy with the pull request overall and would like to see it merged soon.

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 10, 2015

Member

My point is that even -blocknotify leads to that problem, and it should be noted elsewhere that a reorg may not call the script for some blocks.

Member

promag commented Sep 10, 2015

My point is that even -blocknotify leads to that problem, and it should be noted elsewhere that a reorg may not call the script for some blocks.

@jtimon

This comment has been minimized.

Show comment
Hide comment
@jtimon

jtimon Sep 10, 2015

Member

Anyway, it seems my opinion is controversial here. I'm fine with merging
if the documentation is fixed, but I would really prefer a mechanism that
doesn't make it this easy to misunderstand.

Assuming users read the updated docs, I don't see how offering noth
only-hash or full blocks is more confusing than only offering hash-only.

Member

jtimon commented Sep 10, 2015

Anyway, it seems my opinion is controversial here. I'm fine with merging
if the documentation is fixed, but I would really prefer a mechanism that
doesn't make it this easy to misunderstand.

Assuming users read the updated docs, I don't see how offering noth
only-hash or full blocks is more confusing than only offering hash-only.

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 10, 2015

Member

@jtimon Assuming users read the updated docs...

@promag Agree, it could be clearer there too.

Member

sipa commented Sep 10, 2015

@jtimon Assuming users read the updated docs...

@promag Agree, it could be clearer there too.

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 11, 2015

Member

@jonasschnelli @jgarzik @jtimon please review the documentation change.

Member

promag commented Sep 11, 2015

@jonasschnelli @jgarzik @jtimon please review the documentation change.

@jonasschnelli

View changes

src/zmq/zmqpublishnotifier.cpp
@@ -0,0 +1,176 @@
+// Copyright (c) 2009-2010 Satoshi Nakamoto
+// Copyright (c) 2009-2014 The Bitcoin Core developers

This comment has been minimized.

@jonasschnelli

jonasschnelli Sep 11, 2015

Member

nit: remove L1 AND s/2009-2014/2015?

@jonasschnelli

jonasschnelli Sep 11, 2015

Member

nit: remove L1 AND s/2009-2014/2015?

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 11, 2015

Member

Just git a build error on debian:

./configure --enable-debug --with-gui=no
[...]
checking whether to build ZMQ support... yes
checking for ZMQ... yes
[...]
make -j5
[...]
  CXX      wallet/libbitcoin_wallet_a-walletdb.o
zmq/zmqpublishnotifier.cpp: In function ‘int zmq_send_multipart(void*, const void*, size_t, ...)’:
zmq/zmqpublishnotifier.cpp:36:61: error: ‘zmq_msg_send’ was not declared in this scope
         rc = zmq_msg_send(&msg, sock, data ? ZMQ_SNDMORE : 0);
[...]
zmq/zmqnotificationinterface.cpp:118:33: error: ‘zmq_ctx_destroy’ was not declared in this scope
         zmq_ctx_destroy(pcontext);                                                          ^

The source of the problem is probably an outdated libzmq. I have installed libzmq1(-dev):
i A libzmq1 - lightweight messaging kernel (shared library)

During configure we might ask for zmq_msg_send to detect if the library is available (or similar)?

Member

jonasschnelli commented Sep 11, 2015

Just git a build error on debian:

./configure --enable-debug --with-gui=no
[...]
checking whether to build ZMQ support... yes
checking for ZMQ... yes
[...]
make -j5
[...]
  CXX      wallet/libbitcoin_wallet_a-walletdb.o
zmq/zmqpublishnotifier.cpp: In function ‘int zmq_send_multipart(void*, const void*, size_t, ...)’:
zmq/zmqpublishnotifier.cpp:36:61: error: ‘zmq_msg_send’ was not declared in this scope
         rc = zmq_msg_send(&msg, sock, data ? ZMQ_SNDMORE : 0);
[...]
zmq/zmqnotificationinterface.cpp:118:33: error: ‘zmq_ctx_destroy’ was not declared in this scope
         zmq_ctx_destroy(pcontext);                                                          ^

The source of the problem is probably an outdated libzmq. I have installed libzmq1(-dev):
i A libzmq1 - lightweight messaging kernel (shared library)

During configure we might ask for zmq_msg_send to detect if the library is available (or similar)?

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 11, 2015

Member

Using libzmq3 solved the problem. But IIRC, Debian 7 (wheezy) does not support libzmq3 out of the standard repository. A tiny check during ./configure would be nice.

Member

jonasschnelli commented Sep 11, 2015

Using libzmq3 solved the problem. But IIRC, Debian 7 (wheezy) does not support libzmq3 out of the standard repository. A tiny check during ./configure would be nice.

@jonasschnelli

View changes

src/zmq/zmqnotificationinterface.cpp
+#include "main.h"
+#include "streams.h"
+#include "util.h"
+#include "netbase.h"

This comment has been minimized.

@jonasschnelli

jonasschnelli Sep 11, 2015

Member

nit: i think this include can be removed. Compiles fine on OSX/Debian without this include.

@jonasschnelli

jonasschnelli Sep 11, 2015

Member

nit: i think this include can be removed. Compiles fine on OSX/Debian without this include.

@jonasschnelli

View changes

src/zmq/zmqpublishnotifier.cpp
+
+#include "zmqpublishnotifier.h"
+#include "main.h"
+#include "net.h"

This comment has been minimized.

@jonasschnelli

jonasschnelli Sep 11, 2015

Member

nit: I think this include can be removed. Compiles fine on OSX/Debian without this include.

@jonasschnelli

jonasschnelli Sep 11, 2015

Member

nit: I think this include can be removed. Compiles fine on OSX/Debian without this include.

This comment has been minimized.

@jtimon

jtimon Sep 11, 2015

Member

Are you sure this is not because net.h is included in main.h? The same goes for the other removed include [ie netbase.h is included in net.h iirc].

@jtimon

jtimon Sep 11, 2015

Member

Are you sure this is not because net.h is included in main.h? The same goes for the other removed include [ie netbase.h is included in net.h iirc].

This comment has been minimized.

@sipa

sipa Sep 11, 2015

Member
@sipa

sipa via email Sep 11, 2015

Member
@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 11, 2015

Member

Using libzmq3 solved the problem. But IIRC, Debian 7 (wheezy) does not support libzmq3 out of the standard repository. A tiny check during ./configure would be nice.

@theuni can you take a look?

Member

promag commented Sep 11, 2015

Using libzmq3 solved the problem. But IIRC, Debian 7 (wheezy) does not support libzmq3 out of the standard repository. A tiny check during ./configure would be nice.

@theuni can you take a look?

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik Sep 15, 2015

Contributor

In general, this appears merge-ready once

  • travis is happy
  • the @sipa lock comment is addressed.

We can iterate further once it's in-tree.

Contributor

jgarzik commented Sep 15, 2015

In general, this appears merge-ready once

  • travis is happy
  • the @sipa lock comment is addressed.

We can iterate further once it's in-tree.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 15, 2015

Member

Agree with @jgarzik. There are serval proposal for improvements (autodetect libzmq3, avoid locking by passing CBlockIndex*, minor cleanups), but shouldn't hold back the merge.
The only thing that might should be looked at is the travis issue: https://travis-ci.org/bitcoin/bitcoin/jobs/79866093. Maybe its not related to this PR. Maybe a rebase&force-push solves that issue.

Member

jonasschnelli commented Sep 15, 2015

Agree with @jgarzik. There are serval proposal for improvements (autodetect libzmq3, avoid locking by passing CBlockIndex*, minor cleanups), but shouldn't hold back the merge.
The only thing that might should be looked at is the travis issue: https://travis-ci.org/bitcoin/bitcoin/jobs/79866093. Maybe its not related to this PR. Maybe a rebase&force-push solves that issue.

theuni and others added some commits May 6, 2015

Add ZeroMQ support. Notify blocks and transactions via ZeroMQ
Continues Johnathan Corgan's work.
Publishing multipart messages

Bugfix: Add missing zmq header includes

Bugfix: Adjust build system to link ZeroMQ code for Qt binaries
@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 16, 2015

Member

@jonasschnelli rebase@force-push solved.
@laanwj @sipa I believe it's all in place to merge.

Member

promag commented Sep 16, 2015

@jonasschnelli rebase@force-push solved.
@laanwj @sipa I believe it's all in place to merge.

@jgarzik jgarzik merged commit 029e278 into bitcoin:master Sep 16, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

jgarzik pushed a commit that referenced this pull request Sep 16, 2015

Jeff Garzik

@promag promag deleted the uphold:zmq branch Sep 16, 2015

@WilliamDunne

This comment has been minimized.

Show comment
Hide comment
@WilliamDunne

WilliamDunne Sep 19, 2015

That's funny, I just had to create this almost exact system in BitcoinJ. Should be helpful.

That's funny, I just had to create this almost exact system in BitcoinJ. Should be helpful.

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Oct 15, 2015

@str4d str4d referenced this pull request in zcash/zcash Jan 31, 2017

Merged

Add ZeroMQ notifications #2050

zkbot added a commit to zcash/zcash that referenced this pull request Feb 9, 2017

zkbot added a commit to zcash/zcash that referenced this pull request Feb 9, 2017

zkbot added a commit to zcash/zcash that referenced this pull request Feb 9, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment