[RPC] Add RPC long poll notifications #7949

jonasschnelli · 2016-04-26T15:33:04Z

Reasons for another notification interface

Currently there is no interface that could be extended to "private" notification secured behind the authorization (like peers connected/disconnected or a new wallet relevant transaction notification)
HTTP long poll notifications are very easy to set up and require almost no dependencies
HTTP long poll notifications can easily pushed over the internet using httpd reverse proxy together with a popper authentication method (certs or http auth digest) together with TLS.
HTTP long poll would allow connecting applications to do all kinds of things with just a single communication channel (currently you need RPC & ZMQ for most use cases which would require VPN or a fancy multi port stunnel connection to broadcast the notification over the internet)

How does it work

The listener calls the pollnotification RPC command.
If no notifications are available, the RPC thread will idle for a given timeout (30s by default)
If a notification was fired during the 30 seconds, the longpoll call will be responded with the new notification(s)
The client/listener can immediately reconnect and wait again
If notifications are already in the queue, the pollnotification command will immediately response.
Notifications can't get lost (possible to lose them during http transfer and if one exceed the queue limit)

Downsides

JSON encoding overhead

New RPC calls

setregisterednotifications [<notificationtype>] (possible types are hashtx and hashblock)
getregisterednotifications
pollnotifications

Missing

More tests
Documentation

I'd like to use a such interface to work on a remote GUI (use case: GUI on your local desktop, node on a VPS).

laanwj · 2016-04-26T15:40:03Z

I like the concept of being able to listen for events through http, however I think this is severely limited by having server-side state, limiting the number of listeners to only one.

What I'd personally prefer is, instead of longpolling, to subscribe to a 'stream' of events (e.g. websocket or just chunked encoding), where the set of events to listen to is in the request. This avoids having to store any client-state in the server - at least for longer than the request lasts.

jonasschnelli · 2016-04-26T15:45:30Z

[... ] having server-side state, limiting the number of listeners to only one

Right. The current implementation limits to only one listener. Extending this PR so it would support a client chosen UUID would not be very complicated (a set of queues and a set of registered notification types). Clients could register notification types along with a client-chosen UUID.
I might extend this PR to support multiple listeners.

jonasschnelli · 2016-04-26T17:45:23Z

Added a commit that allows multiple clients at the same time.

The new RPC commands require now a clientUUID parameter (a per client unique string, ideally a UUID after RFC 4122). Bitcoind keeps a queue, sequence numbers and registered types per client.

There is currently not max client limit and no way to remove clients (though you can unregister all notification types but not empty the current queue).

jonasschnelli · 2016-05-06T09:53:11Z

Rebased.
Would be nice to get some concept NACKs/ACKs.

MarcoFalke · 2016-05-06T10:10:47Z

Sign in to view

@@ -0,0 +1,64 @@
+#!/usr/bin/env python2


Nit: python3

jonasschnelli · 2016-05-12T10:54:32Z

Rebased.

laanwj · 2017-01-11T13:09:26Z

Gah we need to take a look at this again after 0.14 is released.

jonasschnelli · 2017-01-11T13:27:05Z

Yes. Sure. I'll try to re-base and overhaul this soon.

TheBlueMatt · 2017-09-28T16:47:06Z

Plan on rebasing this, or should just be closed?

jonasschnelli · 2017-09-28T16:57:42Z

I'm currently rewriting this... will be ready soon.

jonasschnelli · 2017-10-20T06:02:06Z

Overhauled and rebased.

This is still server based (server keeps track of what the client has) queue max size is currently 1024^2 and does only contain hashes of blocks or transactions.
Each notification comes with a sequence number to detect lost transactions (which then should trigger a "full-client-sync").

TheBlueMatt

Concept ACK, though should definitely get more general concept review. I think this could use a more explicit register/deregister process eg registernewclient [ThingsClientCaresAbout] -> provides UUID (instead of registering taking a client-provided UUID), then an explicit deregister which removes queues for that client, instead of setting notifications to 0 and the queues for that client simply leaking.

TheBlueMatt · 2017-11-06T18:51:19Z

src/rpc/notifications.cpp

+    void UpdatedBlockTip(const CBlockIndex* pindexNew, const CBlockIndex* pindexFork, bool fInitialDownload) override
+    {
+        LOCK(m_cs_queue_manager);
+        BOOST_FOREACH (NotificationQueue* queue, m_map_sequence_numbers | boost::adaptors::map_values) {


Ugh, can we not use more BOOST_FOREACH garbage? Should be really easy to rewrite this without, no?

TheBlueMatt · 2017-11-06T18:51:19Z

src/rpc/notifications.cpp

+        }
+    }
+
+    void UpdatedBlockTip(const CBlockIndex* pindexNew, const CBlockIndex* pindexFork, bool fInitialDownload) override


I dont think this is the notification we want here - dont we want to use BlockConnected/Disconnected to notify clients of all connected blocks, not just new tips after reorgs which potentially connect multiple blocks?

TheBlueMatt · 2017-11-06T18:51:19Z

src/rpc/notifications.cpp

+{
+public:
+    std::deque<NotificationEntry> m_queue;
+    std::map<NotificationType, int32_t> m_map_sequence_numbers;


I generally prefer 64 bit ints here - sure, its unlikely you'd overflow 32 bits, but if you're online for 3 or 4 years you may start getting closeish.

TheBlueMatt · 2017-11-06T18:51:19Z

src/rpc/notifications.cpp

+        size_t queueSize = m_queue.size();
+        if (queueSize > MAX_QUEUE_SIZE) {
+            m_queue.pop_front();
+            LogPrintf("RPC Notification limit has been reached, dropping oldest element\n");


I'd be somwhat worried we'd fill someone's drive with debug log entries in case they forget to deregister a listener, here.

TheBlueMatt · 2017-11-06T18:51:19Z

src/rpc/notifications.cpp

+{
+public:
+    CCriticalSection m_cs_queue_manager;
+    std::map<clientUUID_t, NotificationQueue*> m_map_sequence_numbers;


It'd be great if we could de-duplicate the queues here - no need to have a queue per client, just have a global queue and keep track of how far delayed all the clients are in terms of the sequence number and just clean things up to the furthest-back client.

Also, please use unique_ptr here instead of manual management and maybe remove the queue when there are no registered types (and, I suppose, the client is caught up) instead of keeping around a null queue.

TheBlueMatt · 2017-11-06T18:51:19Z

src/rpc/notifications.cpp

+        }
+        if (startTime + timeOut + (500 / 1000.0) < GetTime())
+            break;
+        MilliSleep(500);


Should probably use a CV and a way for Interrupt to interrupt it instead of calling ShutdownRequested in a 500ms loop.

TheBlueMatt · 2017-11-06T18:51:37Z

Also, looks like the test is failing.

NicolasDorier · 2018-01-10T16:10:46Z

Strong Concept ACK for this one !
@jonasschnelli any idea if you will bring this one from the dead?

jonasschnelli · 2018-01-10T18:50:10Z

@NicolasDorier
I think there is no consensus about an additional push channel... also, @sipa brought up the idea of having a push channel (could be long poll) that acts similar then listsinceblock where the server doesn't need to keep track of clients (keep a queue).
I haven't looked closer at this approach.

NicolasDorier · 2018-01-14T19:15:58Z

I implemented a similar solution in NBXplorer. Basically there is a GetUTXOs(xPub) call, this call replay all the transactions of the xpub in topological order to create the current utxo for this xpub. While playing the transactions, it hashes them along the way (the hash after each transaction is effectively the equivalent of a bookmark). Then the bookmark + the UTXO is sent back to the client.

The client process the result, then call again GetUTXOs(xPub, bookmark). The server does the same operation, replaying all transactions while calculating bookmarks along the way, when it reaches the bookmark passed by the client, it knows that what is after is a differential to the previous bookmark. If there is no differential, it just long poll. If there is a differential, it sends it back to the client.

If the bookmark in parameter was not reached, then the full UTXO is sent back again to the client, with a flag indicating it is not a differential.

This solution does not involve server side state.

practicalswift · 2018-09-02T18:37:16Z

src/rpc/notifications.cpp

+static const char* MSG_HASHBLOCK = "hashblock";
+static const char* MSG_HASHTX = "hashtx";
+
+/* keep the max queue size large becase we don't


Typo found by codespell: becase

practicalswift · 2018-09-02T18:37:40Z

src/rpc/notifications.cpp

+
+    // populates a json object with all notifications in the queue
+    // returns a range to allow removing the elements from the queue
+    // after successfull transmitting


Typo found by codespell: successfull

promag

@jonasschnelli do you plan to pick this again?

promag · 2018-11-30T14:51:37Z

src/rpc/notifications.cpp

+            "\"  ,...\n"
+            "\"]\"\n"
+            "\nExamples:\n"
+            "\nCreate a transaction\n" +


jonasschnelli added the RPC/REST/ZMQ label Apr 26, 2016

jonasschnelli force-pushed the jonasschnelli:2016/04/rpc_signals branch 2 times, most recently Apr 26, 2016

jonasschnelli force-pushed the jonasschnelli:2016/04/rpc_signals branch May 6, 2016

MarcoFalke reviewed May 6, 2016
View changes

qa/rpc-tests/rpcsignals.py Outdated

@@ -0,0 +1,64 @@

#!/usr/bin/env python2

This comment has been minimized.

Sign in to view

MarcoFalke May 6, 2016
Member

Nit: python3

This comment has been minimized.

Sign in to view

jonasschnelli May 6, 2016
Author Member

Fixed.

jonasschnelli force-pushed the jonasschnelli:2016/04/rpc_signals branch 2 times, most recently May 6, 2016

jonasschnelli force-pushed the jonasschnelli:2016/04/rpc_signals branch to 8660db9 Jul 8, 2016

pstratem mentioned this pull request Jul 12, 2016

Block postprocessing is (relatively) slow. #8333

Closed

jonasschnelli mentioned this pull request Sep 9, 2016

Address Travis spurious failures #8680

Merged

jonasschnelli mentioned this pull request Feb 12, 2017

[RPC] Getting confirmations command #9745

Closed

laanwj added this to the 0.15.0 milestone Mar 5, 2017

laanwj modified the milestones: 0.16.0, 0.15.0 Jul 11, 2017

jonasschnelli force-pushed the jonasschnelli:2016/04/rpc_signals branch 2 times, most recently Oct 20, 2017

jonasschnelli mentioned this pull request Oct 23, 2017

ZMQ: add publishers for wallet transactions. #10554

Closed

jonasschnelli added 2 commits Sep 28, 2017

Add RPC long poll notifications

Verified

This commit was signed with the committer’s verified signature.

jonasschnelli Jonas Schnelli

GPG key ID: 1EB776BB03C7922D Learn about signing commits

ca6a3f8

[QA] Add rpcnotification test

Verified

This commit was signed with the committer’s verified signature.

jonasschnelli Jonas Schnelli

GPG key ID: 1EB776BB03C7922D Learn about signing commits

Loading status checks…

2652c55

jonasschnelli force-pushed the jonasschnelli:2016/04/rpc_signals branch to 2652c55 Oct 23, 2017

TheBlueMatt reviewed Nov 6, 2017

View changes

NicolasDorier mentioned this pull request Jan 10, 2018

[RPC] Remove waitforblock and waitfornewblock #10519

Closed

laanwj modified the milestones: 0.16.0, 0.17.0 Jan 11, 2018

NicolasDorier mentioned this pull request Mar 8, 2018

Add waitfor RPC methods nopara73/MagicalCryptoWallet#65

Merged

jonasschnelli mentioned this pull request May 17, 2018

Wallet/RPC: Add listsincetx method with a stateless (server-side) long polling option #13262

Closed

1 of 2 tasks complete

MarcoFalke added the Needs rebase label Jun 6, 2018

jonasschnelli modified the milestones: 0.17.0, 0.18.0 Jul 19, 2018

practicalswift reviewed Sep 2, 2018

View changes

src/rpc/notifications.cpp

static const char* MSG_HASHBLOCK = "hashblock";

static const char* MSG_HASHTX = "hashtx";

/* keep the max queue size large becase we don't

This comment has been minimized.

Sign in to view

practicalswift Sep 2, 2018
Contributor

Typo found by codespell: becase

practicalswift reviewed Sep 2, 2018

View changes

promag reviewed Nov 30, 2018

View changes

src/rpc/notifications.cpp

"\" ,...\n"

"\"]\"\n"

"\nExamples:\n"

"\nCreate a transaction\n" +

This comment has been minimized.

Sign in to view

promag Nov 30, 2018
Member

👀

laanwj removed this from the 0.18.0 milestone Feb 6, 2019

fanquake added the Up for grabs label Jun 17, 2019

fanquake closed this Jun 17, 2019

laanwj removed the Needs rebase label Oct 24, 2019

jonasschnelli mentioned this pull request Nov 10, 2019

GUI: Who is the GUI being designed for and how do we better avoid bikeshedding? #17395

Closed

Bosch-0 mentioned this pull request Aug 5, 2020

Who is the GUI being designed for? -Continued Discussion- bitcoin-core/gui#45

Open

bitcoin / bitcoin

[RPC] Add RPC long poll notifications #7949

[RPC] Add RPC long poll notifications #7949

jonasschnelli commented Apr 26, 2016

laanwj commented Apr 26, 2016 •

edited

jonasschnelli commented Apr 26, 2016

jonasschnelli commented Apr 26, 2016 •

edited

jonasschnelli commented May 6, 2016

This comment has been minimized.

This comment has been minimized.

jonasschnelli commented May 12, 2016

laanwj commented Jan 11, 2017

jonasschnelli commented Jan 11, 2017

TheBlueMatt commented Sep 28, 2017

jonasschnelli commented Sep 28, 2017

jonasschnelli commented Oct 20, 2017

TheBlueMatt left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

TheBlueMatt commented Nov 6, 2017

NicolasDorier commented Jan 10, 2018 •

edited

jonasschnelli commented Jan 10, 2018

NicolasDorier commented Jan 14, 2018 •

edited

This comment has been minimized.

This comment has been minimized.

promag left a comment

This comment has been minimized.

bitcoin / bitcoin

[RPC] Add RPC long poll notifications #7949

[RPC] Add RPC long poll notifications #7949

Conversation

jonasschnelli commented Apr 26, 2016

Reasons for another notification interface

How does it work

Downsides

New RPC calls

Missing

laanwj commented Apr 26, 2016 • edited

jonasschnelli commented Apr 26, 2016

jonasschnelli commented Apr 26, 2016 • edited

jonasschnelli commented May 6, 2016

This comment has been minimized.

MarcoFalke May 6, 2016 Member

This comment has been minimized.

jonasschnelli May 6, 2016 Author Member

jonasschnelli commented May 12, 2016

laanwj commented Jan 11, 2017

jonasschnelli commented Jan 11, 2017

TheBlueMatt commented Sep 28, 2017

jonasschnelli commented Sep 28, 2017

jonasschnelli commented Oct 20, 2017

TheBlueMatt left a comment

This comment has been minimized.

TheBlueMatt Nov 6, 2017 Contributor

This comment has been minimized.

TheBlueMatt Nov 6, 2017 Contributor

This comment has been minimized.

TheBlueMatt Nov 6, 2017 Contributor

This comment has been minimized.

TheBlueMatt Nov 6, 2017 Contributor

This comment has been minimized.

TheBlueMatt Nov 6, 2017 Contributor

This comment has been minimized.

TheBlueMatt Nov 6, 2017 Contributor

This comment has been minimized.

TheBlueMatt Nov 6, 2017 Contributor

TheBlueMatt commented Nov 6, 2017

NicolasDorier commented Jan 10, 2018 • edited

jonasschnelli commented Jan 10, 2018

NicolasDorier commented Jan 14, 2018 • edited

This comment has been minimized.

practicalswift Sep 2, 2018 Contributor

This comment has been minimized.

practicalswift Sep 2, 2018 Contributor

promag left a comment

This comment has been minimized.

promag Nov 30, 2018 Member

laanwj commented Apr 26, 2016 •

edited

jonasschnelli commented Apr 26, 2016 •

edited

MarcoFalke May 6, 2016
Member

jonasschnelli May 6, 2016
Author Member

TheBlueMatt Nov 6, 2017
Contributor

TheBlueMatt Nov 6, 2017
Contributor

TheBlueMatt Nov 6, 2017
Contributor

TheBlueMatt Nov 6, 2017
Contributor

TheBlueMatt Nov 6, 2017
Contributor

TheBlueMatt Nov 6, 2017
Contributor

TheBlueMatt Nov 6, 2017
Contributor

NicolasDorier commented Jan 10, 2018 •

edited

NicolasDorier commented Jan 14, 2018 •

edited

practicalswift Sep 2, 2018
Contributor

practicalswift Sep 2, 2018
Contributor

promag Nov 30, 2018
Member