[RPC] Add RPC long poll notifications

[jonasschnelli]

Reasons for another notification interface

• Currently there is no interface that could be extended to "private" notification secured behind the authorization (like peers connected/disconnected or a new wallet relevant transaction notification)
• HTTP long poll notifications are very easy to set up and require almost no dependencies
• HTTP long poll notifications can easily pushed over the internet using httpd reverse proxy together with a popper authentication method (certs or http auth digest) together with TLS.
• HTTP long poll would allow connecting applications to do all kinds of things with just a single communication channel (currently you need RPC & ZMQ for most use cases which would require VPN or a fancy multi port stunnel connection to broadcast the notification over the internet)

How does it work

• The listener calls the pollnotification RPC command.
• If no notifications are available, the RPC thread will idle for a given timeout (30s by default)
• If a notification was fired during the 30 seconds, the longpoll call will be responded with the new notification(s)
• The client/listener can immediately reconnect and wait again
• If notifications are already in the queue, the pollnotification command will immediately response.
• Notifications can't get lost (possible to lose them during http transfer and if one exceed the queue limit)

Downsides

• JSON encoding overhead

New RPC calls

setregisterednotifications [<notificationtype>] (possible types are hashtx and hashblock)
getregisterednotifications
pollnotifications

Missing

• More tests
• Documentation

I'd like to use a such interface to work on a remote GUI (use case: GUI on your local desktop, node on a VPS).

[laanwj]

I like the concept of being able to listen for events through http, however I think this is severely limited by having server-side state, limiting the number of listeners to only one.

What I'd personally prefer is, instead of longpolling, to subscribe to a 'stream' of events (e.g. websocket or just chunked encoding), where the set of events to listen to is in the request. This avoids having to store any client-state in the server - at least for longer than the request lasts.

[jonasschnelli]

[... ] having server-side state, limiting the number of listeners to only one

Right. The current implementation limits to only one listener. Extending this PR so it would support a client chosen UUID would not be very complicated (a set of queues and a set of registered notification types). Clients could register notification types along with a client-chosen UUID.
I might extend this PR to support multiple listeners.

[jonasschnelli]

Added a commit that allows multiple clients at the same time.

The new RPC commands require now a clientUUID parameter (a per client unique string, ideally a UUID after RFC 4122). Bitcoind keeps a queue, sequence numbers and registered types per client.

There is currently not max client limit and no way to remove clients (though you can unregister all notification types but not empty the current queue).

[jonasschnelli]

Rebased.
Would be nice to get some concept NACKs/ACKs.

[MarcoFalke]

Nit: python3

[jonasschnelli]

Fixed.

[jonasschnelli]

Rebased.

[laanwj]

Gah we need to take a look at this again after 0.14 is released.

[jonasschnelli]

Yes. Sure. I'll try to re-base and overhaul this soon.