Integrate optional web sockets layer for C/S API.

[evoskuil]

Headers only, optionally built on boost::asio (only):
https://github.com/zaphoyd/websocketpp

[evoskuil]

It would be nicer to layer a zeromq-based implementation over the zeromq client-server API, but the implementations don't look as mature, and there are cURL dependencies. The ASIO-based implementation is much better in these aspects.

[narodnik]

The ASIO one looks kind of old and not well-written. Maybe I'm being judgemental.

Since websockets + JSON is inefficient anyway, what about a Python based daemon that creates the websockets API? A Python server is also hackable for people who want different transports while ZMQ delivers the data locally (which is what it does best - inter-daemon communication).

I already have the code for this, it just needs to be updated for the latest libbitcoin-server.

[evoskuil]

Nothing wrong with that as an option, but a couple of considerations.

We worked very hard on getting the Python dependencies out of sx (for bx). Presently, all of the libbitcoin applications can be "installed" by simply placing a statically linked file on any Linux, Mac or Windows box. Adding a second process and a Python dependency breaks that paradigm and complicates both deployment and management. Additionally, libbitcoin is self-described as a C++ toolkit. Adding a second language means that two language skills are required.

As such I would still want to produce an in-process interface. Bridging zeromq and websockets in-process is straightforward. As a headers only library based on ASIO the only question would be quality/support of the implementation, but looking at it superficially I haven't found any seen any reason for concern. The only issue is that use of TLS requires OpenSSL, but I don't anticipate any implementation avoiding that presently. See also: https://docs.websocketpp.org

I would really like someone to be able to launch the single file executable, open a browser to the local host, and see the blockchain - that easy.

[narodnik]

ok, I will play with websocketpp and look further at this.

[narodnik]

Also an option:

https://libwebsockets.org/

"Libwebsockets is a lightweight pure C library built to use minimal CPU and memory resources, and provide fast throughput in both directions as client or server."

Active community:

https://libwebsockets.org/pipermail/libwebsockets/2016-June/thread.html

https://github.com/warmcat/libwebsockets

[evoskuil]

That looks nice from a community and quality standpoint, but unless I'm missing something I'd consider the dependency chain a significant problem. Apart from SSL/TLS (which is is probably inescapable, but is optional) there may also be zlib, libev, lubuv, and sqlite. Given we are ASIO and ZMQ - based I would not be keen on bringing in additional redundant asynchronous I/O dependencies. And every dependency in general is a major headache. Right @thecodefactory ?

[thecodefactory]

True about the additional dependencies! @RojavaCrypto: @evoskuil has been working hard to reduce the number of dependencies because they become unmanageable for all of the build configurations and platforms that are supported. There's a complex enough build/configuration matrix already that it requires very careful consideration on whether bringing in new dependencies is worth the amount of additional maintenance headaches (for all maintainers/builders/testers). I'm referenced here because some optional bx features I've worked on brought in additional dependencies (e.g. libpng/libqrcode), and while those were easy enough to get working locally on my configuration of GNU/Linux where I develop/test -- they couldn't be properly supported on other platforms without additional work and testing that I frankly didn't think about and ultimately didn't want to deal with (for example, the Windows builds since I don't run Windows). As @evoskuil mentioned previously, these dependencies also tend to introduce new issues that break static builds of functional single file binaries, which potentially impacts the end users negatively.

[narodnik]

Yep, OK scrap libwebsockets it was just an idea.

But how about a Python daemon for websockets which is simple and easily/quickly improved?

https://stackoverflow.com/questions/14165398/a-good-python-to-exe-compiler

If you use Tornado (which I'd use), you not only get websockets, but also a web server. So you could use Python to make a wrapper around libbitcoin-server that is responsible for stopping, starting and configuring it, as well as displaying the output (that you can view in the web browser). All of this could be wrapped up in a single unpacking .exe

The code for a websockets bridge already exists:

https://github.com/darkwallet/gateway

Python has strong string manipulation and very good template libraries, which means it would be efficient for developing a dynamic hosted website with all modern features (login .etc).

https://wiki.python.org/moin/Templating

Which will be necessary on a production scale. For now, I'd only allow localhost access, but in the future you'll want authentication, tokens, XSS protection, ... lots of stuff which is in Python already.

Also Python is fast to write and have working. Because it's dynamic, you can very easily write code to be plugin based.

I can do a small experiment to try packaging a Python program with some data files, Tornado and a bunch of other libs if @evoskuil feels this is the right direction.

[narodnik]

So I'm clear, I mean Python websockets <-> ZMQ bridge which is responsible for the server, vs directly writing websockets serialization into the server.

Python websockets:

• Easy to write, quick to modify.
• Lower entry for other developers, especially with complex web management features.
• Code already exists.
• Libraries are better developed for websockets, web, ...

C++ websockets:

• More performant (but use ZMQ if that matters - websockets is terrible for performance).
• Directly integrated (is this a good thing?)
• Less dependencies.
• More efficient (instead of db <-> ZMQ <-> websockets).

[evoskuil]

I'm supportive of an additional library based on Python if someone wants to maintain it, but it would be a different animal than the core C++ libs.

The direct integration simplifies deployment, management and maintenance of the overall system. As for the integrated approach, websocketpp has the following benefits (over the other integrated options):

• C++ 11 (optional, can downgrade)
• Boost based (with C++11 optimization to std libs)
• ASIO based (consistent with our proactor design)
• headers only (easy static link for single file build on all platforms)
• single dependency (minimal maintenance for installs and continuous integration)

Given the similarity of the code with our own, it would be straightforward for us to patch issues in it as well. I've had to do this several times with zeromq.

[lastcanal]

I have just open-sourced a Websocket and REST server for BS written in Elixir that runs on the massively concurrent Erlang VM. It supports most version 2 API and provides subscriptions (although subscriptions are currently undocumented).

It uses an Erlang port process written in C to communicate with BS via CZMQ. Right now there is no ZMQ-cert support in the Elixir client, but it is very possible to get it working due to using CZMQ under the hood.

It uses an Erlang NIF written in C++ to parse transactions via libbitcoin: libbitcoin-nif

We have also built a block explorer using this API which can be seen live here: https://voyager.cancoin.co

I'd love to know what you think! We are planning on using this as the blockchain backend of our on-chain multisig OTC marketplace.

[narodnik]

Damn, that's great. Erlang is a strong language, especially for concurrency.

I just spent this month working on something too, using Python asyncio. We should combine our efforts.

Python is the 4th most popular language (Java, C, C++, Python) with strong C/C++ integration, and the asyncio in the new Python 3 is very mature & working well.

https://github.com/rojavacrypto/python-libbitcoin

https://github.com/rojavacrypto/libbitcoin-gateway

Your work looks great. Well done. I'm going to download it all now.

I'm using Tornado web which is a library used by Facebook for making web applications. So in the future, we could integrate a web management platform for the bs daemon. Does Erlang have support for that?

[lastcanal]

I'd love to combine efforts. There are still some rough spots that I am working to clean up (especially with libbitcoin-nif) so you might have some trouble building it against libbitcoin master at the moment. I saw this issue and rushed to open-source it ;). The API/names, etc can all be changed fairly easily; I was trying to keep it consistent with the darkwallet gateway but I don't think is anymore.

Are you thinking of a web management platform for reconfiguring a running libbitcoin server?

There are 2 ways to integrate C++ into Erlang: one is a Native Implemented Function
(NIF) which is C/C++ code loaded into the Erlang virtual machine; the other is a Port process, which runs as a separate OS process spawned by Erlang that communicates via stdin/stdout. NIFs can only be used for functions that execute in a very short amount of time, or else it breaks all of Erlang's pre-emptive scheduling, so I don't think that would work.

A Port process that spawns BS then communicates with it via stdin/stdout could work. A web interface could be built that sends messages to the Port process via a webserver, which in turn would send a message to bs via stdin, which could then reconfigure and reply via stdout. The plus side is the bs process would then be supervised by Erlang's supervisor tree.

Here is the library I use to serialize C++ into erlang terms: https://github.com/goertzenator/nifpp