Distributed Services / Surveyor Protocol

[roscopecoltran]

Hi,

Hope you are all well !

I had a quick question about your interesting repo using nanomsg:

I seek to create a distributed services tree for another project called find-object like described in this article (ref. repo)

A GUI and console mode are available, and the interesting part is that you can handle an extensive set of options for launching the cli/tcp server (ref) with an .ini config file (link).

I wanted to wrap them with cpp server, and build up some RPC/HTTP api endpoints (add/delete images or update config), and to launch 2 or 3 services with different configurations. On top, I wanted to re-use cpp server as an api gateway (surveyor) for bindings the 3 services (respondents) and aggregate the final response.

The ultimate goal is to compare results for the same input image in real time from a stream of images. (ref); that's why I thought that the nanomsg's surveyor protocol could be a good fit.

As I saw that you integrated some cpp nanomsg components to cpp server, I wanted to get your input or point of view about that.

Questions:
How would u see such distributed integration with your project ?
Is it possible to manage also the configuration of your cpp server from configuration file (yaml or ini) ?

Cheers,
Richard

[chronoxor]

Currently my CppServer provides TCP, UDP, WebSocket, Nanomsg endpoint. In future I'm going to add HTTPS/REST endpoint, some serialization engines and RPC.

You can fully configure server/client from code. Just use some configuration lib to parse yaml or ini and create and configure server/client with required parameters. I'm not plan to add file based configuration directly to CppServer library.

[roscopecoltran]

Hi,

Hope you are all well !

I have a couple of additional suggestions/questions for you ^^

Cross-compilation, dependencies management for CMake:

I was wondering if you were interested by a pull request with CppServer bundled with Hunter and Polly for managing dependencies ? I started to modify the CMakeLists.txt as several dependencies are already hunterized.

• RabbitMQ-C: https://github.com/ruslo/hunter/blob/master/examples/rabbitmq-c/CMakeLists.txt#L12
• Websocketpp: https://github.com/ruslo/hunter/blob/master/examples/websocketpp/CMakeLists.txt#L7-L9
• Protobuf: https://github.com/ruslo/hunter/blob/master/examples/Protobuf/CMakeLists.txt#L11-L28
• ZeroMQ: https://github.com/ruslo/hunter/blob/master/examples/ZeroMQ/CMakeLists.txt#L9-L19
• ZMQPP: https://github.com/ruslo/hunter/blob/master/examples/ZMQPP/CMakeLists.txt#L9-L15
• zookeeper: https://github.com/ruslo/hunter/blob/master/examples/zookeeper/CMakeLists.txt#L9-L15
• YAML-CPP: https://github.com/ruslo/hunter/blob/master/examples/yaml-cpp/CMakeLists.txt#L9-L15
• AutoBahn-CPP: https://github.com/ruslo/hunter/blob/master/examples/autobahn-cpp/CMakeLists.txt#L7-L14
• Cereal: https://github.com/ruslo/hunter/blob/master/examples/cereal/CMakeLists.txt#L13-L23
• Msgpack: https://github.com/ruslo/hunter/blob/master/examples/msgpack/CMakeLists.txt#L7-L14
• Flatbuffers: https://github.com/ruslo/hunter/blob/master/examples/flatbuffers/CMakeLists.txt#L13-L19

Web requests and routes

Would it be possible to create a backend to manage web routes binding Nanomsg (btw I do not know if u saw the next version of Nanomsg called NNG, with a user-defined respondents setup per routes ? (eg. http://my-classes.com/2014/04/26/combining-zeromq-and-nanomsg-for-serving-web-requests/ ).

The intent is to make a programmatic load balancer, like vulcand, and gather responses like Krakend.

Some useful dependencies in C++11:

• MsgFlo-CPP: https://github.com/msgflo/msgflo-cpp
• etcdcpp: https://github.com/suryanathan/etcdcpp#key-space-operations

I can make a backend with vue.js + etcd through a docker compose orchestraction. Even more, I will check if node-red or msgflo could be good candidates for such administration web-ui.

Have a great day !

Cheers,
Richard

[chronoxor]

Recently I created a serialization library with RapidJson and Flatbuffer backends - https://github.com/chronoxor/CppSerialization

I've already add WebSocket support using Websocketpp and now adding RESTfull server/client based on restbed library (https://github.com/Corvusoft/restbed).

I'll see if it is not difficult to integrate all my C++ libraries with your Hunter and create corresponding pull requests.

Also I'm very interesting now in some C++ solution for service discovery and configuration management (between server nodes, between data-centers) with automatic conflict resolving. Something like Zookeeper or Consul does, but more lightweight.

[roscopecoltran]

Hi,

Thanks for your reply :-)

For anything related to deployment, I always watched golang based services as much more convenient to dockerize and deploy/test with docker-compose, minikube or k8s. Here are a couple of links that I bookmarked by the past, if it can helps you for c++ and golang.

C++:

• https://github.com/beschulz/asio_service_discovery (With Asio)
  • by the same author, interesting project for serialization: https://github.com/beschulz/thrift_asio
• https://github.com/mpwolfe/gRPC-C-Go-Registry-Consul
• https://github.com/rock-core/tools-service_discovery
• https://github.com/mcparland/upmuGateway-serviceDataDiscovery

Golang:

• https://github.com/daizuozhuo/etcd-service-discovery
• https://github.com/smallnest/rpcx
• https://github.com/dahernan/gopherdiscovery (With nanomsg)
• https://github.com/nanoscaleio/gateway (With nanomsg)
• https://github.com/ibmendoza/iris (decentralized cloud messaging, sadly not maintained but had a great potential as it was zero configuration, self organized and based on semantic addressing)
• https://github.com/dahernan/goHystrix

My take is that a service connecting to an etcd kv store would be the most flexible way to build something simple. I mentioned some etcd c++ libraries in a message above (they are usually based on curl).

Have a great day !

Cheers,
Richard

[roscopecoltran]

Forgot to share with you some concurrency librairies for concurrency in C (same author as zeromq and nanomsg), you probably already noticed them, if yes, skip it :-) :

• https://github.com/sustrik/wsock (WebSocket library for libmill)
• https://github.com/sustrik/dsock (easily composable microprotocols, using libdill)
• https://github.com/sustrik/libmill (Go-style concurrency in C)
• https://github.com/sustrik/libdill (Structured concurrency in C)

[chronoxor]

@roscopecoltran, I'd like to inform you that from now CppServer supports HTTP/HTTPS server & client out of the box with very good performance!

[chronoxor]

@roscopecoltran, I'd like to inform you that from now CppServer supports WebSocket server & client out of the box with very good performance!