Establishing direct connections through a central broker

[SegfaultCreator]

Hello everyone,

I am currently trying to exchange our token-ring based network communication on a broadcast basis (spread toolkit) with a more efficient approach using Zeromq.
We have a distributed architecture which consists of up to 25 separate executables, which run on 10+ computers in a local network.
In this architecture, we have some kind of the Dynamic Discovery Problem described in chapter 2 of the tutorials as it is flexible which computer runs which executables.
We implemented the XPub/XSub pattern proposed for this, but we noticed that all of the data now runs through the proxy server.
What we want to have instead is a central broker that is responsible for establishing direct connections between pubs and subs (like the host of a big party that organizes the event, but the communication happens between the party joiners).
However, I did not find any pattern, that fits our use case, in the documentation. I thought about having the broker save all the publishers and subscribers to manage new communication partners. Is there anything I have overseen or are there any major issues that contradict using such an approach?

Thanks in Advance
SegfaultCreator (now mutated to MessageDropper)​

[rich-coe]

I've used the following before (on another protocol) and have implemented and using this on zmq.

Central broker uses XSUB/XPUB and listens for and republishes 'HELLO' messages.
It may also maintain a list of service endpoints and responds to requests
for the service list.

Client connects to central broker subscribing to HELLO messages.
Client sends HELLO message contains list (service name, listen socket) it provides.
Client ignores HELLO messages sent from self.
When client receives HELLO message from other clients, update internal
registry of service names, with socket endpoints.

Client responds to HELLO with a OLLEH (or some other message) with it's
service list, or the broker can respond to the joining client with the current
service list.

When a 'client' wants to send to service name, initiate connection endpoint with
service by service name.
If expecting async reply, create unique 'reply' service name on local 'client's'
listener endpoint. Include REPLY (service name, listen socket) in the protocol
message to service.

On zmq there's the 'slow initial connection problem' so you may have
establish the pub/sub connection between services during service registration
instead of at message initiation.

On the Central broker, when sockets close, remove the service from the list
and broadcast GOODBYE with service name(s) to all clients.

[sappo]

Dafka uses the approach you described. Have a look the central tower (https://github.com/zeromq/dafka/blob/master/src/dafka_tower.c) and the beacons used in the peers of the network (https://github.com/zeromq/dafka/blob/master/src/dafka_beacon.c)

[SegfaultCreator]

Thanks for these fast and promising answers.
Dafka seems to fit very well.
If i may ask @sappo , as you are a maintainer of Dafka as well:
Has it been tested on 20.04 LTS? I have some dependencies problems with leveldb upon configuring even though i installed it (apt install libleveldb-dev)

[sappo]

@SegfaultCreator I do my development on 20.04, so it should work!

[SegfaultCreator]

@sappo Great to hear that you have it running under 20.04!
Unfortunately, i still get linking errors with leveldb after trying out clang-11 with lld-11 and gcc with gnu-ld linker. Do you have any special installation of leveldb or did you also use the liblevel-db-dev=1.22-3ubuntu2 provided by ubuntu repository?

[sappo]

No I don't have any special installation. This is what apt says I have installed:

libleveldb-dev/focal,now 1.22-3ubuntu2 amd64 [installed]
libleveldb1d/focal,now 1.22-3ubuntu2 amd64 [installed,automatic]