Add new ALM transport

[Neverlord]

Currently, Broker requires its users to provide loop-free deployments. The new ALM transport makes it much easier to deploy Broker, since Broker no longer simply floods published data but instead discovers and manages available paths.

I've added a new doc/devs.rst file to the documentation to cover how Broker works internally as well as how the (new) code is structured. At this point, feedback to this would be most welcomed.

Remaining ToDos:

• Extend the routing table to store paths rather than distances
• Implement source routing
• Add path revocation to propagate loss of connectivities
• Implement ordering & loss detection (for data stores)
• Implement re-attaching of clones on message loss

[rsmmr]

I'm seeing compiler errors: ```
/Users/robin/bro/master/aux/broker/build/caf-build/libcaf_core/sec_strings.cpp:105:15: error: no member named 'remote_lookup_failed' in
'caf::sec'; did you mean 'remote_linking_failed'?
case sec::remote_lookup_failed:
~~~~~^~~~~~~~~~~~~~~~~~~~
remote_linking_failed
/Users/robin/bro/master/aux/broker/build/caf-build/libcaf_core/caf/sec.hpp:92:3: note: 'remote_linking_failed' declared here
remote_linking_failed,
^
/Users/robin/bro/master/aux/broker/build/caf-build/libcaf_core/sec_strings.cpp:105:10: error: duplicate case value 'remote_linking_failed'
case sec::remote_lookup_failed:
^
/Users/robin/bro/master/aux/broker/build/caf-build/libcaf_core/sec_strings.cpp:69:10: note: previous case defined here
case sec::remote_linking_failed:
^
2 errors generated.

Submodules are up to date, am I missing something else?

[rsmmr]

Focussed on the new dev guide for now. That's nice, certainly very helpful. Couple suggestions:

• I'd turn the sections around and start with the architecture, then the ALM can refer back to that as helpful.

• A couple diagrams would be helpful: one for the architecture showing the main components and data flows (core actor, mixins, etc.; also where it goes down into CAF); and one (or two) for the flow of messages, ideally with an example of what happens as they get forwarded.

I'll leave some more smaller comments/questions on the RST code.

[Neverlord]

Thanks for the feedback! Just a quick note (content updates in progress):

Submodules are up to date, am I missing something else?

Sorry, the 3rdparty submodule was lagging behind. Should work now.

[Neverlord]

@rsmmr I've integrated your feedback on the devs section. Let me know what you think of the additions / changes and how I can improve the section further. 🙂

[rsmmr]

Nice, thanks for the updates to the devs section. The ALM description sounds all good, and the architecture section is quite helpful - I'm feeling like I'm starting to understand Broker finally ;-)

[rsmmr]

(... and no further particular comments)

[Neverlord]

A quick update, I've wanted to share: while some unit tests still fail (working on it), the ALM branch runs stable enough for a first quick performance comparison with current master. I've used the broker-benchmark tool with Broker compiled as Release version:

broker-benchmark -r 100000000 -s 1 -t 2 localhost:9090

For the quick overview, I've let it run for some time, then took 30 values (output of the server for received messages per second) each. Here are the results:

Branch	Median	Average
master	31,588.5	30,992.6
ALM	32,471.0	32,368.5

I wouldn't read too much into this, since the values do fluctuate. However, the take-away is that the new source routing seems to have no negative effect on the performance.

Here is the raw data I've compiled this from: 2020-03-28 Broker ALM branch comparison.txt.

Of course, most insights will come when looking at a full cluster. Once I have ported the new cluster benchmark, I'll do a more thorough comparison.

[Neverlord]

@rsmmr I was thinking about a path forward for this PR. I think as it stands, this is hard to review / integrate in its entirety. The branch contains several big changes to the entire code base plus a new communication backend. I think there are two options we could choose: 1) do some incremental reviews (you've already did one) and eventually merge everything at once or 2) separate refactoring from actual new features with individual PRs.

For option two, I would factor out at least these two steps as separate PR:

• The switch to a mixin-based design. This encompasses the first chunk of commits minus the routing table.
• Extending status with the new codes endpoint_discovered and endpoint_unreachable. This affected more code than I originally though (almost done with that refactoring). While they'd be unused in master for a while, I think we could still merge this change ahead of the actual ALM implementation.

Personally, I favor option 2. Aside from making reviews more manageable and focused, merging individual parts earlier also helps to avoid this PR running out of sync.

[rsmmr]

Yeah, let's go with Option 2, and merge the refactoring & static notifications first.

If you can split things out further into more, smaller chunks, that would be worth the effort; both for the refactoring and then for the new functionality. We'll review each to the degree we can, with a particular focus on not breaking existing cluster topologies.

Also, let's remain flexible on timeline: The closer we get to the release, the more risky a merge of complex changes will be. Depending on how things progress, a viable model could be getting the foundational commits in before 2.2, and then the new logic after the release early in the next cycle. Let's see.

[Neverlord]

While porting the data store actors to the new channels, I've also streamlined the communication between master and clone. So this PR closes #99 and closes #125.