How router-to-router work?

[SunFulong]

Hi all,

for a long time, Router-to-Router Links still in state "write me".

could we discuss how does R2R working? I have multiple data center, both of them have at least one router, and have central router on cloud.

will this chapter talk about how to connect them?

[SunFulong]

currently, I wrote a "bridge" which connect to 2 routers, and registered different prefix topic and forward message if hit.

e.g. router A have "neuedu.dnui." topics, and router B have "neuedu.nuit." topics, a "bridge" registered to both of them, so, if a client call "neuedu.nuit.*" on router A, the message will received by "bridge" and forward to router B, then return result back.

[KSDaemon]

Well.... R2R Links are not as easy as someone might expect...
Once ago I started thinking of them and we even had some basic discussions around that.

Let me refresh my thoughts and post them so we can discuss them.

[oberstet]

will this chapter talk about how to connect them?

well, yes, "how they connect" at the networking level (which is the easy part), and then of course how they create the illusion of regular/unified realms on top of distributed meshes of router nodes (full, tree, etc) of both a single operator, and nodes by different operators ("federated meshes"), which was always the ultimate end-goal rgd WAMP (at least in my mind).

fwiw, Crossbar.io has a working implementation of this stuff - it does "work", but there are bugs left, both implementation wise, and I'm pretty sure rgd design gaps as well, e.g. when you start to combine r2r links with other complex WAMP features like meta API etc

I had a quick look at the "text" you've linked: forget it, this is not even close to what is actually already there in Crossbar.io ;) anyways, stopped working on the text some time ago, sorry, I need to earn money, and this stuff isn't doing that.

I have multiple data center, both of them have at least one router, and have central router on cloud.

sure. that is probably the most simple topology supported. 3 nodes, one of them network reachable from the others (with a public routable IP address). I call that "core-node/edge tree". what I used when developing was already going further: "core-mesh/edge tree" with a HA / scale-out full mesh at core in cloud, and off-hanging (hierarchical) trees of edge nodes distributed in WAN - and this does indeed work. it's pretty cool already, and I'm not aware of anything else that does similar. I'm also not aware of any other implementation of r2r besides Crossbar.io

anyways, hope this helps!

[SunFulong]

hi @oberstet, thank you for your answer.

this is not even close to what is actually already there in Crossbar.io

will the Crossbar.io's implementation be the spec, or just a standalone behavior?

and this does indeed work. it's pretty cool already

if the implementation will be spec, could you share how does it working current now?

I did similar things and running a production on it. I want to migrate my implementation as closer as the "next" spec.

[oberstet]

Hi,

there are "docs" mostly rgd design, also examples etc sprinkled across this repo (outside main spec) and the Crossbar.io and some more repos, and in fact, I just notice I have unfinished branches even for this repo.

anyways, if you want to see some kinda top-level pic of how all the pieces (r2r links, e2e encryption, decentralization) fit together, fwiw, here (this pic is incomplete still, but should give a first clue)

_rlink_topologies.pdf

---

large parts of above is implemented in Crossbar.io

privately, I am working on a reimplementation of Crossbar.io in Rust which runs on anything from MCU with 256k RAM to cloud, and FaaS via WASM

if the implementation will be spec, could you share how does it working current now?

a) I don't know, and b) sorry, no time, doesn't earn me bucks

[konsultaner]

Just wanted to note here that I think R2R should not be part of the WAMP definition. I don't see a reason why one would use 2 different router implementations to establish a router network. So I see R2R as something that should be router implementation specific. WAMP is quite a complex but therefore very powerful protocol. Adding this much more complexity will most probably stop people from implementing it. This is just an opinion, no offence!

[oberstet]

thanks a lot for your comment and perspective, and yes, this is definitely one way to look at it, no offense taken;) let me nevertheless comment, give my personal view, but in any case, this stuff isn't in the spec (there is an AP "draft" which hasn't even the full proposal -.- so it is WIP) as of today

I don't see a reason why one would use 2 different router implementations to establish a router network

say you want to mesh between two networks operated by different operators / parties each preferring their router implementation

Adding this much more complexity will most probably stop people from implementing it.

we had discussed this specific point among implementors quite some time ago .. can't find it now .. but it is true;) as a consequence of this discussion we have split the spec into Basic Profile (BP) and Advanced Profile (AP) to achieve both goals:

• provide an "easy spec" which can be implemented quickly, and provides already core WAMP powers, and at the same time
• do not let BP hold us back in moving WAMP forward even further with fancy/complex features

[konsultaner]

say you want to mesh between two networks operated by different operators / parties each preferring their router implementation

I see that point. But I also see, that most super-heavy-advanced users will most probably go with crossbar anyway. If I would start over today, I wouln't bother to write my own router anymore. That was more a challange to me. I could have also just contributed to crossbar. It is what it is now.

I do see why one would write a client implementation. But as we talk R2R I don't see it in the specs.

But still this is a good point and since you started it why not finish it one day. My biggest SASS solution has more than 100k users per day, max. 1.6Gbit throughput and I don't need R2R atm.

[oberstet]

If I would start over today, I wouln't bother to write my own router anymore. That was more a challange to me. I could have also just contributed to crossbar. It is what it is now.

Of course contributing to Crossbar.io would have been nice, but another router implementation is highly welcome too!

Implementation diversity is good!

And all WAMP implementations (clients and routers) - which make an honest effort to comply to the spec - are welcome, closed source and OSS!

I do see why one would write a client implementation. But as we talk R2R I don't see it in the specs.

Yeah, that's because R2R is compatible with any client implementation!

Crossbar.io implements that, as well as E2E, and uses that also in R2R ... but that wouldn't be necessary I think - now that I think about triggered by our discussion.

What does absolutely build on / require R2R with E2E is federated R2R - because trust needs to be established not only between router nodes of different ops, but even more importantly between clients connected to different nodes by different ops (potentially). without relying on any one router op.

Alice trusts Bob (both clients), and neither Alice nor Bob trusts or cares about David and Erin (see above PDF)

Of course, without E2E, has this consequence: clients MUST trust router operators. be it alone, or together with R2R (without E2E)

[konsultaner]

Ok I see... why not have R2R as a third profile next to basic and advanced. Maybe be call it enterprise or something similar? I think in about 2 years my largest customer will reach a point where we would have to geographically scale out our router instance. I would be happy to share my experience with R2R and add some input to that topic and draw some sequence diagrams 😅

[oberstet]

why not have R2R as a third profile next to basic and advanced.

the distinction and segmentation between BP and AP is well defined:

• for an implementation to call itself "WAMP", it MUST implement ALL of the BP
• on top of that, an implementation MAY implement ANY PARTS of the AP at its own choice (but if it implements part X of the AP, it MUST implement it trying to reflect the spec for X "correctly" and "completely")

tbh, from that angle, I have trouble seeing a sane definition of 3rd profile. also, the AP already has portions required for R2R (in a federated setting), namely E2E and such.

but of course there are different angles possible and there will be different opinions, and in the end, I don't care that much about the headlines or structuring into chapters, but most importantly: it's not yet ready for prime time anyways;) finally, there currently only is exactly one implementation of R2R (to my knowledge), which is Crossbar.io/Python. my private Rust reimplementation of Crossbar.io will be the 2nd. and being a paranoid perfectionist, I don't believe R2R is real until I see at least 2 implementations talking. ideally, those would come from different implementers, but at least two in two languages would be the bare minimum for me. if I can find time/reason/money to make it happen that is;)

[KSDaemon]

Okay. I finally got time to write some my thoughts on the topic. It is mostly brainstorming ideas rather than ready-made solutions. But I hope they still might be useful and together we can really create a cool and reliable design and specification (and of course implementation then)

[KSDaemon]

At the moment Router-to-router links (R2RL) are not described in the WAMP specification and are implemented in Crossbar WAMP Router only and are based on regular WAMP connections between routers.

The current implementation in Crossbar works something like this (@oberstet correct me if I’m wrong):

• Regular WAMP connection is established from one router to another router in the same realm
  • If routers need to share N realms, then N WAMP connections using N underlying TCP connections will be established between the same routers.
• Every router listens to META events like new RPC registration, subscription, publications, and so on and does the same things as an internal client on the connected router: create a new RPC registration, subscription, or publish event.
• Upon receiving addressed messages on one side, the WAMP router or its client counterpart deserializes the message first and then prepares a new message on the other side storing in runtime message relation data

This approach wastes some resources: memory for storing runtime objects, CPU for finding related peers, multiple independent network TCP connections, and serializing/deserializing WAMP messages.

Real R2RL should work differently. Here are the main ideas:

• Use application-level protocols that support multiplexing, like QUIC for the underlying transport. In the WAMP protocol, the transport layer is loosely coupled with application one, so this can be described and done separately. But should be taken into account for performance and efficiency reasons.
• As realm plays a crucial part in WAMP protocol, even if two routers need to share a few realms, it is still required to create one WAMP connection/session per realm
  • That is because different realms can have different access control policies
  • And that is why so important to use transport that allows processing a few application-level connections in one transport connection
• R2RL should be done as the special client connected to another router:
  • It must support WAMP handshaking (HELLO/WELCOME/CHALLENGE/AUTHENTICATE)
  • It must not make any registration, subscription, or publication calls from itself as a regular WAMP client
  • It should be treated like a router/gateway uplink. What does it mean?
    • The Source router that is initiating a connection to the target router after successful WAMP session establishment must advertise meta-information about its local registrations and subscriptions
    • During the session lifetime, every side must track changes in local registrations and subscriptions and advertise these changes (let's name them URI Route Updates URU) to another side
    • Every router involved in R2R relations must keep and update in a timely fashion so-called Routing Database (RDB). RDB stores routes to every RPC and Topic registered in the current realm
    • Routing table can be (re)build with Bellman–Ford algorithm, used in RIP Routing protocol with some modifications (do not store only best routes but rather store all routes)
    • On R2RL disconnection updated information about registrations and subscriptions must be immediately advertised to active neighbors
    • During message transfer, the R2R client must mark messages with routing information that can be used to build a response route
  • The R2R Client works as a relay: it only transfers WAMP messages between router nodes
• R2RL ends do not serialize/deserialize messages, they only transfer them from one router to another. Routers still need to serialize/deserialize messages as usual to check URI, options, and mark messages with routing information
• To optimize performance and scaling RDB can store not only exact URI routes but also routes to prefix-based and wildcard-based URIs
• Routes can be configured statically on the WAMP Router side and/or dynamically during URI route advertisement
• When the router receives an RPC CALL (no matter if is it from the local client or via R2RL) it should check the local registration list and if the requested URI exists — proceed CALL locally, otherwise the router needs to check RDB, if the requested URI exists — use route information for transferring the message to the remote router, also marking message with routing information. Other cases must be failed with ERROR wamp.error.no_such_procedure
• In an environment with local and remote (via R2RL) [shared] registrations, local RPCs always have priority over remote ones.
• When the router receives a PUBLISH message (no matter if is it from the local client or via R2RL) it should check the local topic list and if the requested URI exists — proceed with publication locally, independent from that the router needs to check RDB, if the requested URI exists — use route information for transferring the message to the remote router, also marking message with routing information.
• Publication events should be transferred to remote routers only if there are any subscriptions on the target router or if the remote router is intermediate on the way to the final destination
• To work in a real decentralized environment where routers can be connected any to any for the successful delivery of messages routers require all information about possible routes. So RDB plays an important role — it stores routing information to every URI. How this can be achieved?
  • Routers must exchange information about their URIs in a timely fashion
  • To prevent storm and cyclic updates router must not advertise updates via the same R2RL by which updates were received (split horizon)
  • To compensate for short-term network outages hold down timeout must be used to pause URU advertisements
  • Every router must create a personal routing table with distance vector information to URIs
  • Router must store all known routes. So it is possible to have more than one route to the same target URI.
    • This is a discussion point: store all routes or only best one?
      • All routes:
        • Have actual load balancing info in place
        • Have backup routes info in place
        • Bigger size of RDB (that may matter only with thousands of URIs and tens of routers connections)
      • Only best routes:
        • Smaller RDB size (that may matter only with thousands of URIs and tens of routers connections)
        • Faster lookups (that may matter only with thousands of URIs and tens of routers connections)
        • Need more time to rebuild RDB on routes drop
  • Every route in RDB has a metrics, vectorized distance from itself to the target router
  • If there is more than one route to the target, the router must choose a route with a lower metrics value (the route that is closer)
  • If there is more than one route to the target with the same metrics value, the router may use both routes in a load-balance manner

Known problems

There are some questions that need to be solved in the proposal above to achieve success:

• Global and Router Scope IDs. As there is more than one source generating IDs and IDs are just numbers (not uuids or some other more random things) - we can not rely on randomness and collision might happen. This question may be solved in different ways. Anyway, remember that good old CAP theorem? That is it
  • The most reliable and the slowest one: use some sort of consensus algorithm between all router nodes serving the realm for every ID agreement. This may be done internally in router implementation or this might be delegated to external services like etcd, consul or any other distributed key-value store.
  • This can be optimized with ID precaching. This approach was (and I believe is still) used in Oracle DB Real Application Cluster (RAC). The idea is that every router node preallocates some number of IDs (100, 1000, 10000, whatever) (this can be configured statically or updated dynamically depending on the real-time usage). And be sure that every time a new ID is required - there will be no collision. During runtime as the preallocated IDs pool is exhausted - the router nodes ask for a new portion of IDs. Whom it is asking and how - it depends…. In the simplest form, there can be one master node responsible for ID allocations (but not so durable) or it can be the same approach from the bullet above.
• Multiple WAMP Message Deserialization. As every router needs to look at service WAMP message properties (besides the payload itself) it has to deserialize every message received. The only workaround I have in mind is to use streaming deserializes. So router may start deserializing WAMP message:
  • Look at the first value in the list - get the message type
  • depending on the message type - look at the next 2nd, 3rd, and 4th service properties.
  • There is no need to deserialize the whole message including the payload.

[ecorm]

As every router needs to look at service WAMP message properties (besides the payload itself) it has to deserialize every message received.

That is one aspect of WAMP that is regrettable and is too late to change. Payloads should never have been embedded in the "control message" structure. There should have been distinct headers and payloads in the way WAMP messages are framed. By "headers" I mean at the WAMP level, and not to be confused with raw socket headers.

That is,

[header_field_1, ..., header_field_n] | payload |
[header_field_1, ..., header_field_n] | payload |

would have been preferable to

[header_field_1, ..., header_field_n, payload] |
[header_field_1, ..., header_field_n, payload] |

[KSDaemon]

Heh yeah! But we have to deal with what we have.

[oberstet]

just a quick note (need to catch some sleep, really) on this single aspect .. which seems to indicate you both believe in the same thing ... which isn't true though;) sorry, or lucky? the flatbuffers WAMP serializer (implemented in autobahn/crossbar) allows zero touch / serialization / copy for application payloads (args/kwargs). please note, true zero copy means: octets within app payload are DMA'ed by the NIC into main memory, and from there again into the NIC via DMA, without the CPU ever touching those bytes, but they are still moved over PCIe bus and into/from main memory (thus, increase bus and memory pressure). this is not in ABPy/CB, mostly as it requires dealing with new kernel APIs (rings etc) and no GC. however, there is no deserialization or serialization happening or any CPU involvement for that, even today, and on ABPy/CB. one downside of course is: the transparency of WAMP wrt to the serializer used by clients, and the normalization of serializations wrt to the implementation used by the router is gone .. obviously. anyways, I hope I find time to reply to the other things, and also to the limits stuff soonish ...

[ecorm]

What about JSON and CBOR?

[oberstet]

What about JSON and CBOR?

my view:

• JSON and CBOR are not zero-serialization (and cannot be made so, not fully, and not even only partially)
• probably the original, first true zero-serialization format is https://capnproto.org/
• in general, I think Cap'n Proto and FlatBuffers are the only true contenders in the field
• Protobuf (another one coming up in this context often) supports zero-copy of strings and byte arrays embedded in a message, but does not support zero-copy of the entire message structure (pls see the author's statement here if you don't believe me https://news.ycombinator.com/item?id=23589132)
• FlatBuffers supports typed / schema based payloads and schema-less payloads https://flatbuffers.dev/flexbuffers.html
• what you seem to suggest above with "distinct headers and payloads in the way WAMP messages are framed" would provide even less, as it inter-mixes the original problem (application payload serialization) with transport (message framing) - both of which have been kept orthogonal by design since the beginning of WAMP - and only does so "ad-hoc" and for selective elements (and also not for structured data I guess .. only "opaque message payloads" .. not sure, since you don't discuss payload much in above)

[ecorm]

what you seem to suggest above with "distinct headers and payloads in the way WAMP messages are framed" would provide even less, as it inter-mixes the original problem (application payload serialization) with transport (message framing) - both of which have been kept orthogonal by design since the beginning of WAMP

What I meant is that the transport layer treats the WAMP {header, payload} pair as one big opaque message. The WAMP header would have its own serialization format which is not applied to the payload. As far as WAMP is concerned, the WAMP payload would be opaque and only be meaningful to the application layer above WAMP.

So there would be framing at both the transport and WAMP levels:

| -------------- Transport Frame -------------- |
| Transport Header | ------- WAMP Frame ------- |
                   | WAMP Header | WAMP Payload |

This is a simplified sketch that assumes an entire WAMP frame could be transferred within one transport frame.

But this is moot, as it's too fundamental a change to the existing WAMP protocol. I probably shouldn't even have brought it up.

[oberstet]

correct me if I’m wrong)

I'm not going to answer to your whole post (which is awesome feedback and great for discussion!) as I need to work on other stuff right now still, but I will comment more .. this is just to "correct" some misunderstandings rgd R2R in Crossbar.io ..

are implemented in Crossbar WAMP Router only and are based on regular WAMP connections between routers

yes, Crossbar.io is currently to the best of my knowledge the only router that implements R2R, and yes, R2R is based on as much/many already existing other WAMP features as possible - this is by design

most importantly, it reuses WAMP itself to transport R2R, it reuses WAMP meta API to keep routing data in sync, it reuses opaque payloads for "no touch of app payload" possibility of R2R routing (!) - true zero copy R2R when using R2R with Flatbuffers and WAMP opaque payloads (or E2E payloads which are opaque by nature) - and it reuses E2E for both a) zero-trust R2R (from client PoV), as well as federated WAMP (multiple router ops) - this is again only in Crossbar.io, but also in AutobahnPython and AutobahnJS for the real final deal: zero trust decentralized app messaging

using N underlying TCP connections will be established

Crossbar.io can move R2R over any WAMP transport it supports. This includes TCP, TLS, Unix domain sockets (one-time-copy), serial COM ports and ...

also function-based transport (zero-copy in-process/thread .. essentially "WAMP-over-Stack")

the latter is obviously tied to the language run-time of the router then though - if there is any, like in CB.

A WAMP router based on WASM (eg the new Crossbar.io Metal) will provide zero-copy zero-serialization via stack (it doesn't get more efficient, since stack based communication also provides zero-context-switch), and (due to the nature of WASM) is able to do so across different languages.

E.g. you can call a WAMP procedure from Python (on WASM) in C++ (on WASM), and have the results returned, both with zero copy and zero serialization, and within the same process or thread, thus zero context switching

Of course this then needs a static type description in Flatbuffers (via FB IDL) - at least for the 2 clients, not necessarily the router if e.g. using E2E as well .. "zero trust decentralized messaging for the win!" ;) - as Python is pretty dynamic by nature, so processing the result in Python requires you to dump the idea of "get the result as a regular Python" object, but an exposed object (rather than deserialized and put into a native language general object) result which "looks like" an object for normal situations though

---

The WAMP header would have its own serialization format which is not applied to the payload.

yeah, I got that! which is why I said: " and only does so "ad-hoc" and for selective elements ..". so in essence, for me (my view), this is pointless. but at the same time: the original itch is already solved! just use Flatbuffers WAMP serializer

[ecorm]

just use Flatbuffers WAMP serializer

How well is Flatbuffers implemented/supported on the web browser?

[oberstet]

it has native upstream (the flatbuffers main project) support

https://flatbuffers.dev/flatbuffers_guide_use_javascript.html
https://flatbuffers.dev/flatbuffers_guide_use_typescript.html

and the project is driven by a google employee (at least he was last time I looked)