Add support for external bridges

[lulf]

Overview

A feature requested by several users is the ability to connect address spaces to external AMQP endpoints. NOTE: this proposal does not cover bridging between address spaces on the same cluster.

The use cases covered by this proposal are:

1. Forwarding messages from a local queue in a local address space to a destination on a remote AMQP endpoint
2. Forwarding messages to a local queue in a local address space from a destination on a remote AMQP endpoint
3. Sending messages to a remote AMQP endpoint via a local address space - by creating a connector and using prefixing
4. Receiving messages from a remote AMQP endpoint via a local address space - by creating a connector and using prefixing

The external bridging feature requires a dispatch router to support the above use cases. This means that this feature will only be available in the standard address space to start off with, but may be implemented in the brokered address space as well by adding a router to handle the bridging.

Design

External Bridging Design Doc

Tasklist

• Dynamically create and update router configuration independently of the template in address-space-controller
• Add connectors and address config to router config based on address space config
• Update address space status based on connector status
• Report connector metrics in address space controller prometheus endpoint
• Validate connectors in api-server when creating address space
• Validate forwarders in api-server when creating addresses
• Create/update router linkroutes based on addresses in agent
• Update address status based on forwarder status in standard-controller
• Report forwarder metrics on standard controller prometheus endpoint

[lulf]

@rgodfrey @k-wall

Regarding bridging and forwarding, I believe we have 2 options for the implementation:

1. Message routing based forwarding (original proposal): Create a pair of autolinks to set up forwarding. 1 autolink that is configured on the remote connector, and 1 autoLink that is configured for the local queue/broker(s). This means that you have 1 link from each router to the remote endpoint, and 1 link per queue shard to one of the routers.

2. Link routing based forwarding: Create another connector from broker to router with source/target set to the remote address so that it is link-routed (already configured) to the external endpoint. This means you get only 1 link to the remote endpoint (or, 1 for each shard of a queue). We also know that link routing has better performance fwiw.

[grs]

How does option 2. work? What triggers the link?

[lulf]

(As we discussed in chat, just mentioning here for reference) In option 2, the broker creates the link using the remote address, thereby triggering the linkroute.

[lulf]

Notes to self.

Validation TODO:

• Sanity check on connector names
• Sanity check on connector address pattern
• Sanitize connector name to filesystem path in address space controller router config generator
• Sanity check on forwarder names
• Sanity check on forwarder remoteAddress (valid AMQP address)
• Sanity check on remoteAddress prefixed with a valid connector name
• Check that there is no cycle amongst forwarders.
• Sanity check on forwarder allowed only for queue and subscription address types, and only out direction for subscription

Other potential improvements:

• Allow both pattern and prefix for the connector addresses list to expose these distinct way to match addresses
• Reference documentation and examples

[lulf]

@famartinrh A couple of additional test cases we should verify:

• Use valueFromSecret for certs for instance to ensure this works
• Configure a queue in broker with low memory limit and fill it up. Then test that messages can be fetched via an address '!!GLOBAL_DLQ' on the local broker.