Introduce new AMQP 1.0 address format

[ansd]

What?

Introduce a new address format (let's call it v2) for AMQP 1.0 source and target addresses.

The old format (let's call it v1) is described in
https://github.com/rabbitmq/rabbitmq-server/tree/v3.13.x/deps/rabbitmq_amqp1_0#routing-and-addressing

The only v2 source address format is:

/queue/:queue

The 4 possible v2 target addresses formats are:

/exchange/:exchange/key/:routing-key
/exchange/:exchange
/queue/:queue
<null>

where the last AMQP value format requires that each message’s to field contains one of:

/exchange/:exchange/key/:routing-key
/exchange/:exchange
/queue/:queue

Why?

The AMQP address v1 format comes with the following flaws:

1. Obscure address format:

Without reading the documentation, the differences for example between source addresses

/amq/queue/:queue
/queue/:queue
:queue

are unknown to users. Hence, the address format is obscure.

2. Implicit creation of topologies

Some address formats implicitly create queues (and bindings), such as source address

/exchange/:exchange/:binding-key

or target address

/queue/:queue

These queues and bindings are never deleted (by the AMQP 1.0 plugin.)
Implicit creation of such topologies is also obscure.

3. Redundant address formats

/queue/:queue
:queue

have the same meaning and are therefore redundant.

4. Properties section must be parsed to determine whether a routing key is present

Target address

/exchange/:exchange

requires RabbitMQ to parse the properties section in order to check whether the message subject is set.
If subject is not set, the routing key will default to the empty string.

5. Using subject as routing key misuses the purpose of this field.

According to the AMQP spec, the message subject field's purpose is:

A common field for summary information about the message content and purpose.

6. Exchange names, queue names and routing keys must not contain the "/" (slash) character.

The current 3.13 implemenation splits by "/" disallowing these
characters in exchange, and queue names, and routing keys which is
unnecessary prohibitive.

7. Clients must create a separate link per target exchange

While this is reasonable working assumption, there might be rare use
cases where it could make sense to create many exchanges (e.g. 1
exchange per queue, see
#10708) and have
a single application publish to all these exchanges.
With the v1 address format, for an application to send to 500 different
exchanges, it needs to create 500 links.

Due to these disadvantages and thanks to #10559 which allows clients to explicitly create topologies,
we can create a simpler, clearer, and better v2 address format.

How?

Design goals

Following the 7 cons from v1, the design goals for v2 are:

1. The address format should be simple so that users have a chance to
   understand the meaning of the address without necessarily consulting the docs.
2. The address format should not implicitly create queues, bindings, or exchanges.
   Instead, topologies should be created either explicitly via the new management node
   prior to link attachment (see Enable AMQP 1.0 clients to manage topologies #10559), or in future, we might support the dynamic
   source or target properties so that RabbitMQ creates queues dynamically.
3. No redundant address formats.
4. The target address format should explicitly state whether the routing key is present, empty,
   or will be provided dynamically in each message.
5. Subject should not be used as routing key. Instead, a better
   fitting field should be used.
6. Exchange names, queue names, and routing keys should allow to contain
   valid UTF-8 encoded data including the "/" character.
7. Allow both target exchange and routing key to by dynamically provided within each message.

Furthermore
8. v2 must co-exist with v1 for at least some time. Applications should be able to upgrade to
RabbitMQ 4.0 while continuing to use v1. Examples include AMQP 1.0 shovels and plugins communicating
between a 4.0 and a 3.13 cluster. Starting with 4.1, we should change the AMQP 1.0 shovel and plugin clients
to use only the new v2 address format. This will allow AMQP 1.0 and plugins to communicate between a 4.1 and 4.2 cluster.
We will deprecate v1 in 4.0 and remove support for v1 in a later 4.x version.

Additional Context

The address is usually a String, but can be of any type.

The AMQP Addressing extension
suggests that addresses are URIs and are therefore hierarchical and could even contain query parameters:

An AMQP address is a URI reference as defined by RFC3986.

the path expression is a sequence of identifier segments that reflects a path through an
implementation specific relationship graph of AMQP nodes and their termini.
The path expression MUST resolve to a node’s terminus in an AMQP container.

The Using the AMQP Anonymous Terminus for Message Routing Version 1.0
extension allows for the target being null and the To property to contain the node address.
This corresponds to AMQP 0.9.1 where clients can send each message on the same channel to a different {exchange, routing-key} destination.

The following v2 address formats will be used.

v2 addresses

A new deprecated feature flag amqp_address_v1 will be introduced in 4.0 which is permitted by default.
Starting with 4.1, we should change the AMQP 1.0 shovel and plugin AMQP 1.0 clients to use only the new v2 address format.
However, 4.1 server code must still understand the 4.0 AMQP 1.0 shovel and plugin AMQP 1.0 clients’ v1 address format.
The new deprecated feature flag will therefore be denied by default in 4.2.
This allows AMQP 1.0 shovels and plugins to work between

• 4.0 and 3.13 clusters using v1
• 4.1 and 4.0 clusters using v2 from 4.1 to v4.0 and v1 from 4.0 to 4.1
• 4.2 and 4.1 clusters using v2

without having to support both v1 and v2 at the same time in the AMQP 1.0 shovel and plugin clients.
While supporting both v1 and v2 in these clients is feasible, it's simpler to switch the client code directly from v1 to v2.

v2 source addresses

The source address format is

/queue/:queue

If the deprecated feature flag amqp_address_v1 is permitted and the queue does not exist, the queue will be auto-created.
If the deprecated feature flag amqp_address_v1 is denied, the queue must exist.

v2 target addresses

v1 requires attaching a new link for each destination exchange.
v2 will allow dynamic {exchange, routing-key} combinations for a given link.
v2 therefore allows for the rare use cases where a single AMQP 1.0 publisher app needs to send to many different exchanges.
Setting up a link per destination exchange could be cumbersome.
Hence, v2 will support the dynamic {exchange, routing-key} combinations of AMQP 0.9.1.
To achieve this, we make use of the "Anonymous Terminus for Message Routing" extension:
The target address will contain the AMQP value null.
The To field in each message must be set and contain either address format

/exchange/:exchange/key/:routing-key

or

/exchange/:exchange

when using the empty routing key.

The to field requires an address type and is better suited than the `subject field.

Note that each message will contain this To value for the anonymous terminus.
Hence, we should save some bytes being sent across the network and stored on disk.
Using a format

/e/:exchange/k/:routing-key

saves more bytes, but is too obscure.
However, we use only /key/ instead of /routing-key/ so save a few bytes.
This also simplifies the format because users don’t have to remember whether to use spell routing-key or routing_key or routingkey.

The other allowed target address formats are:

/exchange/:exchange/key/:routing-key

where exchange and routing key are static on the given link.

/exchange/:exchange

where exchange and routing key are static on the given link, and routing key will be the empty string (useful for example for the fanout exchange).

/queue/:queue

This provides RabbitMQ beginners the illusion of sending a message directly
to a queue without having to understand what exchanges and routing keys are.
If the deprecated feature flag amqp_address_v1 is permitted and the queue does not exist, the queue will be auto-created.
If the deprecated feature flag amqp_address_v1 is denied, the queue must exist.
Besides the additional queue existence check, this queue target is different from

/exchange//key/:queue

in that queue specific optimisations might be done (in future) by RabbitMQ
(for example different receiving queue types could grant different amounts of link credits to the sending clients).
Therefore, a target address containing the default exchange is forbidden.
A write permission check to the amq.default exchange will be performed nevertheless.

v2 will prohibit the v1 static link & dynamic routing-key combination
where the routing key is sent in the message subject as that’s also obscure.
For this use case, v2’s new anonymous terminus can be used where both exchange and routing key are defined in the message’s To field.

(The bare message must not be modified because it could be signed.)

The alias format

/topic/:topic

will also be removed.
Sending to topic exchanges is arguably an advanced feature.
Users can directly use the format

/exchange/amq.topic/key/:topic

which reduces the number of redundant address formats.

v2 address format reference

To sump up (and as stated at the top of this commit message):

The only v2 source address format is:

/queue/:queue

The 4 possible v2 target addresses formats are:

/exchange/:exchange/key/:routing-key
/exchange/:exchange
/queue/:queue
<null>

where the last AMQP value format requires that each message’s to field contains one of:

/exchange/:exchange/key/:routing-key
/exchange/:exchange
/queue/:queue

Hence, all 8 listed design goals are reached.

The disadvantage of this PR is that the new v2 address formats deviate from the STOMP address formats.

Documentation is provided in rabbitmq/rabbitmq-website#1874