Proxy support remoting protocol client compatibility

[echooymxq]

For Proxy support remoting protocol, as far as I can currently see, the bname field will be added to the request header. Low version clients do not have this field. Can we do queue mapping at the proxy layer? For example, a named topic TopicTest, it contains broker-a{0,1,2,3} and broker-b{0,1,2,3}, which are converted to the TopicTest, broker-gateway{0,1,2,3,4,5,6,7}, and then the original queueId and brokerName are converted according to the queueId requested by the client, it aims to support low-version clients. Any ideas?

[echooymxq]

cc @drpmma

[drpmma]

The queue mapping idea had been considered before, but it's not easy to maintain the queue mapping rule.

First of all, the storage of queue mapping rules is a problem. If it's stored in the broker, then the broker needs to be upgraded. So let's suppose the proxy maintains the queue mapping relationship in memory.

Let's take your example, there is a topic TopicTest, which contains broker-a{0,1,2,3} and broker-b{0,1,2,3} are mapping to broker-gateway{0,1,2,3,4,5,6,7}.

How does proxy deal with the case that the broker-a{0,1,2,3} scale to broker-a{0,1,2,3,4,5,6,7}? If the mapping is append-only, which means { broker-a{0,1,2,3}, broker-b{0,1,2,3}, broker-a{4,5,6,7} }, the rule will be different and inconsistent when proxy restart. If the mapping is { broker-a{0,1,2,3,4,5,6,7}, broker-b{0,1,2,3} }, then the specified mapping will be changed and cause inconsistent pull behavior.

If the mapping rules are stored in the broker or even other storage like a database. The brokers' scale or offline behavior will also be a big challenge in maintaining the rules which need to be corrected when it happens.

So IMO, only supporting the new version client is the simplest way to keep the original protocol semantics. The support of remoting protocol in proxy is introduced in 5.0.1 as planned, so it's natural to support the client afterward.

[echooymxq]

@drpmma If not use the append-only, what can go wrong with this? such as, if the broker-a scaling up the queues, the proxy get metadata broker-a {0, 1, 2, 3, 4, 5, 6, 7} and broker-b{0,1,2,3}, then recalculate the queue mappings. What i can think of is the proxy's queue mapping can be inconsistent when some proxys pull metadata delay. What is the impact on the client? I am very curious how Aliyun MQ implements it, because it's 5.x commercial products support the 4.x sdk, any ideas?

[drpmma]

@drpmma If not use the append-only, what can go wrong with this? such as, if the broker-a scaling up the queues, the proxy get metadata broker-a {0, 1, 2, 3, 4, 5, 6, 7} and broker-b{0,1,2,3}, then recalculate the queue mappings. What i can think of is the proxy's queue mapping can be inconsistent when some proxys pull metadata delay. What is the impact on the client? I am very curious how Aliyun MQ implements it, because it's 5.x commercial products support the 4.x sdk, any ideas?

Firstly, we need to clarify how the metadata is stored before more discussion. What is your opinion on metadata storage?

[echooymxq]

I prefer the proxy to get the topic routing data from namesrv and then the proxy compute and store it in memory.

[echooymxq]

Can we chat this on dingtalk? 16c-h625wjc7si

[drpmma]

It's hard to detect queue mappings immediately cause the broker route is not in real-time. In the meantime, it's also a challenge to handle messaging situations and streaming situations in queue mapping. Because in messaging case, the consumer usually doesn't care about message offset, while in the streaming scenario, offset is important. And proxy is not able to distinguish "messaging pull" or "streaming pull" cases.

In a word, introducing metadata and queue mapping will make a huge complexity on topic and proxy in no doubt. We wish to keep the main logic easy and simple as well as keep the pull semantics as before.

[echooymxq]

We had discuss on dingTalk, the conclusion is too complexity about queue mappings. The current implementation is most simple way.Thanks your the answers @drpmma