Rewrite rule on $SYS topics does not rewrite topic in messages

[oholsen]

Environment:

• EMQ X version 4.2-rc.1 (Docker)

What happened and what you expected to happen:

I added the following rewrite rule (with the intention of bridging $SYS topics to a destination.

module.rewrite.rule.1 = metrics/# ^metrics/(.+)$ $SYS/#

Subscribing to metrics/# topics, I received messages, but I expected to receive metrics/# topics. Instead, the messages have the original $SYS topics as shown using mosquitto_sub -t 'metrics/#' -v -d:

Client mosq-KaIb4Lg1iy8siCUXIw sending CONNECT
Client mosq-KaIb4Lg1iy8siCUXIw received CONNACK (0)
Client mosq-KaIb4Lg1iy8siCUXIw sending SUBSCRIBE (Mid: 1, Topic: metrics/#, QoS: 0, Options: 0x00)
Client mosq-KaIb4Lg1iy8siCUXIw received SUBACK
Subscribed (mid: 1): 0
Client mosq-KaIb4Lg1iy8siCUXIw received PUBLISH (d0, q0, r1, m0, '$SYS/brokers/c4bc43b0b941@172.17.0.7/sysdescr', ... (12 bytes))
$SYS/brokers/c4bc43b0b941@172.17.0.7/sysdescr EMQ X Broker
Client mosq-KaIb4Lg1iy8siCUXIw received PUBLISH (d0, q0, r1, m0, '$SYS/brokers/c4bc43b0b941@172.17.0.7/version', ... (8 bytes))
$SYS/brokers/c4bc43b0b941@172.17.0.7/version 4.2-rc.1
Client mosq-KaIb4Lg1iy8siCUXIw received PUBLISH (d0, q0, r1, m0, '$SYS/brokers', ... (23 bytes))
$SYS/brokers c4bc43b0b941@172.17.0.7
Client mosq-KaIb4Lg1iy8siCUXIw received PUBLISH (d0, q0, r0, m0, '$SYS/brokers/c4bc43b0b941@172.17.0.7/uptime', ... (22 bytes))
$SYS/brokers/c4bc43b0b941@172.17.0.7/uptime 26 minutes, 30 seconds
Client mosq-KaIb4Lg1iy8siCUXIw received PUBLISH (d0, q0, r0, m0, '$SYS/brokers/c4bc43b0b941@172.17.0.7/datetime', ... (19 bytes))
$SYS/brokers/c4bc43b0b941@172.17.0.7/datetime 2020-08-25 13:34:47
^CClient mosq-KaIb4Lg1iy8siCUXIw sending DISCONNECT

How to reproduce it (as minimally and precisely as possible):

Add the above line to emqx.conf, load the rewrite module, and subscribe to metrics/# with mosquitto_sub, wait for sys topics to be published. (Default broker.sys_* settings).

Anything else we need to know?:

The intention is to bridge $SYS topics in a chain of MQTT services, accumulating metrics in metrics/ on the final service in the chain. I guess this is also possible to achieve with the rules engine. However, the above appears to be a bug. Or is there an issue with the rewrite configuration?

[Rory-Z]

Hi, @oholsen
Sorry, I didn't understand what you want to do. In my opinion, emqx has implemented the rewrite rules you defined and subscribed to the $SYS/# topic correctly, so what is the problem?

[oholsen]

As an external client, when I subscribe for metrics/# and get messages on the $SYS/# topics, this is very inconsistent. I would expect the rewrite rule also to apply to the PUBLISH messages, not just the SUBSCRIBE message.

[Rory-Z]

Hi, @oholsen

The rewrite rule will be applied to PUBLISH messages, but it may be different from what you understand. In your rule, if you send a message to the topic metrics/#, it will be sent to the topic $SYS/#. Replacement behavior is consistent with SUBSCRIBE, not the opposite

In addition, the rule you defined is problematic. The last part of the rewrite rule should be the grammar of the target expression in the regular expression, not the grammar of the topic filter of MQTT. For example, in your rule, if I send a message to the subject metrics/test, the message will be rewritten as the subject $SYS/#, and the subject of the PUBLISH message is not allowed to contain # or +, which will cause a protocol error

If you change the rule to the following content, when sending a message to metrics/test, it will be rewritten as $SYS/test correctly

module.rewrite.rule.1 = metrics/# ^metrics/(.+)$ $SYS/$1

[oholsen]

Thanks for the feedback! This rule now works - and bridges nicely from metrics/# to the destination. However, the question is how to escape the $ in $SYS for the regular expression column? This is a workaround (using .) - can we be more specific? ($ does not appear to work).

module.rewrite.rule.2 = $SYS/# ^.SYS/(.+)$ metrics/from2/$1

[Rory-Z]

Hi, @oholsen
In fact, you should not use topic rewriting to subscribe to any topic starting with $, which is inconsistent with the mqtt protocol. We will fix this next.
Can I ask why you use the rewrite rule to rewrite the system theme ?

[oholsen]

Hi,
We have multiple mqtt servers (clusters) in a chain and want to observe the health of all the servers from the final mqtt server in the chain. The system operates under industrial cyber-security constraints where only outgoing TCP connections are allowed, i.e. bridging from one to the next. Our immediate thought is to forward $SYS metrics from one server to the next. The rewrite now works fine, forwarding metrics/# with the bridge plugin. What is the issue with this setup? As is, we forward all $SYS topics. We can easily reduce this to a small subset: number of messages sent and published and the datetime heartbeat, in case we should not forward all $SYS topics.

[Rory-Z]

@oholsen
Maybe mountpoint can meet your needs, you can view the document here