Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(feature request): MQTT Retained message support between local broker and IoT Core #77

Open
shumailxyz opened this issue Aug 24, 2022 · 7 comments
Labels
enhancement New feature or request

Comments

@shumailxyz
Copy link

Describe the bug
I am running MQTT bridge on GG core, configured to forward messages between LocalMqtt to IotCore and vice versa. The bridge does not work with MQTT retained messages. When local client device publishes a retained MQTT message on a topic, forwarded by bridge to IotCore - new subscription of IotCore does not receive the retained message.

To Reproduce

  1. Run bridge with following config:
{
  "mqttTopicMapping": {
      "ClientDevicesToCloud": {
          "topic": "from_local/#",
          "source": "LocalMqtt",
          "target": "IotCore",
      },
      "CloudToClientDevices": {
          "topic": "from_cloud/#",
          "source": "IotCore",
          "target": "LocalMqtt",
      },
  }
}
  1. Publish a MQTT retained message from local client on topic from_local/test.
  2. Go to AWS Iot Core > MQTT test client.
  3. Subscribe on topic from_local/test.
  4. Retained message should appear when you subscribe.
  5. Or retained message should also appear under Retained messages page on AWS IoT Core.

Expected behavior
A local client device publishes a retained MQTT message on a topic, forwarded by bridge to IotCore. I expect that when a new client on AWS IoT Core subscribes on that topic, it receives the retained message.

Actual behavior
When a new client subscribes on topic, it does not receive the retained MQTT message.

Environment

  • OS: Amazon Linux 2 (Linux version 4.14.287-215.504.amzn2.x86_64 )
  • Nucleus version: 2.7.0
  • MQTT Bridge version: 2.2.2
  • aws.greengrass.clientdevices.mqtt.Moquette : 2.2.0

Additional context
We use a specific standard VDA5050 for AGVs - Automated Guided Vehicles (client devices) that requires messages on some specific topics to be sent with RETAINED flag set to true. The services running on AWS cloud following that standard must follow the standard and should be able to receive RETAINED messages sent by local client devices (that communicate through GG core & bridge).

@shumailxyz shumailxyz added the bug Something isn't working label Aug 24, 2022
@jbutler jbutler removed the bug Something isn't working label Aug 24, 2022
@jbutler
Copy link
Contributor

jbutler commented Aug 24, 2022

Hi, thanks for raising this.

Unfortunately this is an issue with the MQTT3 spec itself. The protocol does not provide enough information for us to know whether we should forward a message to another server with the retained bit set. MQTT5 does support this, however, so this can potentially be supported in the future once that is available in IoT Core.

@jbutler jbutler added the enhancement New feature or request label Aug 24, 2022
@jbutler
Copy link
Contributor

jbutler commented Aug 24, 2022

What I would suggest doing, for now, is to add a prefix on topics which must follow this standard. You can set up a rule in IoT Core to forward those messages to a Lambda function, which can re-publish to the original topic with the retained bit set.

https://docs.aws.amazon.com/iot/latest/developerguide/iot-rule-actions.html

Take a look at the targetTopicPrefix configuration that is currently supported: https://docs.aws.amazon.com/greengrass/v2/developerguide/mqtt-bridge-component.html#mqtt-bridge-component-configuration

@shumailxyz
Copy link
Author

@jbutler Thanks for the suggestion to use lambda + IoT rule as a workaround to republish the messages. I'll give it a try.

Would be great to see IoT Core support for MQTTv5 so this limitation can be resolved. I see that Greengrass already supports MQTT v5 so I hope IoT Core supports MQTTv5 soon as well.

@jbutler jbutler changed the title (MQTT Bridge): MQTT Retained messages not working (MQTT Bridge): MQTT Retained message support between local broker and IoT Core Aug 26, 2022
@jbutler jbutler changed the title (MQTT Bridge): MQTT Retained message support between local broker and IoT Core (feature request): MQTT Retained message support between local broker and IoT Core Aug 26, 2022
@jbutler
Copy link
Contributor

jbutler commented Aug 26, 2022

@shumailxyz Let me know how it goes. Do you need both directions? Or is Greengrass to IoT Core sufficient?

@shumailxyz
Copy link
Author

@jbutler Worked for Greengrass to IoT Core using iot-rule & Lambda.
We do need both ways though as VDA5050 protocol specifies publishing messages as retained messages to 2 topics. One topic where IoT Core listens and other topic where Greengrass Client device listens.

Any ideas if we can receive retained messages on Greengrass client using some workaround while we wait for official support to be added? Thanks.

@jbutler
Copy link
Contributor

jbutler commented Sep 7, 2022

The workaround in the reverse direction is somewhat ugly. You could follow a similar pattern, except replace iot rule + lambda with another client device which republishes with the retain bit set. You could deploy this client device as a Greengrass component -- the key is that it would need to connect directly to the broker rather than via PubSub.

Alternatively, we could also add additional publish parameters as part of the topic mapping configuration. For example, something like this:

{
  "mqttTopicMapping": {
      "CloudToClientDevices": {
          "topic": "from_cloud/#",
          "source": "IotCore",
          "target": "LocalMqtt",
          "targetOptions": {
            "qos": 1,
            "retain": true
          }
      },
  }
}

...where targetOptions is dependent on the target transport. In this case, it would include MQTT options. Eventually this approach could also work for the LocalMqtt->IotCore path to replace the Rule+Lambda, but there are some other gaps with our existing IPC APIs preventing this today.

I'm not sure when we can get around to implementing this, but we'd certainly accept a PR if you're willing to help out.

@jcosentino11
Copy link
Contributor

@shumailxyz we just released MQTT5 support for bridge, which supports retain-as-published.

Please note that retain as published isn't yet supported on IoT Core, so this will only work for LocalMqtt -> IotCore bridging. So this half solves your use case :)

To enable, upgrade to bridge 2.3.0 and set add the following configuration:

{
    "mqtt5RouteOptions": {
        "ClientDevicesToCloud": {
            "retainAsPublished": true
        }
    },
    "mqttTopicMapping": {
        "ClientDevicesToCloud": {
            "topic": "from_local/#",
            "source": "LocalMqtt",
            "target": "IotCore"
        }
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants