[Feature Request] MQTT Topic as network source

[fusedFET]

With the rise of IoT many data are already sent back to MQTT on clouds.
I have some devices where all serial lines are already used, or devices are in the field.
(And I cant use Segger RTT/Semihosting as source)
This means instead of going through a Serial, it would be easy to send the data to MQTT topic.
(Bascially you could monitor anything on a cloud service with this approach in real-time).

Describe the solution you'd like
Adding to TCP/UDP drop-down an MQTT option.
Following fields would be required:

• Host
• MQTT Topic
• Authentification mode: Basic (User+PW) Certificates (Private Key Certificate, Certificate Authority)
• MQTT Name
  Each MQTT message received is parsed, like a frame over serial.

(Yes, Serial over MQTT)

Describe alternatives you've considered
I tried many hacks to visualize data from my devices on the cloud, but those are all quite complicated architectures.

Additional context
I am using AWS IoT as backend, but any kind of MQTT broker could benefit from this.
Guess you could connect far more than devices to it, like whole cloud related metrics.

It's a pleasure to use Serial Studio. There simply is no solution which si so simple to use.
Thank you <3

[alex-spataru]

Hi,

This feature will be eventually implemented, I need to get it working for the CanSat competition because they require us to use MQTT to send incoming data over to their server. For the moment, I am currently playing with qtmqtt and learning more about MQTT in general.

So far, I would like to give users the possibility of:

• Receiving data over MQTT and displaying it.
• Generating and sending MQTT frames with incoming serial/network data (e.g. AFAIK, this is what I need to do for the competition).

As I said before, I am still learning about MQTT (and probably I could be saying some nonsense here 😆). The steps that I am considering to take are:

1. Learn (and understand) what MQTT is and how it works.
2. Integrate the qtmqtt library (or another C/C++ MQTT library) with Serial Studio and make sure that it works on all major platforms.
3. Build the GUI options/fields for the MQTT client.
4. Write the appropriate classes to let the GUI and other application modules interact with the MQTT library.
5. Do internal tests, create a release & get community feedback.

Of course, any suggestions are welcome. Again, thanks a lot for the idea & the contributions that you've made.

[fusedFET]

That sound's great!

I have never used qtmqtt before, but it seems to be the most logical choice to keep a unifrom library stack.

Just a proposition:
Serial Studio, when opened opens a defined python file as sub-process.
Each time a new data-set arrives into serial studio, the data is split to the different graph widgets, but also to
a special handler function (per-widget) in this python file with the new data as argument in JSON format.

This would allow to simply process and republish data received on serial studio to some MQTT endpoint.
It might be simpler than changing the GUI to support generating MQTT frames.
At least in python it is quite simple to generate those frames, even more if some specific format is required by your competition :)
(Beside the fact that you could maybe even use this to make some automated testing with serial studio ?)

If I can help you somehow with scripts or, some tests on a cloud architecture or similar feel free to ping me :)

[alex-spataru]

Update: I just forked the Qt-MQTT project so that I can integrate it with Serial Studio using sub-modules.

Serial Studio, when opened opens a defined python file as sub-process.
Each time a new data-set arrives into serial studio, the data is split to the different graph widgets, but also to
a special handler function (per-widget) in this python file with the new data as argument in JSON format.

This could be (relatively) easily done with QProcess, and we could give users more power with what to do with incoming data. I will take your proposal into consideration. I think that letting users modify the Python script to suit their own needs could be very useful.

I am currently in a pretty tight deadline to get this to work, and I am a lot more familiar with Qt/C++ than with Python. Also, the qt-mqtt library has some examples that I can use to get an idea on how to implement support for MQTT in Serial Studio.

The course of action that I am considering is:

• Integrate the Qt MQTT library and get it to work before I reach the deadline.
• Add support for calling Python scripts and pass the received data to these scripts to allow users to extend the functionality of Serial Studio. This can be related to the ideas discussed in Support for CAN protocol #17.

[alex-spataru]

Edit: Unfortunately, I cannot use Qt MQTT due to licensing issues. AFAIK, I cannot use GPLv3 code in a MIT-licensed project. And I personally feel GPLv3 too restrictive for the usage cases that users may have for this project.

[alex-spataru]

Update: with the latest commit, Serial Studio supports sending RX data to a MQTT server/broker in CSV format. Here is a screenshot with the MQTT server for the CanSat competition:

The next step is to allow Serial Studio to act as a MQTT subscriber. This would also allow other instances of Serial Studio in the network/world to visualize the data received and processed by a SS ground station.

Finally, here is a screenshot of the MQTT settings pane:

[alex-spataru]

I just created a new release that allows Serial Studio TO send/receive data over MQTT. RX data is sent in a CSV-like format, and other instances of Serial Studio can display the broadcasted data using the same *.json file (you just need to set the "mode" combobox to "Subscriber").

Please let me know how it works for you. Greetings!

[fusedFET]

That looks great 👍, but do you plan to support MQTT key/cert authentication for MQTT?
I have some issues to add the classic password authentication to my brokers.

Good luck for the can-sat competition 👍

[alex-spataru]

But do you plan to support MQTT key/cert authentication for MQTT?

Taking a look at the headers of the library that I am using for MQTT communications, I think that is possible. I need to do some testing, but this feature will be eventually implemented.

Good luck for the can-sat competition 👍

Thanks a lot! I will go back to working on SS as soon as I have some free time 😄

[bmalcheski]

Any chance you'll be adding TLS encryption w/ and w/o certs?

Any intention to add further MQTT configuration options such as QoS, last will, keep alive, etc?

[alex-spataru]

Hi,

The MQTT library that I use for Serial Studio has support for SSL configuration with and without certificates, QoS, will retain and keep alive. I just need to add the UI controls to interface with the functions of the library.

Do you know of any broker that I can use to test these features? I don't know if HiveMQ allows me to test these configurations...

[bmalcheski]

So no TLS support though? I looked through the header files you mentioned previously and didn't see anything about it TLS support myself. I bring it up because I was going to use SS to help me do some debug logging but can't if it doesn't support TLS. You may already be aware, but using MQTT with AWS requires SSL with certs. The issue I've run into is many edge devices I've used don't have the ability to use cert files hence encryption w/o authentication.

I would think that HiveMQ supports those features. They have a bunch of articles regarding them... Otherwise, Mosquitto is another option. Obviously, you can't change the configs for either of the publicly hosted test servers but you can run your own instance to manipulate the broker config regarding these features.

https://www.hivemq.com/downloads/
https://mosquitto.org/download/

This looks like a great project and a very useful tool!

[alex-spataru]

Thanks a lot for the links! I'll work on this issue during the weekend. I am currently on exam period. Regarding TLS, I believe that the QSslSocket class supports it (and thus QMQTT with SSL compilation flags enabled).

[alex-spataru]

Hi! I am currently implementing the requested features. Advanced MQTT options & SSL/TLS config can be set in a separate dialog that can be shown from the MQTT tab. Here is a screenshot of the dialog:

For the moment, I have finished adding support for QOS, Keep-Alive and retain flags. SSL/TLS support is underway. Please let me know if I should change anything or if I forgot to add something to the MQTT config. dialog.

[alex-spataru]

Update: I can confirm that the TLS/SSL feature works correctly with the latest commit. I will draft a new version in the upcoming days :)

[alex-spataru]

Hi, just to let you know that I just released a new version that adds SSL/TLS support for MQTT connections. Please let me know how it works for you.