[Bug] Issue Transporting DDS Topics Across UART

[McMullan-H]

Describe the bug

This issue regards a wired connection between the FMU running PX4 (NXP FMUK66) and a companion computer (Nvidia Jetson Nano), something which has been discussed for a few weeks now on the Discord server. It is a two-part issue, but one relates to the other. Firstly, when using two-wire UART, it appears that a rate limit of 230400 baud is reached with the Micro XRCE DDS agent, after which communications break down entirely. The default rate of 921600 for the telem2 port is highly optimistic. This noticeably bottlenecks the bandwidth of the line using the existing dds_topics.yaml list within the uxrce_dds_agent module. This could be a fault of the driver itself, the line quality, or the fact that flow control is not being used. However, I was able to exceed this rate sending raw UART data from an Arduino using the same lines (beyond 1M baud).

Secondly, when operating at the maximum achievable rate of 230400 baud, it appeared that the Micro XRCE DDS client driver would periodically halt, and restart communications. This was observed on the terminal running the Micro XRCE DDS agent on the companion computer. This happens every few seconds, making it impossible to use with a ROS node. When topics were received, they were measured at rates of between 1-2 Hz.

What is strange is that one fix for this is to comment out the DDS topics that you aren't using in the dds_topics.yaml list. This ends up dramatically improving the reliability of the link, which would suggest that the large amount of topics that are being sent at such a rate by the DDS client is causing it to crash. One would even suggest rate-limiting the output (not my idea!). Rates improved beyond 25 Hz when listing only a few topics.

Final note; this issue was discovered on the branch release/V1.14 . In fact, when using the main branch, a DDS connection could not be made through UART at any baud rate. However, it hasn't been tested yet whether removing DDS topics would affect this. Perhaps this folds into the same issue.

To Reproduce

Prerequisites

• ROS2 interface installed on companion computer and underlay sourced.
• Micro XRCE DDS agent installed on companion computer and running.
• A 6-pin JST-GH-to-UART cable on hand.

Method

1. Clone PX4-Autopilot V1.14 firmware.
   git clone --recursive git@github.com:PX4/PX4-Autopilot.git -b release/1.14

2. Make the firmware and flash this to your FMU.
   make <vendor>_<model>_default upload

3. Set the following parameters in QGroundControl (or similar GCS) and reboot FMU.

   UXRC_DDS_CONFIG = TELEM 2
   SER_TEL2_BAUD = 230400 8N1
   MAV_1_CONFIG = Disabled
   MAV_2_CONFIG = Disabled

4. Connect the JST-GH connector to the telem1 port and wire the appropriate serial connections to your companion computer.

5. You should immediately start to see the agent destroy and restart the session. Source a ROS workspace with the px4_msgs module prebuilt and listen to one of the topics, e.g.
   ros2 topic echo /fmu/out/sensor_combined
   or
   ros2 topic hz /fmu/out/sensor_combined

6. You should witness the horrible transfer rate. Remove some topics from the dds_topics.yaml file and rebuild/reflash the firmware. You should notice the performance improve from there.

Expected behavior

Expecting a transfer rate similar to that witnessed using UDP with the SITL simulation. However with the bandwidth restrictions over UART, this is impossible.

Screenshot / Media

No response

Flight Log

N/A

Software Version

PX4-Autopilot V1.14

Flight controller

NXP FMUK66 Rev. E (bought direct from vendor)

Vehicle type

Multicopter

How are the different components wired up (including port information)

No response

Additional context

No response

[dirksavage88]

I have been able to reliably stream ROS 2 DDS (specifically visual odometry) via high speed serial at 3 mbps: https://github.com/ARK-Electronics/ark_companion_scripts/blob/3259af3fc411acbe69a6fd49be58598f3db0eaea/services/dds-agent.service#L5

This is with an arkv6 on 1.14 and all default dds topics enabled.

I have an NXP FMUK66v3 I can probably try to test, but these boards are severely resource constrained. I have an old rev of the board so it's not a 1:1 comparison but I can try to connect via high speed serial to the Jetson board and test.

[dakejahl]

Solutions in order of importantance IMO

• driver detects when required bandwidth exceeds available bandwidth and emits a warning (calculated from baud rate and message sizes/rates) so that the user can quickly discover the problem
• Output queue quietly drops messages to keep the rate within the available bandwidth
• Run time mechanism to enable/disable topics

[dirksavage88]

@McMullan-H I've had better luck getting the NXP FMUK66 to work with DDS over the 2-wire ethernet (not to mention it frees up an entire serial port).

For the Jetson you would need a T1-converter board.

With the NXP board I have gotten pretty used to finding ways around the limitations, but they usually involve expanding via DroneCAN sensors, or in this case using the T1 ethernet.

One other thing to try for serial is the following script )with increasing the baud rate and checking the terminal on your PC (given you have an FTDI cable)