Tools: scripts: create mavlink_parse.py

[ES-Alexander]

A parser that finds incoming, requestable, and outgoing MAVLink messages for each vehicle. May not indicate full support, but at least shows the messages which are handled in the code.

Requirements

• Python >= 3.11
• pymavlink installed
  • Used to determine which MAVLink dialect a given message is from, and for the set of known messages
• git command accessible from terminal
  • Used to automatically determine the current branch, for the filename and the markdown code source links
  • Can be manually specified instead via the -b/--branch argument

Features

• Supports generating for a specific vehicle, or all vehicles
• Optionally allows detecting handled MAVLink commands
• Includes breakout of (otherwise undocumented) NAMED_VALUE_FLOAT and NAMED_VALUE_INT message names
• Optionally allows showing breakout of the messages included in each stream group (for the specified vehicle)
• Optionally allows finding unsupported known messages (and commands, if enabled)
  
• Supports multiple output formats:
  • Markdown (full example here, for ArduSub 4.1)
    
  • CSV
    
  • Printed (terminal) output (example generated with --vehicle ALL)
    

[amilcarlucas]

Very nice!!

[patrickelectric]

LGTM

[ES-Alexander]

@Hwurzburg no idea if you're relevant for this getting merged, but thought you might be interested to know about it, in case it's useful for the ArduPilot wiki :-)

[Hwurzburg]

can you give some sample command lines? not quite sure what you are parsing

[Hwurzburg]

please give command line example

[ES-Alexander]

I've spent some time today doing cleanup, as well as adding some extra functionalities.

Parsing overview

not quite sure what you are parsing

• "incoming" messages are detected by finding explicitly handled case MAVLINK_MSG_ID_{message} statements, in .cpp files within the libraries, as well as within the specified --vehicle (-v) folder(s)
  • the SITL and AP_Scripting libraries are excluded by default, because additional messages there are not reflective of messages that standard firmware will handle/send automatically
  • it is possible to exclude additional libraries from consideration using the --exclude-library (-e) argument
• incoming commands (--include-commands (-c)) search the same files, but are found with case MAV_CMD_{command} statements
• "outgoing" messages search the same places as "incoming" ones, but are found by mavlink_msg_{message}_send( calls
• "requestable" messages are found in the in map[] of the mavlink_msg_id_to_ap_message function
• "stream_groups" are found in the STREAM_*_msgs[] within a given vehicle/GCS_MAVLink.cpp file
  • optionally enabled, using the --include-stream-groups (-g) flag
  • ap_messages are converted to MAVLink messages using the mapping in mavlink_msg_id_to_ap_message (see "requestable" above), because some of the names are not direct conversions
• "missing" (unsupported / unhandled) messages are found by subtracting the detected messages from the set of known messages in the relevant pymavlink dialects
  • optionally enabled, using the --include-unsupported (-u) flag
  • the common, icarous, etc subsets are checked first, for specificity, and ardupilotmega is the final check which is a superset of all the 'relevant' messages
  • if --include-commands is specified, unsupported commands will also be generated with this flag enabled

Running Examples

can you give some sample command lines?

Sure :-)

1. As with most command-line programs, it's possible to get help with the --help (-h) flag

   % python mavlink_parse.py --help

   • the help info will also display if an invalid/unknown command / argument is provided
2. To generate only the incoming/requestable/outgoing MAVLink messages (for all vehicles combined), the file can be run with no arguments:

   % python mavlink_parse.py

   • this generates a printed output and a markdown file
3. To specify a particular vehicle (probably useful in most scenarios), you can use the --vehicle (-v) argument:

   % python mavlink_parse.py --vehicle ArduSub

4. To generate only an exported file (and avoid the printed output), you can use the --quiet (-q) flag:

   % python mavlink_parse.py -v ArduCopter --quiet

5. To change the export format you can use the --format (-f) argument:

   % python mavlink_parse.py -v ArduPlane -q --format csv

   • available formats are markdown, csv, and none (to only generate printed output)
6. An example using all the extra parsing options would be:

   % python mavlink_parse.py -v Rover --include-commands --include-search-groups --include-unsupported

   or equivalently:

   % python mavlink_parse.py -v Rover -cgu

7. As a complex example, if I'm on a custom branch derived from Sub-4.1 but still want the markdown links to work properly, and I want to generate a header that plays nice with ArduSub's documentation system, and I want all the extra parsing options, and I don't want printed output, I would use a command like

   python mavlink_parse.py -v ArduSub --header ArduSub -b Sub-4.1 -cguq

[patrickelectric]

• It would be nice to merge the development commits before merging the PR
• A list of valid MAV_CMD supported messages would be really nice, since a good part of the vehicle functionality is under it
  • MAV_CMD_COMPONENT_ARM_DISARM, MAV_CMD_DO_SET_MODE..

[Hwurzburg]

• It would be nice to merge the development commits before merging the PR

I think you mean "squash" the commits into one....a normal requirement...only one commit per library element (ie Tools in this case)

[ES-Alexander]

@patrickelectric thanks for the review :-)

• It would be nice to merge the development commits before merging the PR

Should they just all be squashed together?

A list of valid MAV_CMD supported messages would be really nice, since a good part of the vehicle functionality is under it
MAV_CMD_COMPONENT_ARM_DISARM, MAV_CMD_DO_SET_MODE..

I agree, which is why I added that in the latest set of updates - I'm assuming you didn't see that part? :-)
I have just noticed that I'm generating the links for that incorrectly though, so I'll fix that and update.

[Hwurzburg]

thanks for the examples...will help writing the wiki info for this

[patrickelectric]

Should they just all be squashed together?

Yes. If the code is not yet on the master branch, there is no reason to have development commits for such script on it.

I agree, which is why I added that in the latest set of updates - I'm assuming you didn't see that part? :-)

By 3 minutes 😄

[ES-Alexander]

thanks for the examples...will help writing the wiki info for this

No worries @Hwurzburg :-)

That said, I was more expecting that the output of this script could potentially be useful to include in the wiki, rather than necessarily needing info in the wiki about the script itself :-)

[ES-Alexander]

I've added a default introduction for the markdown output, so it's not just straight into the tables.

I'm done now - happy for this to be merged if there are no outstanding changes that need to be made :-)

[Hwurzburg]

let us know when you are finished changing...

[ES-Alexander]

let us know when you are finished changing...

@Hwurzburg There's nothing left that I'm planning to do. I ended up doing two unplanned updates because I found a couple of bugs, and I wasn't sure if the failed plane test was holding up merging (even though the tests are completely unrelated to the functionality here).

[amilcarlucas]

It would make sense to run this in the build pipelines so that the webpages are kept up-to-date.
@peterbarker @khancyr how hard is to add these to the build pipeline?

[Hwurzburg]

this does not run for me:
henry@hw-ubuntu:~/Desktop/ardupilot/Tools/scripts((HEAD detached at FETCH_HEAD))$ python mavlink_parse.py --h
File "mavlink_parse.py", line 329
match data:
^

[ES-Alexander]

@Hwurzburg sorry for the delay on this - I've been away and just got back.

this does not run for me:

henry@hw-ubuntu:~/Desktop/ardupilot/Tools/scripts((HEAD detached at FETCH_HEAD))$ python mavlink_parse.py --h
File "mavlink_parse.py", line 329
match data:
^

Python's match-case statements were added in Python 3.10. The latest stable Python version (as of October) is 3.11, and this program uses some of the new features, which is why I specified Python >= 3.11 as one of the requirements at the top. As a new and standalone script I don't expect that to be a problem, given it's not removing anything from people with older versions, and people who want to try new features can update to get access to them :-)

If you have already installed Python 3.11 it is possible it is not mapped to the python alias in your terminal (which currently seems to point to a version from before 3.10). The latest stable Python version should support the majority of features from previous versions, so it should generally work to run older programs through the latest version. That said, there are occasionally deprecated features/approaches from old versions that could cause issues if run with a version that no longer supports them, as well as external libraries that are not always maintained, or may not yet have a compatible release.

If it's important for you to be able to run programs from multiple Python versions (especially for testing purposes) then you may need to look into how to do so for your operating system. I personally use pyenv (on macOS), and set the relevant Python version as appropriate. It's also good practice to use virtual environments on a per-project basis, so that each project can have access to its specific library versions without dependencies conflicting with those of other unrelated programs, but I don't believe the ArduPilot codebase or documentation tends to make use of them.

[andyp1per]

Given the trouble we have getting people off python 2 even, requiring 3.11+ specifically doesn't seem great to me. 3+ would be a reasonable requirement IMO. My $0.02 only!

[ES-Alexander]

Given the trouble we have getting people off python 2 even, requiring 3.11+ specifically doesn't seem great to me. 3+ would be a reasonable requirement IMO. My $0.02 only!

While compatibility with existing systems is nice, and is valuable by reducing update friction, not allowing developers to use modern features and tools discourages them from contributing, and also means the entire technology stack is constantly held back by the most restrictive user's configuration, instead of being enabled by technology improvements as they're released.

I agree it generally doesn't make much sense to intentionally break existing functionality on old systems just to refactor (e.g. to use an improved syntax or data structure), but I equally don't think new features should be necessarily beholden to old systems and technologies. If new features can't enter the codebase via even a benign program like a standalone script that generates some useful documentation (which most users will likely never know about or need to run), then I fear the message that sends to potential contributors - especially those starting out, who learn to program and then get told their tools are too good/nice to be allowed in a given project (like not being able to use a calculator for a maths assignment, except not even for the supposed purpose of learning).

As I understand it, software requires updates to get new features, and sometimes new features are reliant on improvements in the underlying technologies, and aren't available without a recent enough update of the base. That doesn't seem unreasonable (e.g. a nail requires only a hammer, but having a hammer doesn't mean it's not also valuable to get a screwdriver so you can use screws).

---

As is, the significant technical debt in ArduPilot's Python tools, caused/required by the extensive backwards compatibility, is one of the primary reasons I don't contribute more to the ArduPilot codebase. I love the mission and open source nature of ArduPilot, and at times want to contribute, but I like the programming tools and features I'm used to, and feel like it's a regression to not be able to use them (not to mention the additional reasoning and programming time required to develop workarounds, and often hacky homebrew alternatives).

Accordingly, I either avoid making any changes (and just be a bit disgruntled with ArduPilot while I instead work on other projects where I have more development freedom), or I develop features in a local branch and don't bother trying to contribute upstream on the grounds that there'll likely be pushback that I don't have the time or motivation to deal with.

I appreciate and understand your 2 cents - hopefully my perspective and experience are also valuable :-)

[patrickelectric]

I agree with @ES-Alexander and share the same feelings as developer.

As open source organization, adopting and improving is one of the best ways to have more collaborators and engage the community. Starting with a standalone script is the best way to go to slowly adopt new python versions.

Maybe @khancyr and @Williangalvani wants to add something on the discussion

[khancyr]

I don't think we should continue support py2, and we already have some tools that don't support it.

But we should focus was is on the latest Ubuntu LTS I would said and not trying to get all the latest features, as that is what we propose to install anyway.

So max python 3.10 for now.
This still make an issue for partners that have long release cycle and still on Ubuntu 20.04 but I think we can live with this.
And those on 18.04 ... Well that is deprecated.

[ES-Alexander]

I don't think we should continue support py2, and we already have some tools that don't support it.

Fair enough

But we should focus was is on the latest Ubuntu LTS I would said and not trying to get all the latest features, as that is what we propose to install anyway.

I don't see why Ubuntu's system defaults get to determine the dependencies used for all operating systems, but that's not my call to make. As far as I'm aware it's generally not recommended to use the system Python install for user programs, so at least in my head this is a question of "which Python version should be installed?", rather than "can we avoid people needing to install Python on Ubuntu?".

So max python 3.10 for now.

Ok. I believe StrEnum is the only used 3.11 feature, so if someone wants to rewrite with an alternative then feel free. Otherwise this can get merged once an Ubuntu LTS supports Python 3.11, which I suppose will likely be some time in the next year or few? I can still use the program in the meantime, so getting this merged quickly isn't a particular priority for me.

This still make an issue for partners that have long release cycle and still on Ubuntu 20.04 but I think we can live with this.

I don't really understand the reasoning here (i.e. why would it be an issue?).

Is the idea that an active Ubuntu LTS version should be able to fetch and make use of any updates to the ArduPilot codebase without needing to install or update any dependencies or programs? If so, why is there a script to install prerequisite packages and libraries for Ubuntu - by that reasoning shouldn't they already be built into Ubuntu, or be included directly (or as submodules) in the ArduPilot codebase?

Or is the idea just that it should be possible to install prerequisites once, and not need to do any updates until also updating to a new version of Ubuntu? If so, why is it ok to need to fetch new code to get new tools/features, but not ok to also need to update dependencies to be able to use those new tools/features?

[ES-Alexander]

Fixed a typo, improved some descriptions, and improved sort order from initial output usage feedback.

[ES-Alexander]

To the subset of Ubuntu folks who aren't currently managing multiple Python versions, you should be able to test this by installing Python 3.11 through pyenv, then setting pyenv back to using the system Python (if you want to do that).

I'm not a huge fan of blanket shell scripts, but it does at least make things easy to follow... Something like this should work, without generally changing your system behaviour afterwards (beyond being able to use pyenv again later without needing to re-install it):

# Install pyenv
curl https://pyenv.run/ | bash
# Restart the shell to ensure its aliases are available
exec $SHELL
# Use it to install the desired python version
pyenv install 3.11
# Make sure you're in the ArduPilot directory
#  Can navigate into `Tools/scripts/` if you want to be extra specific
#   -> but would need to adjust the running line appropriately
cd /path/to/ArduPilot/repo/
# Set local Python version (for cwd and its children) temporarily
pyenv local 3.11
# Make sure latest pymavlink is installed
python -m pip install --upgrade pymavlink

# Run the MAVLink parser script, with whichever arguments you want
python Tools/scripts/mavlink_parse.py --help

# Revert local Python version to system python
#  As mentioned, I don't actually think this should be being used, 
#  but you can continue doing so if you want.
pyenv local system

[khancyr]

I don't see why Ubuntu's system defaults get to determine the dependencies used for all operating systems, but that's not my call to make. As far as I'm aware it's generally not recommended to use the system Python install for user programs, so at least in my head this is a question of "which Python version should be installed?", rather than "can we avoid people needing to install Python on Ubuntu?".

Ubuntu is the most used Distribution on robotics and for ArduPilot, that is what we recommand as starter and the default propose installation is LTS. Bringing new feature to ArduPilot is great and we love your contribution, but we shouldn't forget that a large part of the ArduPilot community doesn't know a lot about programming, so having to deal with multiple Python version is a pain.
That is why , we try to stick with Ubuntu default python version as this simplify maintenance and make most people happy as mostly everything works directly.

Ok. I believe StrEnum is the only used 3.11 feature, so if someone wants to rewrite with an alternative then feel free. Otherwise this can get merged once an Ubuntu LTS supports Python 3.11, which I suppose will likely be some time in the next year or few? I can still use the program in the meantime, so getting this merged quickly isn't a particular priority for me.

I will try to push a patch for this as the script is really interesting and I don't want to lost it into the PR pile...

I don't really understand the reasoning here (i.e. why would it be an issue?).
This feature is mosly intend for developpers and surely professional user. If we advertise that it is available, people will want it and will try it. But if we look at developpers, specially on partners, they will have old ubuntu distro as their release cycle is slow. So they don't update often. Don't ask me why, I don't know. But that is a fact that most companies don't update system/software often...

Is the idea that an active Ubuntu LTS version should be able to fetch and make use of any updates to the ArduPilot codebase without needing to install or update any dependencies or programs? If so, why is there a script to install prerequisite packages and libraries for Ubuntu - by that reasoning shouldn't they already be built into Ubuntu, or be included directly (or as submodules) in the ArduPilot codebase?

Or is the idea just that it should be possible to install prerequisites once, and not need to do any updates until also updating to a new version of Ubuntu? If so, why is it ok to need to fetch new code to get new tools/features, but not ok to also need to update dependencies to be able to use those new tools/features?

As said previouly, most people don't really know what all ArduPilot ecosystem is, so they just run the install script blindly once and trust us that is work. Fetching new code is generally fine as we don't need to install anything, here we need a specific python version. We cannot enforce that for now, and dealing with an update with ask a lot of time on support, so that is best to avoid.

So in resume we aren't opposed to this, on contrary. But we need to rework it a bit to lower the python version requirement as that will make it available simply for most and don't break any feature.

[ES-Alexander]

Hi @khancyr, I just saw your reply.

we shouldn't forget that a large part of the ArduPilot community doesn't know a lot about programming, so having to deal with multiple Python version is a pain.

I'm not suggesting that people should need to actively manage multiple Python versions - I'm querying whether using the Ubuntu system Python as the required Python for everyone is reasonable when there's a more recent stable Python version available.

The macOS 'install-prereqs' script installs pyenv and a relevant Python version. I don't understand why Ubuntu couldn't or shouldn't do the same, particularly given using a system's Python install is generally not recommended (as far as I'm aware).

... most people don't really know what all ArduPilot ecosystem is, so they just run the install script blindly once and trust us that is work. Fetching new code is generally fine as we don't need to install anything, here we need a specific python version. We cannot enforce that for now, and dealing with an update with ask a lot of time on support, so that is best to avoid.

This program is for generating some useful documentation, and will likely be used by very few users. It seems reasonable (to me) that new users could start on a Python version that supports it, and old users who want access to the new functionality could upgrade. This kind of incremental upgrade seems significantly less painful than requiring lots of people to update at once at a particular event that forces lots of things to lose their backwards compatibility, and attempting to avoid such an event seems like it could be a significant driver that's preventing adoption of up to date programming language and library versions in the ArduPilot ecosystem.

I will try to push a patch for this as the script is really interesting and I don't want to lost it into the PR pile...
...
So in resume we aren't opposed to this, on contrary. But we need to rework it a bit to lower the python version requirement as that will make it available simply for most and don't break any feature.

I still don't really understand or agree with the reasoning to push back the language version here, but I appreciate that you're approaching this in good faith, and Python 3.10 at least has many of the more recent features as compared to something like 3+, and is still receiving security updates.

If it's helpful, I believe the uses of StrEnum in this program can likely be replaced by something like

from enum import Enum # TODO: replace with StrEnum once Python 3.11 available

class StrEnum(str, Enum):
    """ Temporary filler until Python 3.11 available. """
    def __str__(self):
        return self.value

[amilcarlucas]

I think that this is used by very few users. And it should go in as it is.

[ES-Alexander]

@peterbarker / @tridge, do you guys have any input on the Python versioning compatibility requirements discussion here?

As it stands,

• me, @patrickelectric, @timtuxworth, @Williangalvani, and @amilcarlucas seem to think it's harmless to leave this as 3.11+, because it allows new features into the codebase via an isolated tool that isn't expected to be used by many people, so those who want the new tool can update their Python to be able to use it
• @khancyr has recommended 3.10+ as a compromise that allows it to be used with the system Python on Ubuntu 20.04
  • I have provided some commentary on how this could be satisfied, if necessary, as well as why I personally don't think it should be necessary
• @andyp1per has recommended 3.0+ given the challenges getting people off Python 2

[ES-Alexander]

Force pushed to improve some error messages if expected things/versions aren't installed, and to rebase over upstream master.

[tridge]

do you guys have any input on the Python versioning compatibility requirements discussion here?

we merged a change this week to require python 3.8 to build ArduPilot.
For a script like the one in this PR which is not a required tool I think it is fine to use features from later versions, but it would be nice if the script check the python version at the top and exited with a clear message if the python version is not sufficient
I would not though that Henry only has 3.8, and would be one of the people that would like to use this.
@Hwurzburg I may be able to help you get 3.10 or later running as an option (so you can switch between 3.8 and 3.10)

[ES-Alexander]

For a script like the one in this PR which is not a required tool I think it is fine to use features from later versions ...

Great to hear! :-)

... it would be nice if the script check the python version at the top and exited with a clear message if the python version is not sufficient ...

That wouldn't actually help (much) - match+case are new keywords in 3.10, so versions below that will raise a SyntaxError before any version-checking code could run. As is I've included a comment on the first usage, and the SyntaxError should show that line in the traceback which then says Python >= 3.10 is required for that feature.

It could help marginally for people on 3.10, but I've also added a comment for the StrEnum import line, which requires 3.11, which should accomplish effectively the same thing as an explicit version check, just without needing the extra code.

@Hwurzburg I may be able to help you get 3.10 or later running as an option (so you can switch between 3.8 and 3.10)

This comment (from above) should be helpful for that :-)

[Williangalvani]

This is super useful and now fails with a nice error message on earlier python versions.

I just had to install Python3.11 and do python3.11 -m pip install pymavlink and then run it.

I'll merge this as it doesn't really harm anyone and we want it for properly documenting ArduSub's MAVLink support =]