Messages

Simple lean message serialization and deserialization (with property tree support!)

Overview

Given a json declaration of binary message structures, automatically generate C++ and python code to implement managing, serializing, and deserializtion (pack/unpack) of the messages.

Messaging is an often repeated task that is subject to mistakes, typos, and inconsistencies between sending and receiving ends. This system helps streamline the process, makes the code more consistant (autogenerated) and reduces the chances that one end becomes out of sync due to a typo or mistake.

A simple/reliable messaging system opens doors to using a distributed systems design in your project.

This messaging system is conceptually similar to mavlink, but without also defining 80 pages of standard message types covering every possible remote vehicle, device, and use-case in the universe. The intention is for this system to remain small and simple.

• autogen.py: python script to auto generate C++ and python structures and code to manage the message types.
• example.json: an simple example message specification.
• messages.h: The autogenerated C++ header file including message structs and pack/unpack methods.
• messages.py: The autogenerated Python module that implements the identical byte stream serialization as the C++ code.
• example.cxx: An example C++ host program.
• example.py: An example python host program.

Why is this important?

• Hand crafted messaging systems can be fragile. They can be difficult to maintain. They often do not scale well. A manual change on one end of the pipe requires changes on the other end (and to all the other places in the code that might use or reference the message.) It is not rocket science, but it is clunky to do this all with by hand. It is very easy to miss a spot and introduce communication glitches in your system.
• Defining messages at a higher level (i.e. in a custom format based on json) allows the code generator to immediately output trusted code any time a message is changed or a new message is added.
• The auto-generated code can be included any place that reads or writes the messages ensuring both ends of the communication pipe will always be in perfect agreement.
• The messaging system can assist with reducing packet sizes and data bandnwidth usage. For example: it can automatically scale 4 or 8 byte floating point values to a new range and represent the value as an int (using fewer bytes.) Then the original floating point value can be reconstructed on the receiving end automatically. For example if I expect a voltage (represented as a floating point value) to never exceed 25.5 volts and I don't need the value beyond a single decimal place, I could multiply the value by 10 and cast it as a single byte (unsigned) int in the transmitted message. The receiving/unpacking end will automatically divide the int value (0-255) by 10 and recast it as a float. Thus I've packed a 4-byte value into 1-byte simply by understanding something about the expected range of values and the necessary precision. (When transmitting messages over a radio modem or logging long flights, a few bytes saved in each message can add up to huge savings over time.)

Where can this system be used?

• In my current work I am using this system for 2-way communication between a big processor/little process architecture. The little processor is arduino. It reads all the sensor data and transmits the values as messages to the big processor. The big processor (beaglebone, raspberry pi, etc.) runs linux and does all the heavy lifting -- things like running the extended kalman filter, wgs84-based navigation, in-line python-based task scripting system, communication, and logging.
• I use this messaging system to efficiently communicate between a remote flying vehicle and a ground station. The over-the-air radio link has limited bandwidth, so packet size, data encoding, and packaging overhead are all important considerations.
• I use the samed packed messages for data file logging. The onboard binary log of messages can be decoded after the flight, and all the original floating point values reconstructed for analysis, plotting, etc.
• I even use this system to write hardware setup configuration messages into the EEPROM of our little processor (a teensy-3.2 or 3.6.)

What about Mavlink or protobuf?

There are plenty of existing systems that already do this. Why create another one?

• Yes, there is nothing new under the sun! If you are familiar with something else and hapily using it, that is great, I have nothing more to say here. :-)
• Mavlink already does pretty much exactly the same things as this messaging system (and much much much more). It was created for UAV's. However, the mavlink spec carries with it a huge list of standard messages that cover every possible use case for every possible vehicle and sensor and mode of operation. This translates in a very large amount of code -- 99% of which we would never use. That amount of size and complexity will never come totally for free. The mavlink2 protocol has a substantial number of bytes of overhead per message, especially if you use the signing feature.
• Yes, protobuf does just about everything as well, but was designed for communication inside google's cloud network with high speed network links passing large informational messages around. Here again, it is a huge, do-everything-for-everyone style tool that carries a lot of weight and complexity with it.
• We decided a lean and simple system would benefit our projects more than some gigantic, do everything tool. We just reinvented the wheel, but we made it the exact size and fit our projects.

Examples of things to watch out for

• I want to represent a floating point value as a uint16_t. Ex: I am sending alitude (floating point) as a uint16_t to save 2 bytes and I only need a value to the closest integer because I am only using this to draw an altimeter gauge.

  Wrong: { "type": "uint16_t", "name": "altitude_msl_ft" } -- Python will spew an error trying to pack a float type into a uint16_t

  Right: { "type": "float", "name": "altitude_msl_ft", "pack_type": "uint16_t", "pack_scale": 1 } -- This is a bit more verbose, but allows the correct type conversions to happen under the hood.

  If your entire communication pipeline will only involve C++, then the 'wrong' approach will work fine because of C++ implicite type casting rules ... however it is probably better to use the 'right' approach anyway.